Help decoding terrainGrid in save file

[jamessimo]

Hey guys, I am making a web viewer that will let people look at their saves maps. I have found a few thread that talk about it but there very old
https://ludeon.com/forums/index.php?topic=1560.0
https://ludeon.com/forums/index.php?topic=1083.15

It used to be that you could just take the terrainGrid Base64 and decode it into an array of ints and then use each int as a texture and map it.

Now however when I decode the terrainGrid it doesn't seem to be the case anymore. Does anyone know what terrainGrid should look like decoded? Is it an array of xml objects? any help would be very appreciated.

Also can anyone explain what is upperGrid and underGrid? (its apart of the terrainGrid)

[neitsa]

I just took a quick look as I'm not familiar with this part of the code. If you can read C#, everything is in Verse.TerrainGrid.

Below is ExposeData() which reads or writes the terrain grid with both the topgrid and undergrid:

Code Select Expand


// Verse.TerrainGrid
public void ExposeData()
{
this.ExposeTerrainGrid(this.topGrid, "topGrid");
this.ExposeTerrainGrid(this.underGrid, "underGrid");
}


ExposeTerrainGrid() calls ExposeUshort(). The latter takes 4 params:

* the map
* code to read from the terraingrid
* code to write to the terraingrid
* the current grid label ("topgrid" or "undergrid")

Code Select Expand


private void ExposeTerrainGrid(TerrainDef[] grid, string label)
{
//
    // code for reader and writer (omitted)
    //
   
MapExposeUtility.ExposeUshort(this.map, shortReader, shortWriter, label);
}


If we are in save mode the code creates an array from the terrain map (see "MapSerializeUtility.SerializeUshort")
which is then passed to DataExposeUtility.ByteArray():

Code Select Expand


// Verse.MapExposeUtility
public static void ExposeUshort(Map map, Func<IntVec3, ushort> shortReader, Action<IntVec3, ushort> shortWriter, string label)
{
byte[] arr = null;
if (Scribe.mode == LoadSaveMode.Saving)
{
arr = MapSerializeUtility.SerializeUshort(map, shortReader);
}
DataExposeUtility.ByteArray(ref arr, label);
if (Scribe.mode == LoadSaveMode.LoadingVars)
{
MapSerializeUtility.LoadUshort(arr, map, shortWriter);
}
}


DataExposeUtility.ByteArray() takes two parameters:

* the array of bytes that describes the map tiles.
* the label of the grid (once again: "topgrid" or "undergrid")

Code Select Expand


// Verse.DataExposeUtility
public static void ByteArray(ref byte[] arr, string label)
{
if (Scribe.mode == LoadSaveMode.Saving)
{
if (arr != null)
{
byte[] array = CompressUtility.Compress(arr);
if (array.Length < arr.Length)
{
string text = DataExposeUtility.AddLineBreaksToLongString(Convert.ToBase64String(array));
Scribe_Values.Look<string>(ref text, label + "Deflate", null, false);
}
else
{
string text2 = DataExposeUtility.AddLineBreaksToLongString(Convert.ToBase64String(arr));
Scribe_Values.Look<string>(ref text2, label, null, false);
}
}
}


As you can see there's an interesting line:

Code Select Expand


byte[] array = CompressUtility.Compress(arr);
if (array.Length < arr.Length)
{
    //...


The code compresses the byte array and if the length of the compressed array is less than the array itself (some compressed data happens to be bigger than their non compressed state)
then the code does the following:

* convert the compressed array to base64
* write the label with "deflate" suffixed so you should get something like "topgridDeflate" or "undergridDeflate" in the save file.

About the compression method used:

Code Select Expand


// Verse.CompressUtility
public static byte[] Compress(byte[] input)
{
MemoryStream memoryStream = new MemoryStream();
DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Compress);
deflateStream.Write(input, 0, input.Length);
deflateStream.Close();
return memoryStream.ToArray();
}


It's a "simple" deflate algorithm, nothing fancy. I guess it can be opened with common tools that handle the ZIP format.

Also can anyone explain what is upperGrid and underGrid? (its apart of the terrainGrid)

There's a debug mode associated with the terrain grid:

* Start Rimworld and enter "dev mode"
* On the top menu click "open the view settings"
* In that menu, check:
    - "god mode"
    - "write cells content"
    - "write terrain"
    - close that window
   
- Go to the main "menu" button (bottom right of the screen)
- Options > Keyboard Configuration
- Find the "Developer tools" category and check which key is associated with the "Toggle Debug Inspector"
- Press that key


It should be clear what both "top" and "under" means in that case

[jamessimo]

neitsa thanks so much for this, really set me on the right track, I am trying to inflate the compressed byte array with Javascript.

I will keep you posted!

[jamessimo]

BOOM! Got a very basic map viewer in my web browser! Thanks again @neitsa for the help.

https://imgur.com/a/wBgUY

I got a gZIP compression algo to inflate the Base64, its not perfect as it seems to over inflate the map along the x axis and my maps were 250 x 500 instead of 250 by 250. Not sure but I think its the way Javascript creates byteArrays, A cheap hack was just to ignore every other byte when rendering the x axis. I need to map the colors to real terrain and configure the tile map.
Then I will use a html5 game engine (phaser) to make this a real tile map that you can zoom into and pan around (right now there just <div>'s with guessed colors.

I have also gotten all the save data into a easy to read JSON format so I will be adding some detailed info about the tiles and pawns

[attachment deleted by admin: too old]