Ludeon Forums

RimWorld => Mods => Topic started by: Ataman on March 16, 2019, 05:17:16 PM

Title: If your metal turns into meat, here's why. (or: Hash Collisions)
Post by: Ataman on March 16, 2019, 05:17:16 PM
Hey there!


Here is one possible cause: Hash collisions on ThingDefs
(Disclaimer: This particular case only affects adding or updating mods in existing saves. New games are fine).

I couldn't find a lot on this matter with Google so I assume nobody has written something up for it yet. This post is in no way meant to criticize Tynan's programming, as far as I see it, Rimworld is one of the finest pieces in terms of programming I've ever seen. This post should be an informational piece for curious people, troubleshooters and modders which run into unexplainable things happening. (pun not intended)

What's happening?
In order to make save games smaller; Rimworld uses a little trick for certain Things on the map which haven't changed from their default state. Things like a piece of mineable steel for example. There is no reason to write a whole XML block into your save for something which is exactly the same as a completely fresh Thing. Instead, every ThingDef has a corresponding 16-bit unsigned integer attached to it and that gets stored instead. If a pawns starts mining that MineableSteel a little and runs off, it gets stored using a full XML block however.

This trick works pretty great, till it bites you in the ass. See, there are only 65'535 possible combinations so every ThingDef has a very small chance to collide with another ThingDef. In case of a collision Rimworld takes the number, increases it by one, and voilĂ , new hash.
It happens even in vanilla. Introducing Meat_Squirrel with an update resulted in the same hash as MineableSteel and suddenly every block of steel turned into little stacks of meat, didn't they, Tynan? ;D

So basically, the save game stores some Things like this: 27292, 27292, 27292, 27292, 27292.
When loading your save, it looks in its DefsDatabase what 27292 is and loads whatever comes up. For a while 27292 was MineableSteel but nobody expected the Meat_Squirrel inquisition and suddenly 27292 became Meat_Squirrel while the poor steel was left with 27293, cue alchemy happening. (Couldn't it at least turn into gold or something?)

Another Example
Genetic Rim (https://ludeon.com/forums/index.php?topic=35158.0) or rather its Megafauna Patch (https://steamcommunity.com/sharedfiles/filedetails/?id=1590579367) ran into the above mentioned scenario. It adds a completely unrelated ThingDef called GR_MegafaunaRodentIncubator which just so happens to produce the hash 27292. In turn, Meat_Squirrel now became 27293 and finally MineableSteel had to result in 27294. So all those little 27293 instances are no longer steel but meat instead.

What can you, as a modder, do?
Test. Test a lot. If you thought you tested enough, test again. Even then, it might still happen and you won't notice it till the comments come up asking why there is suddenly a lot of meat. Or worse, another mod just happens to collide with your mod and there isn't even a way to actually test for it. So, don't blame yourself when it happens to your mod, start playing the lottery instead!
Also watch out when adding new stuff with an update as people are not necessarily aware that something changed.

I'm thinking about writing a little tool which would test for this but it would be way better if it could fit into existing community tools like HugsLib or Fluffy's Mod Manager. If anyone wants to integrate this hit me up for details.

What can you, as a player, do?
The first solution is pretty obvious: Don't modify running games, start new ones instead.
However, if you really want to add a colliding mod, try making a local copy of it, figure out the colliding ThingDef and change its name. In my case I used the process of elimination. (i.e. I deleted half of the mod and if the problem went away, deleted another half of that, repeat as necessary).
Remember to update all references to that name in other Defs of the mod as well. Rimworld's dev console will tell you when you missed something.
The above mentioned tool could help in resolving these collisions, once it exists.

Alright, that's it. I hope there is enough context in here for search engines to pick up on it so some poor souls find this before losing their sanity to Cthulhu.

EDIT: I actually intended to post this into the modding help subforum and made a mistake, if a mod could move it there I'd be very thankful. :)

EDIT2: Wait... Would it be doable to save the current hash table inside the save so the game can reload it from there and continue the numbers? It would only minorly increase the size.
Title: Re: If your metal turns into meat, here's why. (or: Hash Collisions)
Post by: Mehni on March 17, 2019, 01:23:14 PM
I don't think testing all ~2000 mods on the Steam workshop is really a viable solution.

Re the second edit: Tacking on another digit to the short hash would help too.
Title: Re: If your metal turns into meat, here's why. (or: Hash Collisions)
Post by: Ataman on March 17, 2019, 04:52:01 PM
Of course it's not feasible to test all mods, but at least the major ones + vanilla could very easily be tested.
Also Fluffy's Mod Manager might be capable of creating and storing a local cache which gets checked for differences when loading the game, warning the player about conflicts in advance.

Tacking on another digit would only delay the inevitable. But maybe storing the hashes with the save file could be a viable solution. The save is already pretty big, another 200kb for storing the hash database wouldn't matter.

As I see it, adding new Defs to existing mods is inherently dangerous at the moment. Especially since Steam auto-updates mods without notice. Worst case scenario: Your mod causes a collision with some other popular mod and breaks a ton of saves. You'll eventually notice the reports but it might already be too late since removing the colliding Def now breaks the save of others who didn't have the issue and started playing normally. Lose-Lose scenario.