[LIB] Harmony v1.2.0.1

Started by Brrainz, January 13, 2017, 04:59:21 PM

Previous topic - Next topic

Brrainz

HARMONY 1.2.0.1
the right way of patching code at runtime

Hi,

Harmony is a simple dll that you embed in your mods. It will allow you to add code to any method inside RimWorld or other mods. It even works if more than one mod patches a method and it is not a mod itself which means that your users don't have to install it.

You can find out more about it on GitHub. I made it completely open source and hope that we all together can continue to evolve it. It comes with some wiki documentation and there are many mods using it on GitHub too (HugsLib has it build in i.e.). It works on PC, Mac and Linux.

The github repository has a ready made version of the lib in dll form and you can either add it to your Assemblies folder (add reference and keep "Copy" on) or merge it with a tool like ILMerge into your own dll.
Please feel free to give it a try and come back with feedback.

Harmony on nuget: Lib.Harmony

Harmony on GitHub:
https://github.com/pardeike/Harmony/releases/tag/v1.2.0.1
https://github.com/pardeike/Harmony/wiki

Example 1:Camera+ on GitHub:
https://github.com/pardeike/CameraPlus

Example 2:SameSpot on GitHub:
https://github.com/pardeike/SameSpot

Quick starter snippet including a patch to load a specific game on start:
https://gist.github.com/pardeike/12c457e135a42a28e068f2aba5337221

Please also help spreading the word so we all finally get to a point where we don't sabotage each other with detours that create dependencies and force the user to load mods in certain orders (Harmony defines priorities and dependencies internally). I am available for any questions in this thread. We are about to incorporate this into all the major mods and general libs like HugsLib. I will personally try to advertise Harmony in the Unity forums because as it is now, Harmony isn't RimWorld specific.

Thanks for your feedback and support.

Cheers,
Andreas Pardeike

RawCode

Very good, i will check library and contribute if i can.

It will be good place to leave reference to CCL and HugsLib that ultimately have same goal:
https://ludeon.com/forums/index.php?topic=16599.0
https://ludeon.com/forums/index.php?topic=28066.0

reference to BuildProductive code already present.

Brrainz

Thnx RawCode,

good idea.
/Andreas

PS: a friend of mine had a access violation on his AMD 64bit running Win10 yesterday. He has the same problem with the Achtung mod which means that the AsmHelper might have a problem. I have tested on my MacBook and on an intel 64bit machine without problem.

RawCode

there are number of issues with different platforms, first of all, portable and proper way to allocate RWX memory, i had suggested this before, i don't know why this was ignored, still, second try:

[DllImport("__Internal")]
static extern private unsafe void* mono_code_manager_new();

//this method will allocate memory with RWX permissions
//since you have exclusive access to that memory, no reasons to follow commit procedure
[DllImport("__Internal")]
static extern private unsafe void* mono_code_manager_reserve(void* MonoCodeManager, int size);

static private void* u_MonoCodeManager;
static private void* u_RWXM;

static CodeManipulationUtils()
{
u_MonoCodeManager = mono_code_manager_new ();
u_RWXM = mono_code_manager_reserve (u_MonoCodeManager, sizeof(void*)*256);

Console.WriteLine ("RWX memory allocation " + (int)u_RWXM);
}

this is same routine (but with commit procedure omitted) used internally by runtime itself, it's not any better then current implementation, but at least portable without much additional code.

Fluffy (l2032)

Looks incredibly promising!

Also looks very complicated, which means it's going to require extremely good documentation and a lot of testing. I'm definitely keeping an eye on this, but before I'd like to be able to at least get an idea of what's happening in the lib before I start using it. The current documentation doesn't really help there.

Brrainz

Sure. I totally understand your point of view Fluffy. I have a lot of internal documentation that probably is tl;dr for many people so I have not put any effort into preparing them.

The overall concept evolved around at least 10 different approaches which, by the nature of the problem, did not pan out in some way. At one point, I even added my own csharp compiler and had dynamic source code in the project. In the end, the current state is the result of lots of lessons learned.

So until I put up more details, I don't expect anybody to put anything into a something ;-)

But since the whole point is to build trust with the community, the only way forward is to put it out in the open and lab test it and gain insides. Raw already gave me a nice replacement for the memory allocation that I am about to test with my friend.

So for now, let me sketch the basic idea behind Harmony: Instead of having global state, I use the method that is to be patched (I call it the target method) to attach information. So I allocate memory, add a jump from the jit code to that area (just like the old Detour). There, I have a new jump to the replacement method (I call it a wrapper). In that memory area, I prefix the jump with a pointer to a byte array that deserializes to the patch information. Since every mod has its own class hierarchy this is necessary. The first one to patch a targe method allocates this, everyone else that comes later simply reads the patch information, adds to it and replaces the patch information with a new version.

This solves the global state problem. Now for the wrapper method. Since I want to keep the original intact, I copy the IL codes of the target method to a copy. I then construct the wrapper to call all prefix methods, then the copy, then the postfix methods. Every time a prefix or postfix is added, the patch information is rewritten and a new wrapper is created.

All the rest is just support stuff that is necessary to make this easy for the user. The critical and "unsafe" parts here are:

- memory allocation and basic assembler handling for the initial redirect

- creating a copy of the original in CIL

- constructing a wrapper in CIL that has the same signature as the original

Since I don't rely on subclassing and instead juggle arguments around, the whole thing becomes much more safe. It was a bit tricky to get the initial order of things working but once it works it works pretty flawless. Right now, it works with any kind of method unless inlined - I even got it to work with some anonymous subclass method where RimWorld originally defines a anonymous method to be used as a Predicate. Works great. Properties work too, just patch the "g(s)et_Property" method.

So what's on the todo list:

- I want this to work for constructors

- I want to add a replacer where you not only have prefix/postfix methods but also a way to replace methodbase A with B in the original method (since all IL codes are copied this is really trivial as long as A and B have the same call signature). Here, multiple mods patching could register their replacers which would work in a chain:

original calls x.foo() and foo gets passed to replacer1, replacer2 replacer3 who all have their chance to replace it with something else and in the end x.bar() is written. This opens up great ways to instantiate subclasses instead of the original inside the target method. If one would like to go crazy, a general il code processor could be added thus allowing you to modify the original in any crazy way. Of course, totally overkill and quite dangerous.

And hopefully you guys come up with other needed stuff to implement.
/Andreas

Brrainz

Apropos testing. I wrote unit tests for everything I could. The problem with VS is that unit tests run in at least .Net 3.5 which screws things up - you're basically not testing the dynamic method things in the environment of the original Unity application. So if someone comes up with a good way to run those tests properly, I will make the test coverage 100% for everything.

RawCode

"C:\Program Files (x86)\Mono\bin\mono" --debug "D:\User\Desktop\MonoRuntimeTest\bin\Release\MonoRuntimeTest.dll"

this "command line" used by me to execute code with specific mono runtime, in addition, mono.dll and mscorlib.dll are replaced by files provided with game, this allows to test code with "same runtime" without running game itself.
No matter how hard VS try, you always will run properly by similar command line.
OFC you will need mono version 2 installed and files swapped.

as for constructors, first runtime calls allocator, that reserve memory and setup object vtable and then calls constructor as non virtual instance method.

Constructors created by "normal" compiler always calls base class constructors, it's "CLI" feature, still you can emit code that violate this.
Constructors not any special by itself.

Fluffy (l2032)

ok, that sounds good.

Method replacement would definitely be needed, both because it's required is some cases, and because it would make for a drop-in replacement.

I'm not entirely sure allowing multiple mods to replace the same method is a great idea, considering they are quite likely to both call base behaviour that would then be duplicated. But that's an implementation detail.

I'm very much looking forward to being able to attach what are essentially events to whatever methods I want - that opens up a world of possibilities (and sooooo many potential obscure bugs that it scares me, but with great power comes great responsibility I suppose :P)

Brrainz

I don't see the difference between acting on an event and i.e. tail patching a method that runs during that event.

If you see a method you patch as a node in a call graph, you can always choose a different level to add you head or tail patch so you never need to call any base methods. In the end it is up to the responsibility of the person writing the code to not leave any side effects. If your code has desired side effects and another mod has overlapping features it is always difficult to solve the conflict.

Still, not replicating the whole methods code to change a small area is a huge step forward and a structured way to learn about other mods patching the area you self want to change will improve things too.

Event based code has its downsides too. For this, a common lib like hugslib is probably the best solution.

Brrainz

And btw: a Harmony prefix can be defined with very high priority and then return false - effectively canceling the original method or any other prefixes that otherwise would run. Postpatches run always but I am open for discussion on this.

RawCode

It's about "number of features" vs "ease of use".
"singular method replacements" vs "lack of conflicts".

priority settings is illusion of "conflictless", we can make any number of priority settings, still, as soon as two mods try to register top and singular injections, one of them (or both) will fail.

Brrainz

Good reflections Raw,
And I think the only possible solution is to increase transparency so that mod A and B get the tools to discover the conflict and then cooperate to solve it. Ultimately, two mods doing the same thing different will always collide.

Btw, the change to your suggested memory allocation just gave my friend a

KERNELBASE.dll caused an Access Violation (0xc0000005)
  in module KERNELBASE.dll at 0023:7464e512.


Not when patching but when the patch is then executed. :/
/Andreas

scuba156

Attempting to use the example Camara+ Gives me an kernel.dll access violation at the same address. Logs and dump attached

[attachment deleted by admin due to age]

Brrainz

I got some great help with the cross platform stuff and will put up a new release tonight that should work on PC, Mac and Linux in 32 and 64 bits. I'll post it here and in the official Tools thread.