[LIB] Harmony v1.2.0.1

Started by Brrainz, January 13, 2017, 04:59:21 PM

Previous topic - Next topic

Brrainz

Alright,

thanks again RawCode for the original idea to use mono_code_manager_* but I had to add all those *nix dll refs in DllImport to make it fly even on other OSes. I tested with two guys on discord and I am now quite confident that this should work on 32/64 pc, mac & linux.

Both repositories on GitHub have been updated so feel free to test again.

The remaining question for me is: now Harmony has a direct dependency on RimWorld dll paths (at least for the linux stuff). How can I make the DllImport annotations dynamic so they find their files correctly even for other Unity applications? Any takers?

Cheers,
Andreas Pardeike

scuba156

Hmm, I still can't get the Camara+ example to run. It causes an exception when activating the mod.

Non platform assembly: data-1F5EAF10 (this message is harmless)
Fallback handler could not load library C:/Program Files (x86)/Steam/steamapps/common/RimWorld/RimWorldWin_Data/Mono/data-1F5EAF10.dll
Non platform assembly: data-1F5FAD48 (this message is harmless)
Fallback handler could not load library C:/Program Files (x86)/Steam/steamapps/common/RimWorld/RimWorldWin_Data/Mono/data-1F5FAD48.dll
Caught exception while loading play data but there are active mods other than Core. Resetting mods config and trying again.
The exception was: System.TypeLoadException: Could not load type 'Harmony.HarmonyPatch' from assembly 'CameraPlus'.
  at (wrapper managed-to-native) System.MonoCustomAttrs:GetCustomAttributesInternal (System.Reflection.ICustomAttributeProvider,System.Type,bool)
  at System.MonoCustomAttrs.GetCustomAttributesBase (ICustomAttributeProvider obj, System.Type attributeType) [0x00000] in <filename unknown>:0
  at System.MonoCustomAttrs.GetCustomAttributes (ICustomAttributeProvider obj, System.Type attributeType, Boolean inherit) [0x00000] in <filename unknown>:0
  at System.MonoType.GetCustomAttributes (System.Type attributeType, Boolean inherit) [0x00000] in <filename unknown>:0
  at Verse.GenAttribute.TryGetAttribute[DefOf] (System.Reflection.MemberInfo memberInfo, RimWorld.DefOf& customAttribute) [0x00000] in <filename unknown>:0
  at Verse.GenAttribute.HasAttribute[DefOf] (System.Reflection.MemberInfo memberInfo) [0x00000] in <filename unknown>:0
  at Verse.GenTypes.<AllTypesWithAttribute`1>m__A0E[DefOf] (System.Type x) [0x00000] in <filename unknown>:0
  at System.Linq.Enumerable+<CreateWhereIterator>c__Iterator1D`1[System.Type].MoveNext () [0x00000] in <filename unknown>:0
  at RimWorld.DefOfHelper.RebindAllDefOfs (Boolean earlyTryMode) [0x00000] in <filename unknown>:0
  at Verse.PlayDataLoader.DoPlayLoad () [0x00000] in <filename unknown>:0
  at Verse.PlayDataLoader.LoadAllPlayData (Boolean recovering) [0x00000] in <filename unknown>:0

RawCode

__internal tag is magical and should always work on any platform and any OS, it's reference to mono itself.
probably there is some kind of trick related to name mangling or something, sadly i have no resources to test OS other then windows 7-64.
i will check source of mono runtime once again for possible solution.

issue of Camera+ related to modload order, Harmony MUST be second mod, right after Core, all mods that use anything from Harmony must load after it.
If mod try to use classes not yet exists in class pool, loading fail and current implementation of classloader is unable to handle situation properly.

scuba156

Quote from: RawCode on January 16, 2017, 12:14:26 AM
issue of Camera+ related to modload order, Harmony MUST be second mod, right after Core, all mods that use anything from Harmony must load after it.
If mod try to use classes not yet exists in class pool, loading fail and current implementation of classloader is unable to handle situation properly.
Current Mod order is Core > Harmony > CamaraPlus, and it still gives an exception. Using Win 10 x64 Insider Preview 14986

Brrainz

Sorry for the confusion, it was late and I was tired and got too excited. Anyway, I'll clean up things asap and test more. So my thoughts:

Regarding the magical __Internal - it does not work out of the box. Not on my mac and not on linux. Would be cool to get it work though.

But: new idea, and I think this should work great: why not make a new dummy method with enough "content" so the resulting JIT code is large enough to contain those bytes that I need as an in-between area to store the pointer to the shared information and the jump to the final method. It would work on any platform and it does not need any native calls. I tried to implement it yesterday and it worked first in the prototype but then crashed unrelated when doing anything in RW and I was desperate for sleep.

Brrainz

Ok, quick update (about to get to work). I suspected the GC to get into my way, so I changed the memory allocation from what is in on GitHub to:

      static DynamicMethod dynamicMethod; // WARNING, test only. Works only with 1 patch
      static Delegate theDelegate; // WARNING, test only. Works only with 1 patch
      delegate void Dummy();
      public static long GetMemory(int size)
      {
         dynamicMethod = new DynamicMethod("", typeof(void), new Type[] { });
         var il = dynamicMethod.GetILGenerator();
         il.DeclareLocal(typeof(int));
         il.Emit(OpCodes.Ldc_I4, 0);
         for (int i = 1; i <= 64; i++)
         {
            il.Emit(OpCodes.Stloc_0);
            il.Emit(OpCodes.Ldloc_0);
            il.Emit(OpCodes.Ldc_I4, i);
            il.Emit(OpCodes.Add);
         }
         il.Emit(OpCodes.Stloc_0);
         il.Emit(OpCodes.Ret);
         theDelegate = dynamicMethod.CreateDelegate(typeof(Dummy));
         return theDelegate.Method.MethodHandle.GetFunctionPointer().ToInt64();
      }

and it works (on my Mac where it prev. did crash too). But when I quit RW, I get this nasty stack trace:

Thread 0 Crashed:: MainThrd  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib           0x00007fffaf7dbdd6 __pthread_kill + 10
1   libsystem_pthread.dylib          0x00007fffaf8c7787 pthread_kill + 90
2   libsystem_c.dylib                0x00007fffaf741420 abort + 129
3   libmono.0.dylib                  0x00000001029e6244 dlfree + 3045
4   libmono.0.dylib                  0x00000001029e3c1c free_chunklist + 79
5   libmono.0.dylib                  0x00000001029e3bbe mono_code_manager_destroy + 34
6   libmono.0.dylib                  0x000000010285abb1 dynamic_method_info_free + 25
7   libmono.0.dylib                  0x00000001029ee2a6 g_hash_table_foreach + 81
8   libmono.0.dylib                  0x0000000102859890 mini_free_jit_domain_info + 101
9   libmono.0.dylib                  0x0000000102937ef4 mono_domain_free + 574
10  libmono.0.dylib                  0x000000010285a615 mini_cleanup + 1369
11  libmono.0.dylib                  0x00000001028a70de mono_jit_cleanup + 13
12  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x0000000100c8fd09 CleanupMono() + 57
13  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x0000000100c25daf PlayerCleanup(bool) + 143
14  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x000000010135c883 DoQuit(bool) + 19
15  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x000000010135f94d -[PlayerAppDelegate applicationShouldTerminate:] + 45
16  com.apple.AppKit                 0x00007fff97e8e420 -[NSApplication _docController:shouldTerminate:] + 71
17  com.apple.AppKit                 0x00007fff97e8e2d4 __91-[NSDocumentController(NSInternal) _closeAllDocumentsWithDelegate:shouldTerminateSelector:]_block_invoke + 567
18  com.apple.AppKit                 0x00007fff97e8def3 -[NSDocumentController(NSInternal) _closeAllDocumentsWithDelegate:shouldTerminateSelector:] + 1318
19  com.apple.AppKit                 0x00007fff97e8d9a6 -[NSDocumentController(NSInternal) __closeAllDocumentsWithDelegate:shouldTerminateSelector:] + 307
20  com.apple.AppKit                 0x00007fff97e8d5f7 -[NSApplication _shouldTerminate] + 843
21  com.apple.AppKit                 0x00007fff97e8cc96 -[NSApplication terminate:] + 773
22  libsystem_trace.dylib            0x00007fffaf8dfc3d _os_activity_initiate + 61
23  com.apple.AppKit                 0x00007fff98408c9c -[NSApplication(NSResponder) sendAction:to:from:] + 456
24  com.apple.AppKit                 0x00007fff97edce06 -[NSMenuItem _corePerformAction] + 324
25  com.apple.AppKit                 0x00007fff97edcb72 -[NSCarbonMenuImpl performActionWithHighlightingForItemAtIndex:] + 114
26  libsystem_trace.dylib            0x00007fffaf8dfc3d _os_activity_initiate + 61
27  com.apple.AppKit                 0x00007fff97edb985 -[NSMenu performKeyEquivalent:] + 367
28  com.apple.AppKit                 0x00007fff98407897 routeKeyEquivalent + 1024
29  com.apple.AppKit                 0x00007fff984054ad -[NSApplication(NSEvent) sendEvent:] + 3377
30  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x000000010135d72e -[PlayerApplication sendEvent:] + 190
31  com.apple.AppKit                 0x00007fff97c80f81 -[NSApplication run] + 1002
32  com.apple.AppKit                 0x00007fff97c4b850 NSApplicationMain + 1237
33  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x000000010135d9fe PlayerMain(int, char const**) + 654
34  unity.Ludeon Studios.RimWorld by Ludeon Studios   0x0000000100002034 start + 52

scuba156

I have to be doing something wrong. Im getting my last crash on OSX sierra, even with the patch.

Brrainz

Quote from: scuba156 on January 16, 2017, 03:50:42 AM
I have to be doing something wrong. Im getting my last crash on OSX sierra, even with the patch.
All I did was quickly fetch the latest version of Harmony from GitHub (did some changes last night) and replaced the method above. Compiled in Debug and added dll to Camera+, compiled Debug, made sure both dlls were correct and tested in RimWorld from steam. Did work to launch a map and crashed when I quit RW.

scuba156

So recompiling the Camara+ example with the new dll works, just replacing the harmony dll with the updated one does not. I would not of thought that it would need recompiling when it's referencing an external lib.

So now I'm up to date, it crashes after map gen and on exit too. Here's the stack trace from mine:


========== OUTPUTING STACK TRACE ==================

0x076A8B5C (mono) unity_mono_method_is_generic
0x076A8B99 (mono) unity_mono_method_is_generic
0x0768983D (mono) mono_debugger_run_finally
0x0768AA15 (mono) mono_debugger_run_finally
  ERROR: SymGetSymFromAddr64, GetLastError: 'The specified module could not be found.' (Address: 077D0066)
  ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 077D0066)
0x077D0066 ((<unknown>))
0x0438FF12 (Mono JIT Code) Verse.CameraDriver:Awake ()
0x07DDF541 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
0x07691440 (mono) mono_set_defaults
0x075FD7EE (mono) mono_runtime_invoke
0x003BD2DE (RimWorldWin) scripting_gchandle_get_target
0x0044200A (RimWorldWin) ScriptingArguments::AddString
0x00319C3E (RimWorldWin) RenderSettings::GetAmbientProbe
0x0044226C (RimWorldWin) ScriptingArguments::AddString
0x003ADF2F (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003AE496 (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003B0478 (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003AE8BC (RimWorldWin) IClusterRenderer::IClusterRenderer
0x0045AD82 (RimWorldWin) RegisterAllowNameConversionInDerivedTypes
0x0045AF6D (RimWorldWin) RegisterAllowNameConversionInDerivedTypes
0x004405D6 (RimWorldWin) CallbackArray3<int const ,AwakeFromLoadQueue &,enum RuntimeSceneManager::LoadingMode>::Invoke
0x00440786 (RimWorldWin) CallbackArray3<int const ,AwakeFromLoadQueue &,enum RuntimeSceneManager::LoadingMode>::Invoke
0x0043F60A (RimWorldWin) AnimationCurveTpl<Quaternionf>::GetTypeString
0x0041A57F (RimWorldWin) RectT<int>::Contains
0x004B0EAB (RimWorldWin) Append
0x004B360E (RimWorldWin) PlayerWinMain
0x008AC8D8 (RimWorldWin) RectT<int>::GetBottom
0x008DBD88 (RimWorldWin) RectT<int>::GetBottom
0x76FE8694 (KERNEL32) BaseThreadInitThunk
0x771004BA (ntdll) ApiSetQueryApiSetPresence

========== END OF STACKTRACE ===========

Brrainz

Yes, sometimes the lib does not get copied over and you need to have it updated to have your code reference the correct version (simply replacing it does not cut it).

Good news, I replaced DynamicMethod with MethodBuilder and it seems to work fine. Just tested on my Mac so we need people to jump in and test it on Win and Linux. Tonight I can test on my Windows machine.

The git repo has no release yet but is updated to v1.0.3 so you need to build it yourself for now. I think this is good progress and I feel confident that we are getting close. A lot of nasty code has disappeared - usually a good sign :-)

Cheers
Andreas Pardeike

EDIT 1: sorry I made a small change to the IL code generator. My bad, I undo it right away....
EDIT 2: fixed

scuba156

I'll test on Windows as soon as you post the new commit, I just tried it and an IL issue. Ready to recompile when its done.

EDIT: RimWorld starts fine, cant start a new game or load a game without crashing. Can exit without crashing though.


========== OUTPUTING STACK TRACE ==================

0x7682E512 (KERNELBASE) RaiseException
0x074310DD (mono) set_vprintf_func
0x074310F6 (mono) set_vprintf_func
0x0744E4D9 (mono) mono_class_from_mono_type
0x0751BFB0 (mono) mono_print_method_from_ip
0x0751C1CB (mono) mono_print_method_from_ip
0x0751DFC4 (mono) mono_jit_thread_attach
0x0751FDC1 (mono) mono_set_defaults
0x07520C2A (mono) mono_set_defaults
0x07521185 (mono) mono_set_defaults
0x075211F2 (mono) mono_set_defaults
0x0751ACA1 (mono) mono_debugger_run_finally
  ERROR: SymGetSymFromAddr64, GetLastError: 'The specified module could not be found.' (Address: 077F0066)
  ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 077F0066)
0x077F0066 ((<unknown>))
0x1E9AA5B2 (Mono JIT Code) Verse.CameraDriver:Awake ()
0x07D6F541 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
0x07521440 (mono) mono_set_defaults
0x0748D7EE (mono) mono_runtime_invoke
0x003BD2DE (RimWorldWin) scripting_gchandle_get_target
0x0044200A (RimWorldWin) ScriptingArguments::AddString
0x00319C3E (RimWorldWin) RenderSettings::GetAmbientProbe
0x0044226C (RimWorldWin) ScriptingArguments::AddString
0x003ADF2F (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003AE496 (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003B0478 (RimWorldWin) IClusterRenderer::IClusterRenderer
0x003AE8BC (RimWorldWin) IClusterRenderer::IClusterRenderer
0x0045AD82 (RimWorldWin) RegisterAllowNameConversionInDerivedTypes
0x0045AF6D (RimWorldWin) RegisterAllowNameConversionInDerivedTypes
0x004405D6 (RimWorldWin) CallbackArray3<int const ,AwakeFromLoadQueue &,enum RuntimeSceneManager::LoadingMode>::Invoke
0x00440786 (RimWorldWin) CallbackArray3<int const ,AwakeFromLoadQueue &,enum RuntimeSceneManager::LoadingMode>::Invoke
0x0043F60A (RimWorldWin) AnimationCurveTpl<Quaternionf>::GetTypeString
0x0041A57F (RimWorldWin) RectT<int>::Contains
0x004B0EAB (RimWorldWin) Append
0x004B360E (RimWorldWin) PlayerWinMain
0x008AC8D8 (RimWorldWin) RectT<int>::GetBottom
0x008DBD88 (RimWorldWin) RectT<int>::GetBottom
0x76FE8694 (KERNEL32) BaseThreadInitThunk
0x771004BA (ntdll) ApiSetQueryApiSetPresence

========== END OF STACKTRACE ===========

RawCode

mono_lookup_pinvoke_call
handle loading of native function.
nasty trick exists, i managed to detect issue during research of "latebind" native methods

if you execute method, that have references to unbound native method, it will crash\fail ever if you managed to fix method before process reach actual call.
reason is simple - JIT already knows that method is invalid and emit unconditional exception throw.
if you store all natives inside wrapper with "no inline" option, everything works properly, also you can modify wrapper at will.

Most simple and effective way to force JIT to allocate large memory section for method, is exception throws and exception handling.
JIT emit large blocks for each exception section.

Brrainz

Quote from: scuba156 on January 16, 2017, 06:54:39 AM
I'll test on Windows as soon as you post the new commit, I just tried it and an IL issue. Ready to recompile when its done.

EDIT: RimWorld starts fine, cant start a new game or load a game without crashing. Can exit without crashing though.


ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 077F0066)

This very much looks like your camera code not correctly bound to the right version of the Harmony lib. It isn't related to the bug we discuss right now. I suggest you double check that your project refers to the correct version of the lib and that it in fact is named 0harmony in the Assemblies folder to load it first. Too bad you are on Mac, otherwise I would recommend to use ILMerge to join the two dlls into a single one. That would definitely get rid of that error.

Brrainz

Quote from: RawCode on January 16, 2017, 08:15:55 AM
handle loading of native function.
nasty trick exists, i managed to detect issue during research of "latebind" native methods

if you execute method, that have references to unbound native method, it will crash\fail ever if you managed to fix method before process reach actual call.
reason is simple - JIT already knows that method is invalid and emit unconditional exception throw.
if you store all natives inside wrapper with "no inline" option, everything works properly, also you can modify wrapper at will.

Most simple and effective way to force JIT to allocate large memory section for method, is exception throws and exception handling.
JIT emit large blocks for each exception section.
I must admit that I am having a hard time to follow you, Raw. Especially since I don't have any invalid methods anywhere. Are we actually talking about the same topic? Maybe I failed to explain what I am actually doing here. In that case I am really sorry.

scuba156

Quote from: pardeike on January 16, 2017, 08:56:16 AM
This very much looks like your camera code not correctly bound to the right version of the Harmony lib. It isn't related to the bug we discuss right now. I suggest you double check that your project refers to the correct version of the lib and that it in fact is named 0harmony in the Assemblies folder to load it first. Too bad you are on Mac, otherwise I would recommend to use ILMerge to join the two dlls into a single one. That would definitely get rid of that error.
I've double and triple checked my refrences and everything appears to be fine. I've cleaned and rebuilt multiple times. I'm using Windows as my main machine, used ILMerge to make a single dll but it still caused an exception.

Even using the updated 1.0.3 CamaraPlus dll's from your github cause the same exception.