[LIB] ModCheck: patch tests and error logging for mod loading

Started by Nightinggale, October 31, 2017, 06:57:04 PM

Previous topic - Next topic

kaptain_kavern

The file checking thing is clever ;-)

About Mod check distribution: maybe see with hugslib author to merge your mods? As it's the only one library nearly every mod users have.

larSyn

Quote from: Nightinggale on November 29, 2017, 11:41:51 PM
...
Do any of you have anything to say about this before I make up my mind on what to do?

I really like the sounds of faster load times.  I use a lot of patches and anything that speeds up the process is a good thing imo. 

I vote keep it all in one for the same reasons you state.  No need for more mod clutter. 

One question.  Why couldn't we include the Harmony dll in our mods with ModCheck?  I thought that was one of the good things about Harmony, in that users didn't need to download it.

Otherwise, just want to say great work on this so far.  You've made a very useful thing for people like me, and you've been super fast with fixes.  ModCheck is far easier to use than the new vanilla mod checking patch (and far superior), not to mention the other features it adds. :)

Nightinggale

Quote from: kaptain_kavern on November 30, 2017, 03:44:35 AMAbout Mod check distribution: maybe see with hugslib author to merge your mods? As it's the only one library nearly every mod users have.
Contact established and idea rejected. Both parties have concerns about the same problem: what if ModCheck has an update to release and HugsLib has none? There are other issues as well. I kind of like the thinking of an instant distribution it would include though, meaning it was a good proposal, which would need to be considered. I mean when faced with a problem with no clear best solution, all options should be considered.

Quote from: larSyn on December 01, 2017, 11:45:17 AMOne question.  Why couldn't we include the Harmony dll in our mods with ModCheck?  I thought that was one of the good things about Harmony, in that users didn't need to download it.
It could be done. However there is another problem regarding including DLL files. Say we have two mods called A and B. A has ModCheck 1.1 and B has v1.4. B use some of the tags introduced in v1.2. On startup, A is loaded first, then B. A loads ModCheck and when B loads, it detects ModCheck is already loaded and will not reload it. When patching, the patches from A works just fine. When B patches, it encounters a tag not mentioned in the DLL and it causes an error.

I simplified the problem and loading process, but essentially that's the problem. The ModCheck.dll from the first mod in the modlist is used, even if it isn't updated. I have posted in the Harmony thread about this because Harmony must be affected as well and hopefully a solution will be found. This is already a problem and it actually has nothing to do with adding Harmony, meaning a solution should be found regardless of the directions of development.

I have come up with a different idea though. I can make another mod consisting of one DLL file. A mod, which is intended to be perfect on release and will not have to be updated and as such should not suffer from the different version issue. This mod adds one PatchOperation. It works like .isModLoaded, as in it checks if a mod is loaded and it includes a cache to ensure it's only checked once for performance reasons. On failure, it will open a popup where it will display stuff, which can be set with tags in xml. This will include links to steam, direct download, button for enable mod, subscribe to steam (if I can figure out how to do that). Stuff like that. If each mod adds this, then people will be told to get a specific mod, in this case ModCheck and there will only be one ModCheck.dll. Add minimum required targetVersion and it will even catch the case where somebody use v1.1 as direct download and the mod expects v1.4.

I actually like this idea, perhaps even better than figuring out how to load the newest DLL (though that would be good too) because this can be used for multiple mods, not just ModCheck. Since most people use steam, adding the ability to make RimWorld itself subscribe to a steam ID and enable that mod will make adding more mod requirements less of an issue than it is today.
ModCheck - boost your patch loading times and include patchmods in your main mod.

Nightinggale

Turns out Harmony suffers from the same issue and I have now made a bug report for RimWorld, which will hopefully result in us eventually being able to load the newest DLL rather than what happens to be the first. For the time being, update your mod to the newest DLL even if all it does is adding a new feature (like a PatchOperation), which do don't plan to use.

I kind of discarded the idea of having another mod to tell if ModCheck is loaded. It quickly turned into a ModCheck clone  :-[
One interesting thing came out of it. If I write a modName in the new (now discarded) mod and that mod is on the HD, but not enabled, it will be enabled, but then it will call the "mods changed, needs to restart" window and calling it while patching can't be done because the GUI haven't started yet, which causes the screen to turn all black. However if I force quit and start again, it remembers that the required mod has been enabled and it will load it.

Now if I figure out how to make a stack of (failed) PatchOperations and somehow call it and can loop them when main menu has started, then things can get interesting with popup windows with options for what to do with missing mods etc. Calling in main menu should be doable with Harmony, but I haven't figured out how to create memory, which can be written to while patching and then read later.
ModCheck - boost your patch loading times and include patchmods in your main mod.

Nightinggale

#34
I have done some thinking and some research and I now have a plan for what to do now. This post will list the ideas, though I can't say which order they will arrive in or if any of them will be canceled, nor can I say if they will be in the next release or when that will be.

I decided to expand the target area of ModCheck from conditional patching depending on mods loaded/order to patching in general, mainly focused on improving mod interaction/compatiblity as well as reducing xml loading time. The main reason for this is that it make no sense to make multiple mods for different types of PatchOperations. I will however not change the name to reflect this because a new name will only cause unneeded confusion.

This post is partly to allow feedback, partly to remember my own brainstorming.

Use newest DLL
The current system with adding ModCheck.dll can continue. It will require 0Harmony.dll too in the future, meaning it will have to be added too. While Harmony can be compiled into ModCheck.dll, I won't because if it's two files, Harmony can be updated without resorting to a new ModCheck release.

Right now I have the idea of using Harmony to reload ModCheck from the newest DLL. However since it could be complex to do that with a mod, which use transpilar, it could end up as a feature, which writes an error to the log if multiple versions of ModCheck.dll exist and it's not the newest, which is first. It could possibly add a warning for each mod, which use an outdated version.

ModCheck is released as a standalone mod on steam where people are told to load it before Core. This makes it the first loaded ModCheck.dll and since that one is the first to update, the risk of using an outdated DLL is removed for the subscribers.

Faster file identifier while patching
All patches are applied to all files. Most of the time a patch is written to apply to just one specific file. This mean it's a complete waste of time to try to patch other files. Sadly trying to patch files where the patch will do nothing takes up the majority of the time spend on each patch.

To get around this, I plan to add a PatchOperation, which checks if the current file is of a certain name and is part of a certain mod. This check should be much faster than the xpath search, which is the current system for file identifier.

Profiling each patch
When enabling verbose logging, it writes how long time it takes to load each mod. This includes patching, but it doesn't single out the time spend on the patches in any useful manner.

I plan to add profiling to each patch and on verbose logging, print how much time is spend on each Patches/PatchOperation. This will allow identifying mods with poorly written patches as well as which mod they are located inside.

Autodetection of patch owner(mod) and current file/mod
Create an internal engine to supply information at runtime. This includes the name of the patch to supply the currently working patch, which is what is currently in yourMod, which will then no longer be needed. Also read the name of the xml file currently being patched and the name of the mod providing that file. Info like this can then be used by other features as well as logging.

Log PatchOperation
Add a new baseclass and move all logging actions into this one. Make other PatchOperations inherit this one for consistency. Expand logging options, like adding warning (yellow) and only write to log when verbose logging is on. Add parameters, meaning a custom string can contain stuff like (3) and it will then insert say name of the file currently being patched. Actual list of parameters is not decided, but preferably as much as possible.

Allow patch files to use this directly. Since it always passes the check, it will write customMessageSuccess.

Help getting missing mods
Store results of patch operation testing and recall once Main Menu is up. This allows creating popup windows about any issue that may have been encountered, complete with buttons and as such the game can take action on each problem if the user choose to do so. If I can figure out how to do it, it will include (but not limited to): subscribe to missing mod on steam, open download link to missing mod, enable missing mod if present, change mod loading order, pick mod to disable if conflicting.

Due to the planned design, it's entirely possible that the user might be met with multiple restarts, like first subscribing, then enable (might be together with subscribe, not sure), then change order. Not ideal, but it will allow a user to get a modlist, which meet the requirements set in xml without actually opening the Mod Menu and do manual work there.

Translation of log entries
If log entries are stored and then written to the log after languages and settings have been read, then it will be possible to translate the text before it's written. It will still be written before reaching the main menu and the user will not be able to tell the difference, other than possibly changed order of entries.

Clone
Make an xpath patch, which points to an entry and it will make a copy of that one except it will use a new defName provided by the patch. The PatchOperation will then contain a list of PatchOperations, which will work just like regular PatchOperations, except that the current element in the xml document will be the root of the clone. This makes xpath searches much faster.

The idea is that if you want to make say a new bed, you clone a bed from vanilla instead of making your own. You then patch to apply the differences you want. If the vanilla bed changes, your modded bed will update accordingly. If say somebody adds a new medical monitor of some sort and patches vanilla to use it, your clone will use it too.

TODO: figure out if this is a horrible idea. It introduce the issue of adding something new with a patch, which is ok in some cases, but could be problematic if it is something like a bed and some other mod wants to patch it. Maybe the same goal could be reached by assigning a parent in the def file.
ModCheck - boost your patch loading times and include patchmods in your main mod.

kaptain_kavern


Nightinggale

ModCheck 1.5 has been released. It adds support for ModSync, both adding ModSync.xml to itself and the new PatchOperation isModSyncVersion to test the version of other mods.
ModCheck - boost your patch loading times and include patchmods in your main mod.

historic_os

This is amazing, i love this and honestly surprised its not used more often.

1. the main post is kinda hard to read, maybe do some reformatting on it and consider adding real life use case examples in there
2. I'd rather have it is a library attached to my mods(like harmony) to avoid annoying issues such as players forgetting to download dependencies.
3. I don't think you should be worried about mods not having the latest version. as you said - its relatively stable now and if for some reason there is a bug, mods can push an update with the new library.
another take on updating the library for all mods could be just providing the user with a script, if a user is having an issue due to a mod loading an older version of the library, you can offer the library packed with a little bash script to be placed in the mods folder that will replace all instances of your library with the new version.

this is really cool, i'd download all compatibility patches you can find and investigate them, see how you could add more features to the state where they are all obsolete and hopefully this mod will become as widely used as harmony and hugslib.

Nightinggale

Quote from: historic_os on December 06, 2017, 06:38:06 PM1. the main post is kinda hard to read, maybe do some reformatting on it and consider adding real life use case examples in there
I know. I wrote it for the first release and then just added text with each new feature. My plan is to make the PatchOperation to quickly reject "wrong" files while patching, release that and then rewrite all the documentation, here, on the GitHub page and steam. In addition to that, the real documentation should be on the wiki on GitHub with a multi page system, which will allow getting an overview as well as clicking on each feature to get details on it.

Real life cases aren't included because they didn't exist when the text was written. My examples on GitHub are also not good, particularly not compared to what real life files would look like.

If I add my recommendations for vanilla patching, which I wrote in the help forum (patching performance pitfall sticky), then it will be the wiki to use to make efficient patches. I have had feedback telling that one mod reduced time usage by 80%, the proposed new template for A Dog Said patching exceeds a 90% reduction and that's even before adding tools from ModCheck to make it even faster.

Quote from: historic_os on December 06, 2017, 06:38:06 PM2. I'd rather have it is a library attached to my mods(like harmony) to avoid annoying issues such as players forgetting to download dependencies.
Same here. I released it as a DLL file and everybody ignored it. I then put it into a mod and released it on steam and then people noticed it. In fact it went even better than just getting publicity on the forum because now I know of at least one steam only mod, which use ModCheck.

As the documentation says, ModCheck works both as a standalone mod and as a DLL to add to your mod. As far as I'm aware, all mods using ModCheck has decided on including the DLL, which is also my recommendation.

Adding ModSync.xml serves multiple purposes. One is obviously to allow getting a notification when there is a new release. This is a nice feature if you have a mod with the DLL included. The second is to make it appear in the new list of active mods, the third is it would feel wrong not to add it when adding isModSyncVersion and last, all mods should support ModSync. We need it mentioned everywhere as a standard  ;)

Quote from: historic_os on December 06, 2017, 06:38:06 PM3. I don't think you should be worried about mods not having the latest version. as you said - its relatively stable now and if for some reason there is a bug, mods can push an update with the new library.
The problem is if mods fail to do that because the ModCheck being used is the first one loaded.

Quote from: historic_os on December 06, 2017, 06:38:06 PManother take on updating the library for all mods could be just providing the user with a script, if a user is having an issue due to a mod loading an older version of the library, you can offer the library packed with a little bash script to be placed in the mods folder that will replace all instances of your library with the new version.
I plan to add a warning if the first loaded is not the newest in the list. I could expand this to provide a warning for all outdated if verbose logging is on.

Quote from: historic_os on December 06, 2017, 06:38:06 PMthis is really cool, i'd download all compatibility patches you can find and investigate them, see how you could add more features to the state where they are all obsolete and hopefully this mod will become as widely used as harmony and hugslib.
To be honest I don't think I need to come up with new features to do that. All I need is to make the file filter PatchOperation and the profiling for each patch tool. Releasing those two will make people upset that mod X takes 30 times longer than mod Y despite them patching around the same amount of data. That will cause feedback to mod X to optimize the patch files.
ModCheck - boost your patch loading times and include patchmods in your main mod.

kaptain_kavern

Quote from: Nightinggale on December 06, 2017, 07:47:58 PM
As far as I'm aware, all mods using ModCheck has decided on including the DLL, which is also my recommendation.

I was about to ask that. Thanks.

I'm starting to play with your tool in order to get familiarized ATM.

So far it's great, and following your previous recommendation I was able to make patches that start to feel like they're quicker - I'm lacking a real way to measure it though :p

Thank you for all of that  ;)

Nightinggale

Quote from: kaptain_kavern on December 06, 2017, 08:06:41 PMSo far it's great, and following your previous recommendation I was able to make patches that start to feel like they're quicker - I'm lacking a real way to measure it though :p
Enable verbose logging and read the log. It will write how long it spend reading xml files and how long it takes to reach the xml files for each mod. Each of those includes patching. The interesting part is actually core. Even if you do not patch core, your xpath searches will still trigger, meaning you will slow down reading core. Since it contains so many files, I find it to be a good place to benchmark the slowdown of files, which you do not patch. Try without any patches, with the patch you test before optimizing and then with optimizing. Write down the numbers between each run and pay attention to the changes.

As I wrote earlier I plan to make verbose logging print how much time is spend on each patch as this is the most interesting number. However I haven't written the code to calculate this yet.
ModCheck - boost your patch loading times and include patchmods in your main mod.

kaptain_kavern

#41
Hey, thanks for the tip. It will be fair enough, it's more because I'm curious rather than because I have a very big patch  ::)



On a side note, have you thought about a little logo or a banner to make us, modders, help you make ModCheck noticed? Like ModSync Ninja have?

Not something you should prioritize IMHO, but I was thinking about that while reading you talking about getting the tool noticed.

If your not an artist at all, like myself  :P , just ask the community around in here. I often had textures for my mods freely given to me


Edit:

In the mean time I'm gonna use this :     8)

more info: https://shields.io/  - For changing color and style

Nightinggale

Quote from: kaptain_kavern on December 06, 2017, 08:46:40 PMOn a side note, have you thought about a little logo or a banner to make us, modders, help you make ModCheck noticed? Like ModSync Ninja have?
I have started to wonder ever since you wrote that, but I'm not really sure what it should look like. I will need somebody to draw something, but I don't know what I would like to get drawn in the first place.

Quote from: kaptain_kavern on December 06, 2017, 08:46:40 PMIn the mean time I'm gonna use this :     8)

more info: https://shields.io/  - For changing color and style
Now that looks interesting and it got me wondering. What if I host the image on GitHub instead of people using shields.io. That way when I release, I can put up some 1.6 image and then replace the 1.5 image with one, which use yellow. That way all mods using an now outdated version of ModCheck will automatically change color to highlight this fact. It's not like it would be extremely hard to make. The "image" file is actually a text file with this contents:

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="140" height="28">
  <g shape-rendering="crispEdges">
    <path fill="#555" d="M0 0h97v28H0z"/>
    <path fill="#007ec6" d="M97 0h43v28H97z"/>
  </g>
  <g fill="#fff" text-anchor="middle" font-family="DejaVu Sans,Verdana,Geneva,sans-serif" font-size="100">
    <text x="485" y="175" transform="scale(.1)" textLength="730">MODCHECK</text>
    <text x="1185" y="175" font-weight="bold" transform="scale(.1)" textLength="190">1.5</text>
  </g>
</svg>

It looks like I just need to change one fill RGB and the text saying 1.5 and I have a new version. It's plain text and small enough to not cause issues if I add it to git. Even if I have to generate a new one with new x,y data, it's still no big deal to add it to git.

I could make it change to red if it's an important update where everybody should update right away and yellow if it's ok to wait until you update your mod anyway.
ModCheck - boost your patch loading times and include patchmods in your main mod.

kaptain_kavern


Nightinggale

I had a major breakthrough. It turned out that there is a timing issue as the approach I used in BoneMod starts Harmony after xml files have been loaded and patched. I managed to move the start to after patches have been loaded, but just before defs are loaded and patched, which is perfect. Right now it's only used to write mod names to the log whenever a new mod starts to load def files. Useless except it tells me that I can now access all the needed data without timing issues. I can now move on to do what I actually planned to do when I encountered the timing issue.

Messing with all the load stuff made me wonder. Now that I have a method being called in C# before patching, maybe I can use it to reflect all PatchOperation calls to the newest ModCheck.dll, which would then make ModCheck use the newest version instead of the first, hence reducing the issue where mods with outdated dll files prevent others from using updated functionality.

I have been wondering about writing to the log. I think I will redesign it completely and instead of having a success and a failure message, I will make it display the default error if errorOnFail is set. For custom log entries, I will add a list of strings. For each it should have settings like normal/warning/error, success true/false, onlyWithVerboseLogging and so on. Since it's a list, multiple strings can be added and it should support writing multiple strings to the log if needed. This should be usable with a log writing operation (always true) as well as with any other PatchOperation in ModCheck.

If anybody has any requests or comments regarding log writing (what or how to write), it would be nice to know before I get to do the actual redesigning.
ModCheck - boost your patch loading times and include patchmods in your main mod.