[1.0] RWMS -- RimWorld ModSorter .. RimWorld mod sorter (Windows, Linux, OSX)

[ThiIsMe007]

Yeah, I was thinking that requirements should probably override categories, since it's always best to load the mods as high as possible in the load order after respecting dependencies. For example :

Mod A < Mod B gives

Mod C
Mod A
Mod B
Mod D

not

Mod C
Mod A
Mod D
Mod B

even if categorization doesn't agree.

This however supposes that some kind of reliable system exists to identify these dependencies, and if mod authors don't fill them in correctly in whatever tool, someone else will probably have to.

FWIW, I found this page about mod categorizations, but I'm not sure that's still valid or even useful. It wasn't to me, so I have come up with my own categories, which I use to eliminate obvious redundancies between mods in the same category, with the prospect of having your tool do its thing later. I will just have to see how that works out this week-end.

EDIT : I forgot to mention that some kind of warning about incompatibilities would also be useful in the future. For example, don't load Expanded Crops with VGP. All in time though...

[Canute]

Yeah i think too.
These mod's should stick together. So long the dependency isn't a libary mod like hugslib.

[shakeyourbunny]

I actually did think about mod dependencies, but implementing support them is not only a technical challenge (somehow support both Fluffy's Mod Manager's Manifest.xml and Steam workshop dependencies?), but also a design challenge, particularly how mod dependencies should interact with the mod categorization sorting.

This is a planned feature for a 1.0 relase of RWMS, I agree that is a challenge (and this is one of the reasons why the 1.0 release will introduce an very different database format which is incompatbile with the current version, though it will use distinct filename to ease migration to the new coming version). It is possible to parse the mod XMLs, but they are many variations and by far not every mod has a complete set of data.

Another problem is that most mod authors don't bother providing manifests, as you noted. Sometimes they use workshop dependencies instead. Sometimes they just provide explicit guidance in the mod's description. Or sometimes by their very nature, one type of mod just has to load after another type of mod, hence the mod categorization scheme. But sometimes a suitable category doesn't exist.

I don't need to add anything here, would say the same, good explanation.

So rather than provide the ability to create an ad-hoc category, I added the ability to explicitly order mods in the same category: within the same category, mods that appear earlier in the list will load before mods the appear later. For this purpose, the local database's mods are de-facto prepended to the list of mods.

Thank you for your efforts, I'll review your pull request and will integrate it after looking through.

All modifications though, I still accept contributions on the Github tracker, so no worries

[shakeyourbunny]

Oh, so side not from my documentation https://github.com/shakeyourbunny/RWMS/blob/master/docs/documentation.md#license

Note that you are not allowed to take parts of the scripts for using them in your own mod sorting or mod manager scripts without written permission.

This also covers the json database and the categorization. If you wanna do an active fork you have to start from scratch and you are not allowed to do an "import" or something similar. And yes, making an offline copy of the json and then using that is importing. Accessing the database with your fork is also not permitted.

The categorization work was fun, but it also was an huge amount of work on my side and I also had to do a reboot myself (RWMS was originally a set of bugfixes where the fixes outgrow the original script until I came to the point of throwing everything away and did a clean-room reimplementation).

It is better (and encouraged by myself) if you have something useful to contribute and/or bugfixes, I am not shy to include them (and give proper credit where credit is due). They will be merged (or in case added to the database) after review.

Just don't be twitchy, I do have a day job and cannot watch the relevant forums 24/7.

[lbmaian]

shakeyourbunny doesn't like the idea of local databases and requested that I removed the feature. While I may disagree with the decision, I did effectively remove it for the version that was merged back to the master branch.

I'll still be maintaining my own private version of the script, since waiting days, let alone weeks, for correct database updates is not practical for me.

They also reverted the intra-category sorting change from de-facto workshop id sorting to database entry order. I'll be keeping that as well, since it's nice for ad-hoc mod reorderings.

If you want to know how to modify the script to have a local database, just PM me.

Oh, so side not from my documentation https://github.com/shakeyourbunny/RWMS/blob/master/docs/documentation.md#license

Note that you are not allowed to take parts of the scripts for using them in your own mod sorting or mod manager scripts without written permission.

This also covers the json database and the categorization. If you wanna do an active fork you have to start from scratch and you are not allowed to do an "import" or something similar. And yes, making an offline copy of the json and then using that is importing. Accessing the database with your fork is also not permitted.

I have to point out that for a private fork, this is legally as enforceable as a EULA that tries to prevent web scraping. At least in the US (not sure about Europe). The very fact that both the script and database are downloaded to the local user in plain text means it's fair use for the user to modify them, as long as they aren't distributed - and even if they were distributed, fair use can still apply due to the non-commercial nature of the product. That's something only a court can resolve on a case-by-case basis.

If you want me to, I can remove my public fork. I already removed the releases/tags/binaries.

[shakeyourbunny]

The very fact that both the script and database are downloaded to the local user in plain text

The JSON is never downloaded to disk only in the process memory of the script, same for the categorization.

If you want me to, I can remove my public fork. I already removed the releases/tags/binaries.

The main problem on your changes are that it relegates the online database to something which is only considered for updates and fallback instead of the canonical source for sorting. In addition, the scoring system you did is a rollback to the older times' of RWMS which originally had this, but I reworked this in the current form to avoid some really ugly problems on the maintainer's side.

The irony of all this that the very reason of slow updates is that besides working full-time for putting food on my table, there is a major rewrite in the works which anticipated all of your concerns of not letting do your own updates and not letting tinkering with the sort order. In this release it still has the current design choices, but there is some leeway for local modifications which are automatically integrated into the sort order process if present. The beautiful side of these local changes that if you do can also share them by uploading to the issue tracker.

If there are better ideas for speeding up the categorizing process and correcting them, please share your ideas.

I am very grateful for pouring your time and appreciate your motivation in all of this, but ultimately if I had not reacted in such a way, this all would be repeated history.

So I appreciate your offer for removing your public repository, but this decision is on your side.

[ThiIsMe007]

First, I'd like to thank Ibmaian again for producing the fork, which made the initial version actually usable (and useful to me) by fixing the bug involving unknown mods.

Next I'd like to say that I have the feeling that no one (in this thread at least) ever had the intention to "steal" anything from this project. On the contrary, we all contributed to it in different ways, be it either directly through coding and design, the brunt work, which credit rightfully belongs to shakeyourbunny, the tool author, be it through patching/forking, as Ibmaian has done, or be it, much more modestly, through testing and feedback, to validate usability cases.

It's the author's decision to know which direction the project should take, and everyone should respect that, but human nature being what it is, I believe there was room to word concerns (as legitimate as they might be) differently than it was done a couple of posts above.

Finally, I'd like to emphasize that as a player and user I am very grateful to whomever works on a project (tool, mod, art, etc.) and cares to share it with the community. It's in everyone's interest to see interesting projects like this one prosper, but to also protect the motivation of all the parties involved in the project, including the rights of the author but also the possibility to discuss and consider options, opinions or alternatives (like forks).

As a side-note, having experienced and contributed to similar sorting tools on different games with sometimes a much, much higher mod amount than Rimworld, the idea of a proprietary online-only database for sorting purposes is not a healthy design decision.

[lbmaian]

The JSON is never downloaded to disk only in the process memory of the script, same for the categorization.

It's memory either way, just of different durations, and that distinction doesn't really matter for private fair use. For example, consider the use case of a web browser that downloads a copyrighted webpage for the user to interact with. The browser may be either storing the webpage only in memory or also in a disk cache. Suppose further, that the user modifies the copyrighted webpage via browser web developer tools. As long as the user isn't financially gaining from this act or redistributing the modified webpage, this definitely falls under fair use.

The main problem on your changes are that it relegates the online database to something which is only considered for updates and fallback instead of the canonical source for sorting. In addition, the scoring system you did is a rollback to the older times' of RWMS which originally had this, but I reworked this in the current form to avoid some really ugly problems on the maintainer's side.

The irony of all this that the very reason of slow updates is that besides working full-time for putting food on my table, there is a major rewrite in the works which anticipated all of your concerns of not letting do your own updates and not letting tinkering with the sort order. In this release it still has the current design choices, but there is some leeway for local modifications which are automatically integrated into the sort order process if present. The beautiful side of these local changes that if you do can also share them by uploading to the issue tracker.

If there are better ideas for speeding up the categorizing process and correcting them, please share your ideas.

I'm glad you're working on allowing local changes that are stable through sorts, and a better process for syncing those local changes back to the official database.

However, I would urge you do at least SOME level of consistent ordering within categories in the meantime. Your current version has no technical guarantee that sorting is stable between script runs (or more likely, stable sorting between different machines/file-systems given the same mod list).

If there are better ideas for speeding up the categorizing process and correcting them, please share your ideas.

A couple ideas come to mind:

1) Assuming you do allow users to locally change mod categories (and any inter-category ordering) in the future, you should definitely include them in the unknown mod report (well, unknown + changed mods).

2) Assuming you do allow users to locally change inter-category mod order for mods, and if they don't correspond to the default ordering, include such mods with smart context. For instance, if the default order was A B C D E F G, and it was locally changed to A G D E B C F, the "order violating adjacent mods" are G D and E B, and thus those would be included in the report.

3) Including the full mod list, or perhaps if that impinges on user privacy too much, at least the diff between the full mod list and the result of the sort without local changes.

4) A server-side database that stores user-submitted local changes (both changed category, and the above "order violating adjacent mod" thing) and counts of such submitted changes. For the popular mods, you'd see higher counts for any changes regarding them, increasing the likelihood such changes are correct.

I am very grateful for pouring your time and appreciate your motivation in all of this, but ultimately if I had not reacted in such a way, this all would be repeated history.

So I appreciate your offer for removing your public repository, but this decision is on your side.

That sounds quite the interesting history with forking, or at least personal code/data control, you have there. Personally, GPL would be sufficient to ensure I was attributed and could see all changes to my code/data. But I can understand everyone has different standards on the level of personal control, and there have been cases of "abuse" of even GPL in the past (e.g. the KHTML/Webkit divergence way back then).

[shakeyourbunny]

First, I'd like to thank Ibmaian again for producing the fork, which made the initial version actually usable (and useful to me) by fixing the bug involving unknown mods.

This is also fixed now, and one of the main reasons why I wanted a merge back. Although the recent commits were a bit messy, but I wanted to give him proper credit.

I never took the credit for others people contribution and I won't take it. This is also the reason why RWMS always displays the highest contributors to the dataset on running it, having an extra switch for displaying more of them and if you want to look up who contributed what, you have always the option to dig through the Github commits for RWMSDB.

The latter is also the main reason why I insist on doing it through it, if someone has a better idea for quicker updates, a way for checking each mod and giving proper credit for the submitter is appreciated.

I'm glad you're working on allowing local changes that are stable through sorts, and a better process for syncing those local changes back to the official database.

However, I would urge you do at least SOME level of consistent ordering within categories in the meantime. Your current version has no technical guarantee that sorting is stable between script runs

This is true due to the category system, I agree that some finer control would be nice and there will be some sort of user definable override (not replacement!).

ad 1)
good idea, this relieves the user from double duty reporting

ad 2)
something in this line is planned, only problem is time on my side (have a longer weekend, so let's see, how far I can get).

Current status of the rewrite is that the formal description of the new data structure is finished more or less and I should re-run some scraping for it.

ad 3)
I was taught that you should avoid collecting data more than strictly necessary for processing and I strictly adhere to it. If somebody wants to share their changes, they should upload it to the tracker.

[/quote]

That sounds quite the interesting history with forking, or at least personal code/data control, you have there.

One of the lessons I learned that if you are burned and removed from contributing projects, you should not do the same, but the opposite: giving credit where credit is due.

As above said, the reason for the messy commits after the merge is that the attributions for your work remain visible.

[shakeyourbunny]

0.94.8 (2019-07-26):
new:
- build script for windows executable

changed:
- improved mod name cleaner
- 7zip support for build script (thanks Ibmaian)
- 7zip support improvements in build script
- re-added windows binary executable in release
- improved configuration dump, checks now validity of paths
- improved documentation
- pretty print ModsConfig.xml (thanks Ibmaian)

fixed:
- crash on empty ModsConfig.xml
- several small internal cosmetic and lint fixes (thanks Ibmaian)
- fix for older Python versions (thanks Rocketeer999)
- unknown, ACTIVE mods are now not removed anymore (thanks Ibmaian), you can enable / disable this behaviour with the dontremoveunknown switch in the configuration file, which already existed, but did not do the correct thing(tm). Default is keeping them.

For your convienence, download for the 0.98.4 RWMS windows executable is https://github.com/shakeyourbunny/RWMS/releases/download/0.98.4/rwms_sort-0.94.8-win.zip .

[lbmaian]

ad 3)
I was taught that you should avoid collecting data more than strictly necessary for processing and I strictly adhere to it. If somebody wants to share their changes, they should upload it to the tracker.

Sure that's a good principle, but from my long experience with maintaining and debugging high-throughput distributed applications, while it's useful for primary logs and data to be as succinct and noiseless as possible, it's useful to have extra context linkable to the primary data as needed. If your worry is about privacy, performance, or simply storage space, then I understand.

edit: Come to think of it, just printing out a diff to the user would be nice, to see what actually changed.

As above said, the reason for the messy commits after the merge is that the attributions for your work remain visible.

The merge and the resulting code is still rather messy and reverts a lot my fixes and changes (including some python best practices stuff) that were unrelated to local database or sorting at all.

I did bundle way too many changes in the pull request (since it was originally just supposed to be a private fork), so it's understandable if you didn't understand the reasoning behind every single change. And github doesn't make interactive merges (git merge --no-commit -no-ff, git reset, git add --interactive) on pull requests easy.

There are some regressions, one pretty serious, in the last release: https://github.com/shakeyourbunny/RWMS/issues/26

[shakeyourbunny]

0.98.9
bugfix release
fixed:
- show relative paths for local unknown mods in the unknown mods report
-    ensure unknown mod report and ModConfig.xml backup have the same timestamp in the name (in the very rare case that the minute changes during the script)

internal code fixes:

-   use tuples instead of lists for fixed sized objects
-    move unknown mod handling in load_mod_data outside to avoid side effects in it
-    use os.path.join in another place
-    remove unnecessary file.close() after with statements, since the with statements guarantee it's already called

Big thank you to Ibmaian (did the release except wrap-up and windows binary)
Windows binary: https://github.com/shakeyourbunny/RWMS/releases/download/0.98.9/rwms_sort-0.94.9-win.zip

[lbmaian]

To be clear, those were just the minor changes. The more important changes were:
* Fixed: The 7zip support in the exe build script is broken
* Fixed: wait_for_exit function doesn't actually exit the script, so e.g. --contributors still tries to update ModConfig.xml
* Ask for confirmation to write to ModConfig.xml when there are no unknown mods
* Pretty print --reset-to-core written ModConfig.xml
* Include UTF-8 BOM in ModConfig.xml (Rimworld-saved ModConfig.xml always has this, although Rimworld does still load ModConfig.xml without the BOM)
* Outputs total # active unknown mods, instead of # total unknown mods

[VoidRose]

Hi ya i might have some path issues and want to know if this is right?

; -------------------------------------------------------------------------------
; -- installation directories options --
; steam installation directory
[paths]
steamdir = /Users/name/Library/Application Support/Steam
; RimWorld main installation directory (DRM free build)
drmfreedir =
; configuration directory for RimWorld, where game saves etc reside
configdir = /Users/name/Library/Application Support/RimWorld/Saves
; location of the steam workshop mod files (full path, with AppID)
workshopdir = /Users/name/Library/Application Support/Steam/SteamApps/workshop/content/294100
; location of the local mods (full path, Mods/ directory in RimWorld game folder)
localmodsdir = /Users/name/Library/Application Support/Steam/SteamApps/common/RimWorld/RimWorldMac.app/Mods

Did i set up the path right? I get this env: python3\r: No such file or directory when i try to run rwms_sort.py

[shakeyourbunny]

Did i set up the path right? I get this env: python3\r: No such file or directory when i try to run rwms_sort.py

Did you really get the \r in the error line?