Ludeon Forums

RimWorld => Mods => Unfinished => Topic started by: FrodoOf9Fingers on October 09, 2017, 04:38:56 PM

Title: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 09, 2017, 04:38:56 PM
I figured that there should be a thread in the mods section for the work I'm attempting to do.

I'm currently working on making the main update loop of the game multi-threaded. Currently, every "thing" does it's "tick" on the same thread, I'm working on splitting the work into separate threads.

Goals:
Implement this using harmony as much as possible. Currently blocked on Harmony not working with reverse patching, so I've hijacking functions (which will destroy compatibility with mods that modify the same functions).

Progress: I feel like I got a good chunk of the errors while threading resolved, the game no longer explodes with errors. Working on a good way to handle loading resources, as unity wants those loaded from the main thread (apparently). These include things like speech bubbles and sounds.

If there are any modders reading this, you might want to consider making your mod thread safe. Once I get this to a certain point, I might release a version for modders to use to see what if they need to modify their code for parallelism.

Wish me luck!

P.S. If you see FifthElement in discord, thats me.
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on October 09, 2017, 05:39:47 PM
Thanks for this post. Most of my mods will probably work without an issue though two of them could be problematic (Change Dresser and Weapon Storage). I'll pull the dev version down and test it out when it's available  :)
Title: Re: 9thFinger's ThreadingMod
Post by: TastyCookies on October 11, 2017, 12:33:59 PM
Thank you for making this mod, RimWorld definitely needs multi-threading. I wish you the best luck on your progress :)
Title: Re: 9thFinger's ThreadingMod
Post by: SpaceDorf on October 11, 2017, 01:24:06 PM
(http://iv1.lisimg.com/image/1618321/500full-meet-the-fockers-screenshot.jpg)

post to follow.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 12, 2017, 08:14:24 PM
Not to ruin the party, but I thought I should mention...

This may not work. Sometimes communication between threads is greater than the gains from multiple threads. Now, I don't think that's the case here, but it is possible.

Also, I'm changing function definitions at runtime, which is inefficient. We will see if we gain speed rather than lose it.
Title: Re: 9thFinger's ThreadingMod
Post by: CannibarRechter on October 13, 2017, 06:11:51 AM
> Currently blocked on Harmony not working with reverse patching,

I think I asked you this before, but I missed your response. What is this reverse patching thing you are referring to? I know Harmony pretty well, but haven't encountered this terminology or feature.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 17, 2017, 01:20:24 PM
Sorry for getting back to you late, visiting family work and school had me busy for awhile.

Reverse patching...

A good example would be the verse.ai.Pathfinder class in rimworld. All it does is run an algorithm, but the class is stateful (game is still in alpha, things can change...), meaning that while one request for findPath is running, running another one in the same instance will corrupt the results of both (or just cause errors).

The way for a modder to fix this (I cannot emphasize enough the word modder there) is to create a pool of Pathfinder instances. Then, forward calls to findPath to available Pathfinder instances.

One way of patching this with harmony is to have a prePatch that forwards the call to an available PathFinder and call it's findPath. But now you've got a circular loop, because as soon as you call the forwarded findPath, the prePatch code runs again.

The way to fix this is to copy the original findPath code to a new function that is an extension method for Pathfinder (lets call it findPathHijack, or something like that). Then your prePatch forwards calls from findPath to another instance of Pathfinder, but instead of calling findPath again you call findPathHijack so that the prePatch doesn't run again.

In short, reversePatching is taking the original code and patching your own code with the original code. Patching is modifying the original code with your own code. 

There are some workarounds, I can still create a working mod. It just won't play nice with other mods that affect the same methods. I'm not even sure if reverse patching will be possible to the extent that I want it to be.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 17, 2017, 01:22:01 PM
As a heads up, I am taking a short break from the actual coding. I'm trying to make this a senior project for my bachelor's degree, and there's some paperwork I need to fill out, and then I can start reporting to the project coordinator on a weekly basis for credit.
Title: Re: 9thFinger's ThreadingMod
Post by: SpaceDorf on October 17, 2017, 03:06:35 PM
Good Idea  ;D

and real long term comitment.
Title: Re: 9thFinger's ThreadingMod
Post by: Kori on October 17, 2017, 03:43:16 PM
Good luck with this project! :)
Title: Re: 9thFinger's ThreadingMod
Post by: Nightinggale on October 17, 2017, 04:20:00 PM
Now this is a mod, which should really be in Rimworld itself. However based on official statements, it's not going to happen anytime soon, meaning a mod like this is most welcome.

Quote from: FrodoOf9Fingers on October 12, 2017, 08:14:24 PM
Not to ruin the party, but I thought I should mention...

This may not work. Sometimes communication between threads is greater than the gains from multiple threads. Now, I don't think that's the case here, but it is possible.

Also, I'm changing function definitions at runtime, which is inefficient. We will see if we gain speed rather than lose it.
This post makes me so happy about this thread because of past experience with people, who claim they can do wonders if only (certain tech word) is used, like "I can move the graphics from the CPU to the GPU to free up CPU time.", later: "I don't get it. I only get max 1 fps, the CPU load has increased significantly and the screen is drawn upside down" (sadly true story). However the post I reply to here reveals actual insights in threading and added bottlenecks, which can be bigger than the gain.

There are plenty of bottlenecks in going multithreaded and they are not all equally obvious. The most talked about is semaphores, which are used to make sure writing to shared memory stays thread safe, effectively making the threads stall and queue to access memory. If used too little, the code can become unstable. If used too much, the overhead will slow down more than what is gained, effectively making the resulting program run slower. It's a lot harder to get a good stable design with as few semaphores than most new programmers realize.

There are some less talked about issues, which prevents the multithreaded performance from increasing as much as one would think at first glance. Memory I/O and CPU cache activity causes the CPU cores to stall more. Another is CPU temperature or power usage. For instance my CPU clock speed drops 15% when going from one fully loaded core to 4 fully loaded because of a limit to the current (power) to all cores combined and even with reduced speed, it produces more heat. Heat tend to slow down laptops more than this desktop, but I don't have laptop numbers ready. However usually CPU cores aren't fully loaded though as they idle while waiting for memory, even if they show up at 100% in the process overview, meaning it depends on how memory efficient the programming is. That makes this topic quite complex to predict, to say the least.

The real bottleneck in this task is not how much can be spread to other cores, but rather how much the main thread can be reduced. If the main thread is reduced by 10% including the overhead, it doesn't really matter if there are 3 other cores with 10% or 50% load each because it's still the main thread, which will be the bottleneck.

So yeah, predicting performance boost from using multiple CPU cores is not strait forward. Now the question is what to expect from using multiple threads and how to measure it. Combined CPU time makes no sense because that includes time spend on other CPUs and we don't care (much) about that for the time being. I think the best would be how many calculations can we make each tick without visible slowdowns. If we can get a 10% increase there, then I will call it a great success. Sure I hope for 100%+ increase, but I don't find that very likely.

Quote from: FrodoOf9Fingers on October 17, 2017, 01:20:24 PMThe way for a modder to fix this (I cannot emphasize enough the word modder there) is to create a pool of Pathfinder instances. Then, forward calls to findPath to available Pathfinder instances.
This would likely require one instance for each core, multiplied by 2 if the CPU has hyperthreading. This brings up the question: how much extra memory will this require? It's already approaching the limit due to being a 32 bit application, though I have found the out of memory issues seems to go away if playing on a 30% world rather than a 100% world.

Having said that, I also has to point out that it has been announced that pathfinding in A16 is slow, A17 improved performance, but made pawns use weird routes and in A18 it will change again, this time to be equally optimized while using fastest paths. A18 is currently in internal testing, meaning it's very likely that you will have to review your pathfinding code within the near future.

Quote from: FrodoOf9Fingers on October 09, 2017, 04:38:56 PMIf there are any modders reading this, you might want to consider making your mod thread safe.
Good advice, but it's almost useless because the vast majority of modders will not have a clue to how to do that. You will have to write a guide or a list of "do this" and "never do this".

It brings up an interesting question. What if two colonists plans what to do during the same tick and at the very same time on two CPU cores decides to take the same wood to the same furniture construction? What will happen and how do you make it thread safe to prevent the same item from being picked up multiple times?
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 20, 2017, 02:28:49 PM
Quote from: Nightinggale on October 17, 2017, 04:20:00 PMThe real bottleneck in this task is not how much can be spread to other cores, but rather how much the main thread can be reduced. If the main thread is reduced by 10% including the overhead, it doesn't really matter if there are 3 other cores with 10% or 50% load each because it's still the main thread, which will be the bottleneck.

This should be fairly high. I don't believe that graphics takes more than 25% of the CPU time (leaving 70-75% for the main update loop, which I'm threading). But, there is that chance...

Updating the pathfinding algorithm will be rather easy, I haven't modified the actual function code at all, so it'll just be a matter of updating function calls (once Harmony updates). I'm not worried about memory usage right now (I create 8 of everything, I'm on an I7), but before release, I'll go through and re-evaluate my decisions. My first thought is to only spin up extra pathfinders on demand (if I need a 4th, create a 4th and keep it around for awhile), which would be easy to do with my current setup. This is actually my first large C# project, I'm a Java programmer by trade, so I've been playing around alot with coding styles, conventions, etc... that I'll need to clean up / standardize throughout the project before release.

As for your specific threading example: I don't know yet :P. There's tons of hard problems and I haven't gotten tot hat one yet. For the most part, I'll been resolving NREs throughout the code stemming from multithreading. After I get those errors out of the way I get to think about resource contention (finding exceptions is a great way to know the code too, so when I get to the harder debugging I'll stand a better chance). From my ignorant position, I feel like there potentially three possibilities: lock the thought trees so only 1 decision maker can act at a time (eww...), lock some smaller section in the thought tree based on item (I doubt this is possible), or just allow it: two Pawns heading for the same resource trying to do the same task would be resolved when one pawn inevitablely gets there first. If not (VERY unlikely), an NRE would be thrown from picking up an item thats not there, which I would catch and handle with a retick (ewww, but it works).

This would work for an initial release version. I'm not writting perfect code, just working code. Later versions will make things go better.
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on October 20, 2017, 03:58:02 PM
For actions of pawns on shared objects - this is just observation from the game, not actually looking at the code

It looks like items get reserved by pawns before a pawn starts an action on it. In this case it could just be last one wins. If two pawns reserve a production table:
P1 checks and sees table is not reserved and-
P2 checks the table and it's not reserved and reserves it
P1 -reserves the table. P1 verifies it has the reservation on the table and starts going that way.
P2 Verifies it has the reservation but it doesn't. P2 does something else.

Worst case both pawns start doing an action the the bench but the losing pawn gives up after one update cycle.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on October 20, 2017, 07:56:36 PM
Quote from: Kiame on October 20, 2017, 03:58:02 PM
For actions of pawns on shared objects - this is just observation from the game, not actually looking at the code

It looks like items get reserved by pawns before a pawn starts an action on it. In this case it could just be last one wins. If two pawns reserve a production table:
P1 checks and sees table is not reserved and-
P2 checks the table and it's not reserved and reserves it
P1 -reserves the table. P1 verifies it has the reservation on the table and starts going that way.
P2 Verifies it has the reservation but it doesn't. P2 does something else.

Worst case both pawns start doing an action the the bench but the losing pawn gives up after one update cycle.

Actually, the first pawn that reserves it gets it. It may be "overridden" by direct commands, but the first pawn to cycle through reserves the item, assuming that they both tried to get the item on the same tick.

If the same happens in a multi-threaded environment, the outcome is somewhat random (currently, working towards this point).
Title: Re: 9thFinger's ThreadingMod
Post by: Nightinggale on October 20, 2017, 10:26:26 PM
Quote from: FrodoOf9Fingers on October 20, 2017, 02:28:49 PMFrom my ignorant position, I feel like there potentially three possibilities: lock the thought trees so only 1 decision maker can act at a time (eww...), lock some smaller section in the thought tree based on item (I doubt this is possible), or just allow it: two Pawns heading for the same resource trying to do the same task would be resolved when one pawn inevitablely gets there first. If not (VERY unlikely), an NRE would be thrown from picking up an item thats not there, which I would catch and handle with a retick (ewww, but it works).
It's possible that the problem can be solved by reserving the item and then next tick act on it. This way if two pawns reserve the same item, the next tick they both check for it and one of them will realize it's reserved to somebody else and figure out something else to do. The beauty of this approach is that it should be possible to do without semaphores, which would be beneficial for performance. Picking up and dropping items would need some consideration though as even a minor risk of item duplication would be a serious issue.

Quote from: FrodoOf9Fingers on October 20, 2017, 07:56:36 PMIf the same happens in a multi-threaded environment, the outcome is somewhat random (currently, working towards this point).
Random is ok. Nobody will really be able to tell the difference anyway and Rimworld is singleplayer. Introducing that kind of randomness in multiplayer (read: multiple computers doing the same calculations in parallel) is gamebreaking because then it requires that the output of the randomness (race condition) will have to be the same for everybody every single time, which is extremely unlikely.
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on October 21, 2017, 12:48:30 AM
Quote from: FrodoOf9Fingers on October 20, 2017, 07:56:36 PM
Quote from: Kiame on October 20, 2017, 03:58:02 PM
For actions of pawns on shared objects - this is just observation from the game, not actually looking at the code

It looks like items get reserved by pawns before a pawn starts an action on it. In this case it could just be last one wins. If two pawns reserve a production table:
P1 checks and sees table is not reserved and-
P2 checks the table and it's not reserved and reserves it
P1 -reserves the table. P1 verifies it has the reservation on the table and starts going that way.
P2 Verifies it has the reservation but it doesn't. P2 does something else.

Worst case both pawns start doing an action the the bench but the losing pawn gives up after one update cycle.

Actually, the first pawn that reserves it gets it. It may be "overridden" by direct commands, but the first pawn to cycle through reserves the item, assuming that they both tried to get the item on the same tick.

If the same happens in a multi-threaded environment, the outcome is somewhat random (currently, working towards this point).

I should have been clearer, i was assuming a multi-threaded environment where P1 was on one thread and P2 on another - either P1 getting interrupted and both threads being executed on the same core or both executing at the same time. This might be a case where randomness is fine; again ignorant of the code though  ;)
Title: Re: 9thFinger's ThreadingMod
Post by: kubolek01 on October 22, 2017, 03:00:38 PM
No more 3 threads wasting? Yayyyyyyyy! MAKE IT REAL!
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on November 03, 2017, 04:17:43 PM
Progress update:

I just did some paper work to get this mod approved for school credit, meaning that if this counts as my senior project, I can spend more time on it (It has to be approved first). I think I'm almost to the 0 errors thrown stage, but I haven't tried anything with a18 and that might throw everything out of whack.
Title: Re: 9thFinger's ThreadingMod
Post by: Kori on November 04, 2017, 12:39:28 AM
good news! :)
Title: Re: 9thFinger's ThreadingMod
Post by: CannibarRechter on November 04, 2017, 07:14:32 AM
Quote from: Kori on November 04, 2017, 12:39:28 AM
good news! :)

Have you succeeded in getting RW to run in a debugger, out of curiosity?
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on November 04, 2017, 01:51:45 PM
Nope.

What I could do, is resolve all of the compile errors (2k +) leftover from the decompilation, which would take a long time for each new version of the game that comes out. Then I would be able to recompile it in debug mode and be able to attach a debugger (because then it would have a .PDB file with its corresponding sections in the code itself). This may be the killer of the project towards the end when I try to find odd behaviors caused by race conditions. But thats -tons- of work.

For now, I'm just using logging to a file. I might take a break to wait for the versions to settle down though, and then pick up again. I might work instead on a tool (or easy process using git) to find which files were added/changed between versions, and determine if a change needs my attention, making updating to a new version easier.
Title: Re: 9thFinger's ThreadingMod
Post by: CannibarRechter on November 04, 2017, 02:13:00 PM
> Nope.

Crap. ;-P I keep hoping someone will know. ;-P
Title: Re: 9thFinger's ThreadingMod
Post by: Barky on November 29, 2017, 12:19:29 AM
How's this coming along? ^^
Title: Re: 9thFinger's ThreadingMod
Post by: lperkins2 on December 21, 2017, 12:10:37 AM
So, Harmony and HugsLib are open source.  Your reverse patching issue could be solved by creating a modified version of Harmony, which might even get accepted upstream.  The other option would be to use the IL transformation stuff, that lets you see and edit existing methods.  Register a helper mod first in the list that reads the IL code before harmony has mucked about with it, then the multithread mod reinjects it back in later, possibly under a different method name.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on January 26, 2018, 12:10:44 AM
Good news, this has been officially accepted as a senior project for my bachelor's degree.

@lperkins Thats true, but the author of Harmony mentioned that he had the requisite upgrades in the pipeline, I was waiting for him to figure it out. If he hasn't in the past 3-4 months, I'll go ahead and add in the functionality I need. The IL Transformation stuff is the thing that needed to be fixed. Making a second mod seems very excessive. The only benefit I really get from the update though is increased compatability with other mods.
Title: Re: 9thFinger's ThreadingMod
Post by: Nightinggale on January 28, 2018, 12:31:00 PM
Quote from: FrodoOf9Fingers on January 26, 2018, 12:10:44 AM
Good news, this has been officially accepted as a senior project for my bachelor's degree.
I can't believe nobody replied to this yet. It's likely the best thing I have read on the forum this year. If nobody replies it will create the illusion that nobody cares and that's certainly not the case.
Quote from: FrodoOf9Fingers on January 26, 2018, 12:10:44 AMMaking a second mod seems very excessive. The only benefit I really get from the update though is increased compatability with other mods.
The best option is to get Harmony updated. The second best option is for you to update your copy of Harmony. Be aware that this can cause issues with other 0Harmony.dll files loaded in other mods. Either avoid using other mods, which use Harmony or remove 0Harmony.dll from those mods. Odds are that it will work with multiple versions if you load your version first. However loading two versions of the same assembly is undefined (not just in RimWorld) and it might cause weird issues and it's better to be safe than sorry. Loading an unmodified Harmony and then your version will likely prevent you from using your modifications.
Title: Re: 9thFinger's ThreadingMod
Post by: docssy on January 31, 2018, 11:02:05 PM
Definitely watching. I have an i7 3930k waiting to eat Rimworld alive...

Just a request, B18 please. You'll have less work to bring it to Rimworld's 1.0, then having to migrate from A17 to 1.0. That, and B18 is getting more mods daily, which means modders are learning to update their code to be more efficient and subject to less errors.
Title: Re: 9thFinger's ThreadingMod
Post by: Mehni on February 01, 2018, 03:29:56 AM
Re: Debugger.

https://github.com/0xd4d/dnSpy/wiki/Debugging-Unity-Games

I've also seen a recompiled mono dll for RimWorld floating around. Ask forum member neitsa.
Title: Re: 9thFinger's ThreadingMod
Post by: Nightinggale on February 01, 2018, 11:02:12 PM
There is a forum thread about attaching a debugger. https://ludeon.com/forums/index.php?topic=25603.0 (https://ludeon.com/forums/index.php?topic=25603.0). I have to admit that I haven't actually tried it yet, mainly because so far I haven't really needed anything more powerful than writing to the log.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 10, 2018, 03:52:08 PM
I'll give that a try, thanks for pointing that out.

Debugging is painful as is, but I don't think that going through those steps will be good enough. What I really wish I could do is attach a debugger to the native dll. If I correctly fixed all 2k+ errors leftover from the decompilation, I could create the pdb during my own build of rimworld. That, or maybe someone could convince Tynan to release a build in debug mode. Until then, logging and what not will be the bread and butter.

I just now finally got past all of the documentation I needed to write for the senior project (requirements document, a presentation, etc...), and coding can begin again. If I'm to graduate in April, this needs to be done by April, so expect that to be the release time (procrastination is the hardest thing...).
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 10, 2018, 03:53:23 PM
Oh, and don't worry, this will release for B18. If B19 comes out during that time, then I'll see how much work it is to update it to that. Already merged my work to work with B18.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 10, 2018, 03:59:47 PM
Quote from: docssy on January 31, 2018, 11:02:05 PM
Definitely watching. I have an i7 3930k waiting to eat Rimworld alive...

Just a request, B18 please. You'll have less work to bring it to Rimworld's 1.0, then having to migrate from A17 to 1.0. That, and B18 is getting more mods daily, which means modders are learning to update their code to be more efficient and subject to less errors.

Oh, and I should nip this in the bud:

I am making no guarantee whatsoever that the game will run better when multithreaded. Multithreaded can actually make the game slower. While it is typically expected to be the case, sometimes multithreading doesn't work. Again, no guarantee, though I am trying for it. Don't get your expectations too high now :P
Title: Re: 9thFinger's ThreadingMod
Post by: Harry_Dicks on February 11, 2018, 06:59:45 PM
Quote from: FrodoOf9Fingers on February 10, 2018, 03:59:47 PM
Oh, and I should nip this in the bud:

You heard it here first, fellas! FrodoOf9Fingers is promising 200+ fps on a late game, 50 colonist, decked out colony, running 300 mods, with only my Core II Duo!

I'm so grateful we have multi-threading GODS like Mr. 9Fingers here promising better performance for everyone! ;D
Title: Re: 9thFinger's ThreadingMod
Post by: Nightinggale on February 11, 2018, 08:45:15 PM
Quote from: FrodoOf9Fingers on February 10, 2018, 03:59:47 PMI am making no guarantee whatsoever that the game will run better when multithreaded. Multithreaded can actually make the game slower.
Quote from: FrodoOf9Fingers on October 09, 2017, 04:38:56 PMI'm currently working on making the main update loop of the game multi-threaded. Currently, every "thing" does it's "tick" on the same thread, I'm working on splitting the work into separate threads.
I think it would be best to make categories of "thing"s and add the ability to turn each category on/off. It doesn't really matter if it requires a recompile or not to turn one category on/off. The benefits would be:
While 1-2 will help players when the mod is finally released, all list items should be able to help writing a better report. Your grade will not depend on how well your code works, it will come (partially) from how well you can document your code, the impact of your changes and how you use your documented facts to write a report.

Also while we would like to get a working result ASAP, you better talk with your advisor about when you can release anything. We don't want you to get problems because some report checking bot has detected that you handed in code, which can be found online. They do in fact use such automated report checking code and possibly for the code as well since it has become way too easy to copy from online sources. You can include code you didn't write (like Harmony), but then you have do explicitly write that you didn't write it and precisely what you didn't write (like everything in this sub folder) to avoid issues of them claiming you claim credits for writing it. The same goes for writing quotes in the report. It's ok if you explicitly write where you copied the text from.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 12, 2018, 04:32:50 PM
I won't have to worry about someone accusing me of copying someone else's code. I won't go into reasons, but this is for a bachelor's degree and I'm on good terms with most of the teaching staff.

I guess you could at things via categories, plants don't need a mutli threaded pathfinding system, for instance. It's more work though and is beyond the requirements I've set. So it'd be a stretch goal. It might be worthwhile later.

I've only got 2 known bugs atm. Once I get them fixed, it may be time for public testing... So if things go well, you guys should be able to play with a (probably, very broken and buggy) version within a week!
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 20, 2018, 05:22:22 PM
Fix one bug, a million others pop up O.o

Still working, working on a strange bug, something to the effect of:

if (thisQueue.length <= 0)
{
    return;
}
obj = thisQueue.pop();          <---- throws an "Arguement out of bounds" error.

Tis fun :).
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on February 20, 2018, 06:01:11 PM
Haven't been following this thread but the last post has piqued my interest.

Is the queue shared in any way? Are you able to specify the type of queue? ConcurrentQueue.TryDequeue would be advisable if so. Or (slower) ThreadLocal<Queue<?>> might also work.

Otherwise that may need to be surrounded by a try/catch. Maybe in the catch check if the length is 0 with a log error printing the length.  :P

Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 20, 2018, 06:55:37 PM
The code Tynan's, and I'm trying to change as little of it as possible (as using reflection/Harmony will slow the program down). In order to debug it, however, I detach his code and replace it with a copy of my own using harmony.

The probable cause is that a thread is coming in after the conditional portion of the if statement and altering the state of the queue. This shouldn't be happening (in most of the cases), as only one thread even has pointers to the said object. Unless somehow multiple threads are trying to use the same pathfinder object (in this case) at the same time, but I review my code controlling how threads get matched up to pathfinder instances and it doesn't seem possible. It happens in a few other cases. The root cause is unknown O.o, but once I find I'll probably say "Oh duh!"

The interesting thing is, if it happens once, it'll happen thousands of times. Otherwise it doesn't happen at all.

In other news...
There's a crash bug inside the code, though I know the fix (mostly). And there's a bug that makes so you have to restart rimworld before you can load a 2nd game.


EDIT: Multiple threads on the pathfinder object wouldn't cause the issue either. Something is just modifying the queue while what I except to run is working.

EDIT EDIT: The answer, in one case, is that I didn't realize that there was a PawnPathPool.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 20, 2018, 06:58:01 PM
To be specific, the queue is the handmade FastPriorityQueue class made by Tynan.
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on February 21, 2018, 12:57:27 AM
i've done some looking at the pathfinding and iirc it assumes single threaded and shares the same pathfinding object/s - don't remember if it was a shared instance or static class off the top of my head, i think it was static though.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on February 21, 2018, 03:42:06 PM
Quote from: Kiame on February 21, 2018, 12:57:27 AM
i've done some looking at the pathfinding and iirc it assumes single threaded and shares the same pathfinding object/s - don't remember if it was a shared instance or static class off the top of my head, i think it was static though.

It's non static, theres one instance per map.The bypass involves creating multiple instances of the Pathfinder object (a copy actually, to avoid patch looping) in a pool and patching the findPath function of the original pathfinder object to a function that grabs an available copy of pathfinder and running its findPath.

This wouldn't be required if pathfinder didn't store algorithm state data as object members O.o.
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on March 16, 2018, 03:14:24 AM
Checking in, things are progressing. Knocking bugs out one at a time, so fast and some very, very slow (curent bug has taken about 10 hours so far).
Title: Re: 9thFinger's ThreadingMod
Post by: Jagerius on May 23, 2018, 08:26:02 AM
Any news on this one?
Title: Re: 9thFinger's ThreadingMod
Post by: HexCube on June 19, 2018, 07:17:48 AM
Just curious, is this still being worked on? How's it going, if so? No rush, just interested :)
Title: Re: 9thFinger's ThreadingMod
Post by: FrodoOf9Fingers on June 19, 2018, 06:55:58 PM
Well, I got it to a decent point with not that many errors, but the non-intrusive approach I was taking produced minimal results in actual speed. There's just too many parts moving on a global scale.

I still graduated though, so my interest in this has died down. The work was really rough to gain little progress due to lack of good debugging tools. I've moved on to a different hobby project too that maybe someday will be fun for others to play.
Title: Re: 9thFinger's ThreadingMod
Post by: Deathawaits4 on August 28, 2018, 05:10:33 PM
Hei, i was trying to pick up on something like this. My approach on this would be to put pathfinding and unused objects on different threads. I dont really think there is anything else that would profit from multithreading. Pathfinding is eating up most of your cpu late game, while i am still fairly unexperienced, a friend who is a game dev is giving me alot of insight in his multithreading work(he is a network developer for online games). He also played rimworld alot, and this was his answer when i asked him:

Multithreading is not just throwing stuff onto a different Core, it has to be used in a effective and clever way. Most things will get slowed down but one thing that will profit most of the time is the pathfinding. This is really hard to implement but not impossible if you are dedicated enough. It might take 2 years and can break compatibility, there is no way around it. Some mod authors will have to make their mod compatible with yours, it is not possible on your end.

Sadly he is working with c++ and can not really help me with my task except telling me what steps to take. I have now 1 year of experience in working with unity and i do want to create my own game at some point. This would be a good task to increase my skills. Like the OP i cant promise too much, but i am really dedicated since i want to play rimworld lag free myself. I havent started anything yet.. i am still planing, propably releasing some other mods first in order to get used to everything. But i am confident.
Title: Re: 9thFinger's ThreadingMod
Post by: Kiame on August 28, 2018, 09:23:21 PM
Pathfinding should be doable as initially just the claim system would have to be analyzed for thread safety. Might be able to use something like the thing if as the lock (concurency might be possible by adding a new component to things)

I've been curious to try my hand at multithreading rimworld but my other mods keep me busy enough =P
Title: Re: 9thFinger's ThreadingMod
Post by: RawCode on August 30, 2018, 02:02:29 AM
you can't implement MT by dropping random "component" into random place.

start small, implement (yourself) collection (do not take blocking queue or any other stock implementation and do not ever try copyonwrite collections, they are fine but not applicable in case of RW) that support add,remove,read from multiple threads at same time.

to check is write collide, start 10 threads that will add fixed amount of objects into collection at same time, measure result in performance.
best case if you can measure spinlock await time of each thread to estimate real loss of cycles on write sync.

then predict case, where two or more threads calculate path for multiple pawns and end in  pathing into same cell, first thread will be okay, two remaining will be forced to recalculate path, and they may end again in pathing into same cell.
As result, your super MT logic will result in 4x more calculation for no reason.

this will explain you, why trying to implement MT without any reason is bad.

as long as you do not know exact bottleneck, trying to fix it is waste of time, going MT unconditionally also waste of time, as solution may be more simple and effective then MT.
Title: Re: 9thFinger's ThreadingMod
Post by: MostNimbus on January 22, 2019, 02:56:59 PM
Rest in pepperoni