First World Problems (Large colony issues)

[Lechai]

Hi All

Not sure if this is the right place to put this discussion as its part commentary, part story, part bug report and part suggestion. If a mod feels like moving it then go nuts, but 'general' feels right.

My goal is currently to try and get to 100 colonists, this is to see what happens both performance wise and with gameplay for issues that don't appear until over 20 or so colonists. I'm currently at about 52 colonists
.
Firstly, here is my current Colony using Randy Random on version 250, i havn't updated to 254b as not much changed and restarting would set me back.

Currently i have over 100,000 credits, 10,000 food, 3,000 metal and random amounts of all the resources that aren't implemented yet.
I'm not trying to brag here, just paint a picture

Performance:
My rig is about 3 years old but was built for games like Crysis at the time so was pretty top end at the time. Currently it seems to run perfectly fine until it rains, then it drops FPS for about 10 seconds until it stabilizes. Generally outside this I haven't had any game ruining performance issues so far. The memory cleanup that runs every so often, really appears to help improve and maintain the fps/performance!

Colonist Job Allocation:
Ok, so you would think that having 52 colonists would get lots done real quick? Actually the opposite has proven true. I've found that jobs i assign, such as mining constructing etc, takes absolutely ages to complete. And here's why. The colonist management system (I'm going to call it the CMS) assigns the closest available colonist to the job that's required. This means that if i pave a room that is 3x3 that's 9 construction jobs available. It will choose the closest 9 available colonists that dont have a job with a higher priority. the problem is that these colonists are everywhere all over the map. By the time the furthermost arrives to pave his 1 tile, the closest guy could have built the whole lot himself. this is even worse when mining. as each layer revealed draws a colonist from far away, while the miner standing right there wanders off to find something to do as the next tile has already been assigned to someone else. The CMS has clearly been designed with 3-15 colonists in mind. And it actually works really well. But at higher levels all your colonists do is spend the whole day walking from one end of the colony to another. The very worst example of this is if a fire breaks out. The closest firefighter runs over. As he is fighting it it spreads, the next closest colonist runs to get this 2nd tile. The first tile gets put out and then the first guy, job complete, leaves. Meanwhile while Mr2 is on his way the fire has spread to two more tiles that instantly get assigned to 2 more people even further away again. By the time they arrive, tile 2 has been put out but mr2 is now on fire standing in the fire of tile 3, a colonist even further away again is assigned to the guy on fire... rince repeat. This actually happened. I had a single fire on a battery from a short circuit, and despite 52 firefighters. I ended up having an entire room destroyed, 2 colonists in bed and 6 guys wounded. Currently my best solution is to have all jobs restricted to only 5 colonists with the exception of plant cutting cleaning and hauling. This however leads to part 3.

Idle colonists:
This is more frustrating than it seems. They just wander around aimlessly and cause multiple issues.
1. They run outside the base and trigger raiders early.
2. They wander into crappy areas such as corpses and dark mine shafts lowering loyalty for no purpose
3. I have no idea who is and isn't busy and if i need a specific person he's wandered off.
4. They don't add value.
My suggestion/solution is the room on the lower left corner of my colony. It's a recreational room, (complete with bar, billiard tables, arcade machines and a dukebox. Idle colonists need something like this so they can just stay happy and most importantly, out of the way. Additionally if this was in the center of the colony then if they are required for a job they could head there without being at the edge of the map.

Anyways that the main points for now. Things like no endgame-research have already been covered.

Thanks for reading,
Hopefully i can give another report when i hit 75 and 100 colonists

Happy Crash Landing....
Lechai

[Nasikabatrachus]

These problems aren't limited to large colonies. For instance, I've had the same problem with firefighting on my 14-person colony. A short circuit exploded deep inside the mine and the colonists took turns slowly chasing the fire as it spread from tile to tile. Worse, they would repeatedly catch on fire as they were firefighting, and colonists who had just finished fighting a fire would go about their businessâ€"or, in the case of an assassin, wander around placidlyâ€"right next to their burning comrade. It took a while just to sort out who was doing what and who was going to help douse the burning person in their midst (as it turned out: someone clean across the colony).

Similar problems happen when trying to build things, although this mostly becomes evident in tense situations like imminent raids. My preferred solution is to limit the jobs that colonists can perform, like having certain people only construct/repair or mine, since you don't need everyone to do everything even in a small colony, but that still requires a certain amount of micromanagement to avoid delays, not to mention that having only a limited number of firefighters can be dangerous in certain situations.

I haven't had the same problems with mining that you've had, however. I opened up my 34 pop colony just now and designated a straight line into the rock to be mined, and one colonist did about half of the designated work without stopping. In that colony, I have manual priorities turned on and just 6 colonists have mining at priority 1, while everyone else has mining at priority 4, and this was with about 25 idlers at the time. Do you have manual priorities set up? I think that might help your issues.

[Tynan]

This is interesting from a game design perspective. The key reason I want to keep colonist counts closer to The Sims than Dwarf Fortress is that there really is no place in a story for more than 10-15 people. However, a secondary reason is just that large colonies expose a lot of technical, interface, AI, and performance problems in a game primarily designed for 3-15 people. It costs time and effort to develop systems that scale two orders of magnitude (from 1 to 100 colonists), and I'd honestly rather spend that time developing more interesting story-making mechanics for small groups.

You didn't mention the organization burden of an overview screen with 50 people in it, or how you handle drafting, fighting, and undrafting with such a huge population, or how the AI scales challenges up against your massive population. These are areas where I would anticipate problems.

Thanks for exploring the possibility space.

[ShadowDragon8685]

I plan to be exploring that possibility space myself, so...

As regards fires, once your colony is mature, I find the best way to handle them, if possible, is just to sell everything which is flammable around the fire, and everything near it which hasn't caught fire yet, draft everyone to stand well away from it, and just let the motherfucker burn.

It's cheaper than a protracted battle with the fire, doesn't risk colonists, and you get some cash back from the things you successfully hocked.


Tynan: I know your idea is for 3-15, but c'mon. This is so obviously Dwarf Fortress in Spaaaaaace!

We're gonna want more folks. Sooner or later, you're gonna want to sort out some mechanisms for handling a goodly-sized fortress. I doubt it's possible to expose the AI to modding, though - or is it? If it is, I bet you could easily do so, let the crazy modders work out how to do it for you, then crib from their notes.

[EtaBetaPi]

I'd suggest a change to how tasks are assigned, as follows:

Currently tasks are assigned based primarily on whether a colonist has a task already, with distance to the taks being assigned used as a tie-break. With colonists only able to remember one task at a time there's not much more that can be done, but what if colonists were able to plan ahead and store a `to-do' list of tasks? (This could be as simple as a FIFO queue, though of course a priority queue would allow for more sophisticated systems.)

With a list of tasks for each colonist available, tasks could be assigned based on a function of both distance and the number of tasks on the colonist's to-do list. (Note: In this case it makes sense for `distance' to mean distance to the location of the last task on the colonist's to-do list, rather than distance to current location.) The function could be determined heursitically according to what works and also be exposed to modding so that players can change behaviour to suit their own playstyles.

For example, let's say the function is (total time of tasks on colonist's to-do list) + distance/(colonist speed) and suppose there are 15 idle colonists around the map, and 10 metal tiles we want to build. The first tile gets assigned to the closest colonist. The second gets assigned either to the same guy if there is no-one who can get there before he completes the first, and to the closest free colonist otherwise, etc. If there were 3 nearby colonists and many far-away ones, then those three would complete 3-4 tiles each and the rest would sit tight.

[zztangnr]

I actually have the same plan to reach 100 men except I don't have so much time to play it these few days. I'm at about the same stage as you, about 50 colonists.



Like what you mentioned the colonist job allocation at high numbers really reduces the efficiency. I have almost stopped playing it, mainly due to seriously lags. What I usually do is I set the game to run in background on, assign a lot of tasks when the game is paused(it lags not so much as compared to when the game is progressing). Then I switched to my desktop and do something else(watching Youtube, etc). Everytime I hear the notification sound or I feel that the assigned tasks are done I switch back to game, take a look and assign new tasks. Yeah this is much less fun, and switch to desktop may not increase the performance much, but looking at the game with very low performance is extremely painful.

[todofwar]

I plan to be exploring that possibility space myself, so...

As regards fires, once your colony is mature, I find the best way to handle them, if possible, is just to sell everything which is flammable around the fire, and everything near it which hasn't caught fire yet, draft everyone to stand well away from it, and just let the motherfucker burn.

It's cheaper than a protracted battle with the fire, doesn't risk colonists, and you get some cash back from the things you successfully hocked.


Tynan: I know your idea is for 3-15, but c'mon. This is so obviously Dwarf Fortress in Spaaaaaace!

We're gonna want more folks. Sooner or later, you're gonna want to sort out some mechanisms for handling a goodly-sized fortress. I doubt it's possible to expose the AI to modding, though - or is it? If it is, I bet you could easily do so, let the crazy modders work out how to do it for you, then crib from their notes.

Your firefighting is inspired, I do the same. Except sometimes it backfires and my roofs collapse on me, I've gotten better at ensuring I don't screw over a structure entirely. Normally I try to arrange flammable things so that I just need to sell a wire or two, of course if you have a building on fire that is problematic.

In Total War games they have amplified or changed the AI a bunch in different mods, so I wouldn't say it's impossible you just have to know what you're doing. It's not like reskinning a muffalo, increasing it's likelihood to go insane to 70% and renaming it a wolf. You really have to know something about programming.

[Stickle]

I have experienced all of these problems with colony sizes of ~15. I think it's a combination of population and the physical size of your colony. In the beginning, even if your colonists keep switching off it hardly matters, because your colony is small and it takes maybe an extra 2 seconds. But when your colony is approaching 1/4 the map, almost regardless of your population, it becomes an issue.

I don't usually have a problem with mining (when a miner exposes a new wall he is usually closest and gets that job right away), but if I'm trying to build something like a 3-tile-wide wire it can take half a day to complete, because 3 different colonists come running from across the map; the wire would not only be built many times faster if the closest builder did it all himself, but the way it works now it slows down everything! Those colonists spend half the day running around the map instead of actually working.

I would definitely like to see the task management algorithm overhauled, but I can't imagine that's a simple order...

[Zebedeu]

This problem is similar to one I faced writing my PhD thesis on production manufacturing.

My production scheduler had to automatically assign work pieces to one of the machines in a cluster with optimal efficiency.
The problem was that sometimes the closest machine would be busy at the time of decision, so the next closest available machine would be selected. However, given the travel time required to bring the piece to that machine, it would sometimes be more efficient for the piece to simply wait for a closer machine to finish.

The final algorithm I came up with essentially took the travel times, queue size and time-to-finish for each work machine, and selected the lowest (this is somewhat simplified, but if you're having trouble sleeping I can send you my thesis).

I'm not sure how much of a burden this would put on the game's already taxed AI, but my feeling is that a similar solution would fix the general problem of task assignment, which would work as well for 10 colonists as for 100.
I'm also not sure of how complex something like this can be in a game where each colonist can do a lot of different tasks - I had the benefit of each machine doing only one job.

[Tynan]

This problem is similar to one I faced writing my PhD thesis on production manufacturing.

My production scheduler had to automatically assign work pieces to one of the machines in a cluster with optimal efficiency.
The problem was that sometimes the closest machine would be busy at the time of decision, so the next closest available machine would be selected. However, given the travel time required to bring the piece to that machine, it would sometimes be more efficient for the piece to simply wait for a closer machine to finish.

The final algorithm I came up with essentially took the travel times, queue size and time-to-finish for each work machine, and selected the lowest (this is somewhat simplified, but if you're having trouble sleeping I can send you my thesis).

I'm not sure how much of a burden this would put on the game's already taxed AI, but my feeling is that a similar solution would fix the general problem of task assignment, which would work as well for 10 colonists as for 100.
I'm also not sure of how complex something like this can be in a game where each colonist can do a lot of different tasks - I had the benefit of each machine doing only one job.

I'd be interested in reading the thesis, if you could send it over. [email protected]. Thanks man.

I was thinking I'd solve this (eventually, not soon) by just having colonists reserve groups of jobs instead of individual jobs. So if there are 6 wall sections to be built, one guy will reserve all of them and not call anyone else over to help. It doens't solve the problem of someone coming across the map to do the first wall section, but it theoretically alleviates the constant hand-offs. However, this won't happen any time soon, because there are bigger holes in the game right now.

[Sacarathe]

Cant find "link to post" so i'll quote myself from the cheap ideas thread.

Like the home zone, for fire control, the DNL zone is for "Do not leave", if a colonist enters the DNL zone, they cannot leave unless drafted(or on fire, but not to, put out a fire.

The main advantage of this is during raids, to ensure the colony can continue to function without drafting everyone. If a fire in the home zone cannot be reached due to a lack of overlap it will be ignored, as will any other tasks given to that npc.

Perhaps NPCs may leave the DNL zone for priority 1 tasks only. After which there is no reentry obligation. You could have several layers for this, and add it to the overview and tag NPCs to their layer, for example, non violent warden cannot leave cells and* food dispenser, expert grower cannot leave base (which contains hydroponics, food dispenser home) etc.


* Link must exist.

This would eliminate many of your problems, you could also have do not enter areas (that can only be entered through drafting or if they overlap a do not leave). Thus allowing you to for example keep your soldiers in a small area with just a food machine.

Another tool for this, is that you could set it that if a job is assigned in the dont leave zone, new jobs outside the zone cant be accepted AND npcs from outside the zone cannot be assigned jobs either. NPCs would as a restriction however not leave these zones even for food or sleep, only when drafted or if two zones overlapped.

I was thinking I'd solve this (eventually, not soon) by just having colonists reserve groups of jobs instead of individual jobs. So if there are 6 wall sections to be built, one guy will reserve all of them and not call anyone else over to help. It doens't solve the problem of someone coming across the map to do the first wall section, but it theoretically alleviates the constant hand-offs. However, this won't happen any time soon, because there are bigger holes in the game right now.

What would happen if the guy reserved stuff and then needed food, would he complete the work (unless an individual task required X seconds of work)? An advancement of your statement could be work is reserved based on total time required, so npc automatically reserved either an area, or X other items near the initial item reserved and then if the total build time is too high, un-reserve them..

[Produno]

I think not limiting by capping the amount of colonists, or most things for that matter in this type of game is very important. I don't like to feel restricted or like im being forced not to do something. So I think these types of research/discussions are import for Rimworld. But that being said I think their are ways to help limit this without actualy capping the population.
Once theirs much more stuff in the game people will probably be happier with less colonists but we could also aid that with other mechanics.

A couple of ideas I had...

Space - can we try and limit space with out it being a limitation? That probably don't make sense but hear me out. To have a huge amount of colonists we need a lot of space, space for bedrooms, farms, recreational facilities and other stuff needed to keep the colonists happy.
So one idea could be to make your farms produce less food or take longer to produce food. That would mean you need more farms to get more food for a higher population. Thus taking up more space.
Recreational facilities - I hope you will be adding these in the future as these also take up more space.
Make building, mining, harvesting take a lot longer than they currently do. You could then add a mechanic that makes the colonists unhappy if they are cramped, meaning a large base would need to be built before having a lot of colonists.  It also benefits a whole host of other things ive seen and comented on around the forums. Yes - people can still run in background to get the job done quicker but hopefully when more things are added this will keep the player occupied slowing down the building process which in turn hopefully slows down the population of the colony.
Cant the percentage rate of choosing to have a traveller enter the map be slowed down?

Obviously their are drawbacks to this, like the map size and if you wanted to expand it in the future. But a plus for that is if you have map sizes optional as you do now, it would limit the amount you can have on a given map size. So if you want more colonists and you have a good pc you choose a bigger map.


As for management of more colonists the way the AI handles priorities has already been mentioned but one idea I have could be groups. You can set groups or squads or whatever else you want to call it to group a set amount of colonists together effectively as one unit to do jobs. So if you have 4 colonists who are builders, you set them to a group so that group can be selected as one unit or a work force. I also wonder if this could be done to help with the AI, so instead of the building job being assigned to one colonists its shared between the four?


I had more ideas but forgot them as I was writing heh. ill update later

[NazTaiL]

What about creating an area of activities for each colonist. Using the same method as for the home zone to deal with fire. It can be optionnal.

[Lechai]

You didn't mention the organization burden of an overview screen with 50 people in it, or how you handle drafting, fighting, and undrafting with such a huge population, or how the AI scales challenges up against your massive population. These are areas where I would anticipate problems.

yep, those were areas i also had issues with, but i had to go to a meeting at work and ran out of time to write it all down.

The final algorithm I came up with essentially took the travel times, queue size and time-to-finish for each work machine, and selected the lowest (this is somewhat simplified, but if you're having trouble sleeping I can send you my thesis).

Words cannot describe the simple brilliance of this solution! TY read that Thesis!
+1 votes to Zebedeu

[nnescio]

... As regards fires, once your colony is mature, I find the best way to handle them, if possible, is just to sell everything which is flammable around the fire, and everything near it which hasn't caught fire yet, draft everyone to stand well away from it, and just let the motherfucker burn.

It's cheaper than a protracted battle with the fire, doesn't risk colonists, and you get some cash back from the things you successfully hocked. ...

Fastest way to deal with fire ('though mostly impractical): Blow it up.

Draft a couple guys and give them frags. Toss 'nades into fire. The resulting explosion will kill the fire.

Blasting charges also work similarly.

This used to be the only way to deal with fire before Tynan implemented firefighting for colonists.

Of course, the collateral damage can be severe, but hey, it's rated D for Dorf!