Looking for infos regarding long loading times with modded games

Started by Kenneth, December 18, 2018, 07:11:59 PM

Previous topic - Next topic

rdz1122

Generally about 10 minutes from cold start or mod restart.  In game seems fine.
Operating System: Windows 10 Home 64-bit
Processor: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (8 CPUs), ~2.6GHz
Memory: 32768MB RAM
Card name: NVIDIA GeForce GTX 1070
Native Mode: 2560 x 1080(p)

[attachment deleted due to age]

Irritatedcone

I'll add mine as well.
The logs I'll add are made with "Run in Background" on. (In my case it doesn't change the load times either way.)

Number of active mods: 282
Load time: around 40 min
CPU: i9-9900k
GPU: RTX 2080Ti
RAM: 32GB DDR4 (3000Mhz, CL 14)
SSD: 960 EVO 1TB

Files added:
output_log.txt - debug log with verbose enabled
time-snip.txt - a snippet from output_log.txt with only the Thread 2 load times
DxDiag.txt - system information
modlist.txt - Modlist generated by the mod "[1.0] Modlist"
ModsConfig.xml - Modlist generated by Rimworld

[attachment deleted due to age]

da_dark_derp

I'm also experiencing long loading times (fastest one is 20 minutes so far) though usually it's at 30 or more.
Tried with "run in background" on and off. no real difference was noticed.

this is my list with subscribed items: https://steamcommunity.com/sharedfiles/filedetails/?id=1510307437

These are my specs:
CPU: Ryzen 1500x,
GPU: Asus Rog Strix GTX 1060 O6G,
SSD: Samsung Evo 970 NVME 250 gigs,(I tried it on this device, load times were relatively the same(2-3 minutes faster sometimes)),
HDD: WD Blue WD10EZEX, 1TB(this is the device that rimworld currently is on)
RAM: 2x Kingston ValueRAM KVR21N15S8/8,
OS:   Windows 10 Professional.



[attachment deleted due to age]

vale.maio2

Description:
I've got very long loading times, up to 60+ minutes at one time. As mentioned in this Reddit post (I'm OP), the save file was around 480 MB big. Mod RuntimeGC was telling me I had 56,000 dead pawns somewhere in the world, and the line count for the save file was 19 million. I've cleared them with RuntimeGC and loading time went down to 15 seconds (even tho right now is around 1 minute).

Computer specs:

  • Alienware 15 laptop
  • 16 GB DDR3 RAM
  • 2 M.2 Samsung 850 EVO SSD drives
  • 1 Samsung 850 EVO SSD drive (not M.2 format, 2.5")
  • nVidia GeForce 970m 3GB VRAM
  • Processor Intel i7 4710HQ
  • Windows 10 up-to-date

Mods list:

    Core
    HugsLib
    [SYR] Set Up Camp
    Rimatomics
    [SYR] Prosthetic Table
    Blueprints
    RIMkea
    Mad Skills
    Medical Tab
    Allow Tool
    A Dog Said... Animal Prosthetics
    Glitter Tech
    Tilled Soil
    [KV] RimFridge - 1.0
    Expanded Prosthetics and Organ Engineering
    Conduit Deconstruct
    [T] MoreFloors
    VGP Vegetable Garden
    RuntimeGC
    Dubs Bad Hygiene
    Sometimes Raids Go Wrong
    Real Ruins
    Project RimFactory - Core
    Project RimFactory - Drones
    Project RimFactory - Insanity
    Project RimFactory - Storage
    Gear Up And Go
    Better Pawn Control
    Locks
    Prison Labor

This is my log output

I'm not sure I've done the log thing right, if not please let me know.

2.71828

I'll add another piece of info:

Loading times are 20 minutes (±1 m), I'm on (Arch) Linux, or, more precisely:


$ uname -srvmo

Linux 4.20.0-arch1-1-ARCH #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018 x86_64 GNU/Linux


I attached "rimworld_log" from directly after I loaded up my current savefile (that took between 30 and 60s) and unpaused briefly. I only censored my actual username, otherwise the file is unedited. Since I'm also a bit concerned about memory usage, here is the output of "free", from the same time as my "rimworld_log":

$ LANG=en_GB.utf8 free -h

              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       6.6Gi       135Mi       135Mi       1.0Gi       774Mi
Swap:            0B          0B          0B


At the same time, the Xfce taskmanager reported that Rimworld used "3,7 GiB" of memory, and, somehow, "26%" CPU... Since Rimworld tends to get killed by my OOM-killer (usually because I told him so, since Linux's OOM-Killer is pretty much the opposite of aggressive) after (very) long-ish play sessions (today it was after about 2h 15m), because my system gets unusable, I thought I'd report this, too. Maybe that's just how it is, because I use way too many mods, but maybe there's a memory leak, or something? Another thing: Rimworld seems to have some trouble starting up around the time that "BodyDef Squid has BodyPartRecord of SquidBody whose children have more coverage than 1." is written to the log, and it seems to take a few minutes for Rimworld to write any new entries into the log. I have no idea whether that's significant, a mod issue, or whatever, but I thought I'd mention it either way. 

Furthermore, I attached my modlist ("015h.xml") in the form of a "ModList" from Fluffy's Mod Manager. This way you get to see the mod names and Workshop IDs at a glance, and can easily import it, if that's needed.

My hardware, as output by "inxi", shortly after I quit a Rimworld session, so memory is still quite full (I might be a bit paranoid, but I censored my hostname...):

$ inxi -Fx

System:    Host: HOSTNAME Kernel: 4.20.0-arch1-1-ARCH x86_64 bits: 64 compiler: gcc v: 8.2.1 Desktop: N/A
           Distro: Arch Linux
Machine:   Type: Desktop Mobo: ASRock model: H67M-GE/HT serial: <root required> BIOS: American Megatrends
           v: P2.10 date: 04/27/2012
CPU:       Topology: Quad Core model: Intel Core i5-2400 bits: 64 type: MCP arch: Sandy Bridge rev: 7
           L2 cache: 6144 KiB
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 24755
           Speed: 1596 MHz min/max: 1600/3400 MHz Core speeds (MHz): 1: 1596 2: 1596 3: 1596 4: 1596
Graphics:  Card-1: Advanced Micro Devices [AMD/ATI] Barts XT [Radeon HD 6870] driver: radeon v: kernel
           bus ID: 01:00.0
           Display: x11 server: X.Org 1.20.3 driver: ati,radeon unloaded: modesetting,vesa
           resolution: 1920x1080~60Hz
           OpenGL: renderer: AMD BARTS (DRM 2.50.0 / 4.20.0-arch1-1-ARCH LLVM 7.0.0) v: 3.3 Mesa 18.3.1
           direct render: Yes
Audio:     Card-1: Intel 6 Series/C200 Series Family High Definition Audio driver: snd_hda_intel v: kernel
           bus ID: 00:1b.0
           Card-2: AMD Barts HDMI Audio [Radeon HD 6790/6850/6870 / 7720 OEM] driver: snd_hda_intel v: kernel
           bus ID: 01:00.1
           Sound Server: ALSA v: k4.20.0-arch1-1-ARCH
Network:   Card-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 v: kernel port: d000
           bus ID: 07:00.0
           IF: enp7s0 state: up speed: 1000 Mbps duplex: full mac: 00:25:22:bf:37:45
Drives:    HDD Total Size: 1.82 TiB used: 287.00 GiB (15.4%)
           ID-1: /dev/sda vendor: Samsung model: HD154UI size: 1.36 TiB
           ID-2: /dev/sdb vendor: Western Digital model: WD5000AAKX-001CA0 size: 465.76 GiB
Partition: ID-1: / size: 212.25 GiB used: 148.85 GiB (70.1%) fs: ext4 dev: /dev/sdb3
Sensors:   System Temperatures: cpu: 27.0 C mobo: N/A gpu: radeon temp: 34 C
           Fan Speeds (RPM): N/A
Info:      Processes: 181 Uptime: 10h 09m Memory: 7.75 GiB used: 3.59 GiB (46.3%) Init: systemd Compilers:
           gcc: 8.2.1 clang: 7.0.1 Shell: zsh v: 5.6.2 inxi: 3.0.10


[attachment deleted due to age]

Kayrah

i run the sk modpack and have loading times around 10min
3.5ghz 8700k
SSD
32gig RAM

the overall performance of the game is pretty poor. It slows down a lot during the game and only reloading will bring it back to normal speed. Not sure its the modpack or the core though.
I wish there was a multicore client support. Its 2019!

2.71828

Quote from: notfood on December 19, 2018, 01:53:15 PM
Kenneth, investigate caching the patched XML. This one works for me to cut 2-3 minutes from my load of 7-10 minutes.

https://gist.github.com/notfood/893b31b1a32b2043d8def1dbbc177a91

Is there any way I could use that bit of code to speed up my loading times? Could I add this as some kind of mini-mod somewhere?

Quote from: Tynan on January 03, 2019, 01:24:30 PM
I moved this thread to General since it's fair critical and we want to get as much info as possible. Thanks to all those giving info so far!

I don't know whether this question makes any sense, but should the loading times be shorter than they are, i.e. are they longer than expected?

Ira671

Hello just back again will there be updates to bring down loading times. I still have the same amount of time from when I last post to just get into the main menu.

Zaltys

Already posted my log on page one, but now that the BetterLoading mod is out:

Noticed that it spends a large chunk of the loading time on step 'Reloading DefDatabase: RecipeDef'. That seems to be because I have a lot of race and animal mods, plus patches that inject recipes into those.

So I can imagine the 'allowed' list getting huge for each recipe. Which may be an ineffective way of handling it when most recipes are along the lines of "available for all animals" or "for all humanlikes".

notfood

Here is another way that halves loading times dramaticaly. Caching the Assembly lookup:

https://github.com/Zetrith/Multiplayer/blob/master/Source/Client/Optimizations.cs#L98-L114

Creadit to Zetrith for finding it.

Basically, RimWorld does reflection every time it finds a type in XML. The more XML custom types the more reflection and reflection is slow. All mods use it. This patch makes it so there is a cache lookup before returing the type.

Many report 2-4 times faster loading.

2.71828

Quote from: notfood on February 08, 2019, 03:28:01 PM
Here is another way that halves loading times dramaticaly. Caching the Assembly lookup:

https://github.com/Zetrith/Multiplayer/blob/master/Source/Client/Optimizations.cs#L98-L114

Creadit to Zetrith for finding it.

Basically, RimWorld does reflection every time it finds a type in XML. The more XML custom types the more reflection and reflection is slow. All mods use it. This patch makes it so there is a cache lookup before returing the type.

Many report 2-4 times faster loading.

So that is what speeds up loading times so much? Do you think these optimisations could be made into their own mod (unless the Devs decide to put this into Rimworld proper)? When I tried Zetrith's Multiplayer mod out, loading times went down from ~25min to less than 10, but I don't need the multiplayer part, and it doesn't play well together with my current modlist/save, anyway, so I can't use it.

notfood

Another optimization trick:

using Harmony;
using Verse;

namespace FixGamePlz
{
    public class HarmonyPatches : Mod
    {
        public HarmonyPatches (ModContentPack content) : base (content)
        {
            HarmonyInstance.Create ("rimworld.fixgameplz").PatchAll ();
        }
    }

    [HarmonyPatch(typeof(ModMetaData), nameof(ModMetaData.PreviewImagePath), MethodType.Getter)]
    static class HideImages
    {
        static bool Prefix(ref string __result)
        {
            __result = string.Empty;
            return false;
        }
    }
}


Preventing RimWorld from loading the preview images of mods saves a ton of ram, dramatically. It loads the preview files and keeps them in memory even if you aren't using the mods. It explains why my Steam takes so long to load while the SendOwl takes very little time. I have a ton of subscribed workshop mods.

When categorizing the mods for the Multiplayer mod, Parexy had to subscribe to the 2000 available for 1.0, his ram would go up to 6GB! With this patch, his ram on load is 700mb. Dramatic difference!

The ideal solution would be to load those files on demand, not every available mod at start up.

Included the tiny compiled mod.

[attachment deleted due to age]

forahilarm

Quote from: notfood on February 08, 2019, 03:28:01 PM
Here is another way that halves loading times dramaticaly. Caching the Assembly lookup:

https://github.com/Zetrith/Multiplayer/blob/master/Source/Client/Optimizations.cs#L98-L114

Creadit to Zetrith for finding it.

Basically, RimWorld does reflection every time it finds a type in XML. The more XML custom types the more reflection and reflection is slow. All mods use it. This patch makes it so there is a cache lookup before returing the type.

Many report 2-4 times faster loading.

I can confirm that my loading times drastically decrease with this, however, I've found that my world generation junks up pretty badly with my other 200 mods. Is there possibly a way to yank out that code and make it a mod in of itself for the purpose of self caching?
ayyyy lmao

Ronburgundy

My game with mods loads fine till this point then it sits here for  30+ minutes
here is the config any help would be helpful.
https://pastebin.com/HAnMwSAc


2.71828

Quote from: forahilarm on February 13, 2019, 11:05:39 AM

[...]

I can confirm that my loading times drastically decrease with this, however, I've found that my world generation junks up pretty badly with my other 200 mods. Is there possibly a way to yank out that code and make it a mod in of itself for the purpose of self caching?

I guess it's not that easy, since I already asked for it, as well, and notfood hasn't reacted to that, either. This is of course understandable, since that is neither her mod, nor should it be her that has to fix the game, because that's what this would basically be, if I understand it correctly, i.e. a bugfix mod.

Additionally, none of the Devs (especially Kenneth, who has created this thread), have reacted to any of the three (!) possibilities to speed up loading times that notfood proposed, and Kenneth hasn't been seen in this thread since December 19, although Tynan himself has written that this thread is "fairly critical", because they want to fix this themselves. I guess they're busy doing something else, but it would still be nice if one of the Devs dropped in and maybe told us whether the info that gets posted here is even helpful to them, whether we are supposed to continue posting logs (even if we have posted one already), etc. No one, except myself, has looked at my log and my modlist, by the way, assuming the "downloaded x times" thingy is counting correctly.