The issue may be related to something in NVIDA's Geforce Experience manager software, I'm not sure exactly what it was, I was chasing a theory of that the game "Streaming" bs was somehow messing with my game, so I disabled gamestream and exited NVIDA experience completely and re-opened rimworld and the FPS lag seems to have gone away. Unfortunately, I can't reproduce the FPS drop by re-enabling GeForce Experience which is annoying and I am wondering if something was messed up in the settings that got reset when I started messing with it (they use cloud saves to preserve settings). Anyway, time will tell I suppose.
Edit:
I reproduced it, seems to be the Geforce Experience "Share" option under General when turned on causes this issue to start occurring.
http://imgur.com/a/4eImX
Edit:
I reproduced it, seems to be the Geforce Experience "Share" option under General when turned on causes this issue to start occurring.
http://imgur.com/a/4eImX