Unfortunately this save is from 0.3.408, but it does display the bug, and without mods. Forbid the hydroponics tables, and set the speed to 3. As the colonists become idle, the FPS drop starts occurring. The more idle colonists, the worse it gets. On my system, it's difficult to notice, until you pan the screen around with WASD.
I don't have any saves from 409 or 410 that show this bug without mods running.
[attachment deleted by admin: too old]