I'm finding that even if I'm not loading anything else, the time it takes for a pawn to pick up an unconscious prisoner and carry them to the edge of the map (at least on a mountain map where you can't walk in a straight line) gets to be long enough that they go on a mental break. Especially since they'll spend a decent amount of time wandering around before actually picking up the prisoners and heading to the map edge, even if there's nothing else to load. Alternately they'll get hungry halfway through, head back to the base to eat, and then just...never load the prisoner in at all. Often the wardens will take any prisoners that were dropped back to bed and they won't end up in the caravan at all.
I'd rather not drug up my pawns just to get a caravan formed, but I'm not sure what else to do here?
If you know which side the caravan plan to exit on, build a 1 x 2 room near the maps boarder, place a sleeping spot and designate for prisoner, then place a caravan parking spot next to it. That may help you get them off the map quicker.
I've had to do something when attempting to release / save prisoners / visitors with effects like brain damage or severe addictions.
Hope this works!