Pls add the current relation info to the faction mentioned in the encounter-window, where you can decide wheather you attack, trade or just ignore the caravan.
When this pops up, unless you remember, there is no way to tell, if this is a faction you want to annoy or not. You cannot click the faction tab at this point.
Thanks for listening!