It seems as if they would need to be in combat for you to know where they are, judging by the bandit rules. Which I suppose makes sense since otherwise they would probably be nestled inside a castle some where.