Maybe Insane's Zelith could have a reason to teleport you to his tower. Or maybe the Lord could ask such of Zelith. Maybe the brothers have done work for the Lord before and he wants Zelith to teleport them to the castle to do some reconnaissance on the town?