I think a good mix of adventure and downtime is best, but it really depends on the characters. Ulan doesn't want to be there at all, so the more time he spends in Point Respite, the better. He'll obviously be forced into leaving, so if the others want to do a lot of adventuring with only a little downtime, he'll have to go along with that.