My feeling is that outside of combat, it's better to let some people most more often than others if they're so inclined. Just so long as the pace is not too fast--hopefully no one is left completely in the dust or don't get the chance to chime in if something important comes up.