Yes, to work the scene must be driven forward by both/all players.  Was part of two group games, where the person starting it was missing for weeks on end, and he insisted that the rest of us could not go on without him, as he was the one who knew the direction of the story.  Needless to say it slowly died as everyone else in the game got bored and walked away.