I'm pretty sure everyone is going into the cave. It's sort of the only bit of plot you have to follow at this time. The people who don't go in are going to have a pretty boring time waiting while the rest have an adventure raiding an old thieve hideout.