I mean, if you as the GM are no longer enjoying the game, that's a pretty big problem. This WAS a very ambitious game after all, and there's a lot of wildly diverse people and characters, so it's understandable that there'd be issues like that popping up. If you want to keep going with the story cutting back to a small group might be your best option, maybe with some extra instructions/reminders on what you want and don't want from the players? In the end what's important is that you do what feels best for you, whether it's changing things up or cutting the game entirely.