CartPole is actually a very simple problem. Just by using a simple random search you can get more than 200 reward, take a look at this 7.
RNNs should outperform MLP’s when the temporal/sequential information is really relevant. In the case of CarPole, you actually don’t care that much about what happened 2 iterations before, the actual state is much more relevant for you to decide what to do than the previous ones. This is not the case for a lot of applications where RNNs shine, like NLP, where, for instance, the last word of a sentence generally isn’t enough for you to interpret the whole sentence, and more, the order of the words matters a lot in language.