Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
Ghada Sokar Johan Samir Obando Ceron Aaron Courville Hugo Larochelle Pablo Samuel Castro
The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization.
Reinforcement learning, Deep reinforcement learning, Mixture of experts
We demonstrate that one of the main reasons for the efficacy when using mixtures of experts comes from the choice of tokenization, and we demonstrate that even with a single expert, tokenization is enough to yield the previously observed benefits.