The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process . Intriguingly, this phenomenon suggests that initialization strategies for DNNs can be improved substantially, but the lottery ticket hypothesis has only previously been tested in the context of supervised learning for natural image tasks. Here, we evaluate whether "winning ticket" initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL). For NLP, we examined both recurrent LSTM models and large-scale Transformer models (Vaswani et al., 2017). For RL, we analyzed a number of discrete-action space tasks, including both classic control and pixel control. Consistent with work in supervised image classification, we confirm that winning ticket initializations generally outperform parameter-matched random initializations, even at extreme pruning rates for both NLP and RL. Notably, we are able to find winning ticket initializations for Transformers which enable models one-third the size to achieve nearly equivalent performance. Together, these results suggest that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in DNNs.
This paper studies the existence of “winning ticket” initialization in natural language processing (NLP) and reinforcement learning (RL). The lottery ticket hypothesis has been found effective in over-parameterized deep neural networks, which provides better sub-network initialization if not outperforming the original full network. Experiments show that winning ticket initializations generally outperform parameter-matched random initializations on recurrent LSTM models, large-scale Transformer, and discrete-action space RL tasks, including both classic control and pixel control.  A substantial number of parameters can be saved.
This paper is clearly motivated and easy to follow.  The results are interesting.  However, my major concern is its intellectual merit.  The paper does not seem to propose any new algorithm, and the application of lottery ticket (late rewinding) to LSTM, Transformer, and RL looks quite straightforward. There are a few insights drawn from the experiment, such as when only the Transformer weights are pruned.  However they does not appear substantial. Overall, I do not find the paper innovative enough for publication at top conferences like ICLR.  I understand this is subjective, so I leave it to the AC for further evaluation.