Recent work using auxiliary prediction task classifiers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018)  and CoVe (McCann et al., 2017), are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives-language modeling, translation, skip-thought, and autoencoding-on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.
This paper tests various pretraining objectives (language modeling, machine translation, skip-thought, and autoencoding) on two syntactic tasks: POS tagging and CCG tagging. It finds that language modeling outperforms the other pretraining objectives; additionally, randomly-initializing an encoder achieves decent performance when given a large amount of labeled data for the tagging task. The experiments in this paper are very thorough and explained well. By controlling for pretraining data size, the authors are able to reasonably claim that language modeling is superior to translation as a syntactic transfer learning task. On the other hand, I have some concerns regarding the significance of the paper's contributions, and as such I am borderline on its acceptance. 
comments:
- the experiments in the paper feel biased towards language modeling. Language modeling is the only token-level prediction task of the four objectives here, but both of the two downstream tasks are at the token level. It is perhaps unsurprising then that language modeling performs best; perhaps the authors could have considered some sentence-level downstream tasks as well to properly control for this? Or added some more word-level pretraining objectives? 
- sort of relatedly, the authors do not provide any explanations as to *why* language modeling is a better pretraining objective than translation. What kinds of examples do the tagging models using LM pretraining get right that the translation models do not? Such an analysis could help provide more concrete insights into what kind of information each objective is encoding.
- the claim that LMs > translation is not a new finding. The authors cite Blevins et al, who find the same result on the task of dependency arc prediction. Similarly, the surprisingly good performance of random encoders was also found in Conneau et al., ACL 2018. As the main contribution of this paper seems to be a more controlled study of Blevins et al on different syntactic tasks, I don't think there is enough here for an ICLR submission. 
- what is the effect of the specific dataset and architecture on the results? Here we just look at a couple translation datasets (all news data) and LSTM models. Do things change when we move to transformers or more diverse domains?