Published as a conference paper at ICLR 2018 SEARNN: TRAINING RNNS WITH GLOBAL-LOCAL LOSSES
We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We first demonstrate improved performance over MLE on two different tasks: OCR and spelling correction. Then, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes. This allows us to validate the benefits of our approach on a machine translation task.
This paper proposes an adaptation of the SEARN algorithm to RNNs for generating text. In order to do so, they discuss various issues on how to scale the approach to large output vocabularies by sampling which actions the algorithm to explore.
Pros:
- Good literature review. But the future work on bandits is already happening:
Paper accepted at ACL 2017: Bandit Structured Prediction for Neural Sequence-to-Sequence Learning. Julia Kreutzer, Artem Sokolov, Stefan Riezler.
Cons:
- The key argument of the paper is that SEARNN is a better IL-inspired algorithm than the previously proposed ones. However there is no direct comparison either theoretical or empirical against them. In the examples on spelling using the dataset of Bahdanau et al. 2017, no comparison is made against their actor-critic method. Furthermore, given its simplicity, I would expect a comparison against scheduled sampling.
- A lot of important experimental details are in the appendices and they differ among experiments. For example, while mixed rollins are used in most experiments, reference rollins are used in MT, which is odd since it is a bad option theoretically. Also,  no details are given on how the mixing in the rollouts was tuned. Finally, in the NMT comparison while it is stated that similar architecture is used in order to compare fairly against previous work, this is not the case eventually, as it is acknowledged at least in the case of MIXER. I would have expected the same encoder-decoder architecture to have been used for all the methods considered.
 
- the two losses introduced are not really new. The log-loss is just MLE, only assuming that instead of a fixed expert that always returns the same target, we have a dynamic one. Note that the notion of dynamic expert is present in the SEARN paper too. Goldberg and Nivre just adapted it to transition-based dependency parsing. Similarly, since the KL loss is the same as XENT, why give it a new name?
- the top-k sampling method is essentially the same as the targeted exploration of Goodman et al. (2016) which the authors cite. Thus it is not a novel contribution.
  
- Not sure I see the difference between the stochastic nature of SEARNN and the online one of LOLS mentioned in section 7. They both could be mini-batched similarly. Also, not sure I see why SEARNN can be used on any task, in comparison to other methods. They all seem to be equally capable.
Minor comments:
- Figure 1: what is the difference between "cost-sensitive loss" and just "loss"?
- local vs sequence-level losses: the point in Ranzato et al and Wiseman & Rush is that the loss they optimizise (BLEU/ROUGE) do not decompose over the the predictions of the RNNs.
- Can't see why SEARNN can help with the vanishing gradient problem. Seem to be rather orthogonal.