MacNet: Transferring Knowledge from Machine Comprehension to Sequence-to-Sequence Models
Machine Comprehension (MC) is one of the core problems in natural language processing, requiring both understanding of the natural language and knowledge about the world. Rapid progress has been made since the release of several benchmark datasets, and recently the state-of-the-art models even surpass human performance on the well-known SQuAD evaluation. In this paper, we transfer knowledge learned from machine comprehension to the sequence-to-sequence tasks to deepen the understanding of the text. We propose MacNet: a novel encoder-decoder supplementary architecture to the widely used attention-based sequence-to-sequence models. Experiments on neural machine translation (NMT) and abstractive text summarization show that our proposed framework can significantly improve the performance of the baseline models, and our method for the abstractive text summarization achieves the state-of-the-art results on the Gigaword dataset.
Update after Author Feedback:  After reading all the reviews and the author feedback, I have two overall comments. The paper is branded as a transfer learning paper, but I'm left disappointed in this respect. I find it very surprising that the attention can be transferred at all, but it is such a small contribution to the MacNet Architecture's overall improvements, that it seems a hard sell. Focal losses have been used before and encoders have been transferred before, but they also contribute to performance improvements... Second comment: the ablations on summarization are necessary for a camera-ready version -- that seems like a hole right now, so I hope they are included in future versions. Overall, I'm still a 6 because you find a combination of things (with some surprising novelty) that improve performance, and it has shown that I should experiment with those things in the future.  Original Review:  Recent work (McCann et al 2017) had shown that the context vectors  (CoVe) of LSTMs trained as encoders for machine translation contained useful information for other tasks. The authors of this work show that CoVe from question answering encoders can be similarly used to augment performance of other tasks (Eq. 5). More surprisingly, and perhaps more relevant to the community, they also show that very specific higher layers of NLP models trained on one task can be transferred successfully to perform similar functions for other tasks (Eq. 12). This transfer is from question answering (machine comprehension) to both machine translation and summarization, substantial tasks, and experiments show that this transfer improves performance on both. They also demonstrate the increased effectiveness of using a focal loss inspired loss function (Eq. 15).   Pros:   While reading Section 4 MacNet Architecture, I found myself taking notes on the kinds of ablations I would want to see, and I was glad to find most of these ablations in Section 5.  The information conveyed by the writing was clear. MacNet as a package seems to clearly be superior to not using MacNet.  Not really a pro or a con:  I spent a while puzzling over the transfer of G_{enc}. It just isn’t clear to me why the authors would not use the same kinds of inputs G_{enc} was trained with, and the authors don’t justify why transferring by concatenation where the authors chose to is better than using those representations as additional embeddings.    Cons:  I worry a bit that the paper focuses mostly on discussion of transfer learning, but then nearly as much of the gain in performance for NMT comes from the alternative loss function in Eq. 15. In summarization, the performance gains are about 1 point across the board, but again, is this really just because of the alternative loss function? I would prefer to see the NMT ablations for summarization as well. The second biggest contributor to gains seems to be the Encoding layer for NMT, but this is the transfer that we shouldn’t be surprised would work given the prior work in this direction. That the Modeling layer helps at all is surprising, but on its own its at most it gives a gain of .4 on NMT; it is not clear how many runs it took to get that .4, so it seems possible that the transfer of the modeling layer doesn’t help much at all for most training runs.   Overall:  I’m left somewhat skeptical that any of the three contributions on their own are really very significant, but all together as MacNet, they work, there is one surprise, and I learned that I should experiment with a new loss for sequence generation tasks. I tend to think this makes for a good submission, but I’m skeptical about the significance of the transfer learning of the modeling layer. Given that seems to be the biggest novelty to me, this means I’m a bit hesitant about the overall contribution as well, which leads me to give it a 6. I’ve included some more detailed comments below with some additional suggestions and clarification questions. Given that the empirical results aren’t there yet to clear up some of my concerns, my confidence is 4, but I look forward to hearing from the authors during the rebuttal phase.  Combinations:  “We use a simple MLP classifier on the combination of {mj}nj=1 and G to locate the start and end positions of the answer.” (line 13) What is the combination here? Concatenation? Summation? Something else?  Suggested citation:  Question Answering through Transfer Learning from Large Fine-grained Supervision Data (Min et al. 2017) In that work, they show that pretraining on question answering datasets like SQuAD can benefit downstream tasks like SNLI. Its clearly very different from your work in key ways, but it is related in that it shows how QA models trained on SQuAD can transfer well to other tasks.   Citation spacing:  It looks like all of your citations would be better off with a space between them and the preceding text. Try using ~\cite instead of \cite (or corresponding ~\citep and ~\citet)  Pointer-Generator Comparison:  Are the examples in Table 5 randomly chosen? This would be worth mentioning. Otherwise, many will assume you’ve cherry-picked examples that make your model look better. If that is what you did, well… I would suggest randomly choosing them to be more fair.  Transparency in Equations:  In Eq. 2, you use f_{att} and cite Set et al. 2017, but it would be nice to include those equations in an appendix so that readers don’t have to then go lookup another paper just to see how your model is actually implemented.  Embeddings, tokenization, etc.:  Character-level embeddings are obtained by training a CNN over the characters of each word (lines 111-112). G_{enc} is then trained on GloVe and character embeddings from that CNN, which are summarized as f_{rep}. (Eq. 1) Then in Eq. 5, G_{enc} is used to take in Emb(x_s). The authors say that a fully connected layer transforms Emb(x_s) to the right size for G_{enc}, but this is left out of Eq. 5. It should be included there to avoid confusion.   This fully connected transformation is confusing because for NMT, unlike for MC, the data is split into subword units, so it is not clear that G_{enc} would have even seen many of the newly tokenized forms of the data. Why not use f_{rep} again for the inputs? How many subword units were used?  Enough to make sure that G_{enc} still sees mostly words that it saw during MC training? Similar questions arise again for the Pointer-Generator + MacNet. It seems like that fully connected layer has to learn the transformation from Emb(x_s) to f_{rep}, in which case, why have f_{rep} at all if Emb + fully connected can do that, and to reiterate why not just use f_{rep} instead of learning it?  Point of transfer for G_{enc}:  What if you just used \tilde e_s (Eq. 5) as additional embeddings the way that CoVe suggested? Instead of the concatenation with the other encoder?  Contribution of alternative loss:  It seems like Baseline + MacNet differs from Baseline + Encoding Layer + Modeling Layer only in the use of Eq. 15 in the former. Is this correct? If so, then this seems to be almost as important as the actual transfer learning in your final performance gains over the Baseline in Table 2. How much gain comes from just switching to that kind of loss? And how does this kind of ablation work with summarization? Is it all about the loss in that case?  I'm reserving very confident on reproducibility for code that is going to be open sourced. Anything with transfer learning and big tasks like NMT and summarization can be finnicky, so I'm sure people are bound to run into problems. The details seem to be in the paper so that wouldn't be the headache.