Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new multitask question answering network (MQAN) that jointly learns all tasks in decaNLP without any task-specific modules or parameters more effectively than sequence-to-sequence and reading comprehension baselines. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and that performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.
I appreciate the work that went into creating this paper, but I'm afraid I see little justification for accepting it.  I have three major complaints with this paper:                                                                         
                                                                                                     
1. I think the framing of decaNLP presented in this paper does more harm than good, because it perpetuates a misguided view of question answering.
                                                                                                     
Question answering is not a unified phenomenon.  There is no such thing as "general question answering", not even for humans.  Consider "What is 2 + 3?", "What's the terminal velocity of a rain drop?", and "What is the meaning of life?"  All of these questions require very different systems to answer, and trying to pretend they are the same doesn't help anyone solve any problems.
                                                                                                     
Question answering is a _format_ for studying particular phenomena.  Sometimes it is useful to pose a task as QA, and sometimes it is not.  QA is not a useful format for studying problems when you only have a single question (like "what is the sentiment?" or "what is the translation?"), and there is no hope of transfer from a related task.  Posing translation or classification as QA serves no useful purpose and gives people the wrong impression about question answering as a format for studying problems.
We have plenty of work that studies multiple datasets at a time (including in the context of semi-supervised / transfer learning), without doing this misguided framing of all of them as QA (see, e.g., the ELMo and BERT papers, which evaluated on many separate tasks).  I don't see any compelling justification for setting things up this way.
                                                                                                     
2. One of the main claims of this paper is transfer from one task to another by posing them all as question answering.  There is nothing new in the transfer results that were presented here, however.  For QA-SRL / QA-ZRE, transfer from SQuAD / other QA tasks has already been shown by Luheng He (http://aclweb.org/anthology/N18-2089) and Omer Levy (that was the whole point of the QA-ZRE paper), so this is merely reproducing that result (without mentioning that they did it first).  For all other tasks, performance drops when you try to train all tasks together, sometimes significantly (as in translation, unsurprisingly).  For the Czech task, fine tuning a pre-trained model has already been shown to help.  Transfer from MNLI to SNLI is known already and not surprising - one of the main points of MNLI was domain transfer, so obviously this has been studied before.  The claims about transfer to new classification tasks are misleading, as you really have the _same_ classification task, you've just arbitrarily changed how you're encoding the class label.  It _might_ be the case that you still get transfer if you actually switch to a related classification task, but you haven't examined that case.
                                                                                                     
3. This paper tries to put three separate ideas into a single conference paper, and all three ideas suffer as a result, because there is not enough space to do any of them justice.  Giving 15 pages of appendix for an 8 page paper, where some of the main content of the paper is pushed to the appendix, is egregious.  Putting your work in the context of related work is not something that should be pushed into an appendix, and we should not encourage this behavior.
                                                                                                     
The three ideas here seem to me to be (1) decaNLP, (2) the model architecture of MQAN, (3) transfer results.  Any of these three could have been a single conference paper, had it been done well.  As it stands, decaNLP isn't described or motivated well enough, and there isn't any space left in the paper to address my severe criticisms of it in my first point.  Perhaps if you had dedicated the paper to decaNLP, you could have given arguments that the framing is worthwhile, and described the tasks and their setup as QA sufficiently (as it is, I don't see any description anywhere of how the context is constructed for WikiSQL; did I miss it somewhere?).  For MQAN, there's more than a page of the core new architecture that's pushed into the appendix.  And for the transfer results, there is very little comparison to other transfer methods (e.g., ELMo, CoVe), or any deep analysis of what's going on - as I mentioned above, basically all of the results presented are just confirming what has already been done elsewhere.