A significant roadblock in multilingual neural language modeling is the lack of labeled non-English data. One potential method for overcoming this issue is learning cross-lingual text representations that can be used to transfer the performance from training on English tasks to non-English tasks, despite little to no task-specific non-English data. In this paper, we explore a natural setup for learning crosslingual sentence representations: the dual-encoder. We provide a comprehensive evaluation of our cross-lingual representations on a number of monolingual, crosslingual, and zero-shot/few-shot learning tasks, and also give an analysis of different learned cross-lingual embedding spaces.
This paper proposes a novel cross-lingual multi-tasking framework based on a dual-encoder model that can learn cross-lingual sentence representations which are useful in monolingual tasks and cross-lingual tasks for both languages involved in the training, as observed on the experiments for three language pairs. The main idea of the approach is to model all tasks as input-response ranking tasks and introduce cross-lingual representation tying through the translation ranking task, introduced by Guo et al. (2018). All components of the framework are quite standard and deja-vu, but I like the paper in general, and the results seem quite encouraging. I have several comments on how to further strengthen the paper and improve the presentation of the main findings.
The proposed framework does not offer any substantial modeling contribution (i.e., all major components are based on SOTA models), but the framework is still quite interesting as a mixture of these SOTA components. I believe that some additional experiments would make the main contributions clearer and would also provide additional insights into the main properties of the proposed framework: 1) cross-linguality and 2) multi-tasking. 
*Most of all, I am surprised not to see any ablation studies. For instance, what happens if we remove one of the two monolingual tasks in each language? How does that reduced model compare to the full model? Which monolingual task is more beneficial for the final performance in downstream tasks? Can we think of adding another monolingual task to boost performance further? I think that this sort of experiment would be more beneficial for the paper than a pretty long analysis from Section 5 (this analysis is still valid, but should be shortened substantially). Evaluating only multi-tasking without any cross-lingual training would also be very beneficial to recognise the extent of improvement achieved by adding cross-linguality to the model.
*How much does the proposed architecture depend on the choice of the encoding model for the function g? Have the authors experimented with other (recent and (near-)SOTA) encoding models? I would like to see a comparative analysis of this 'hyper-parameter'.
*I would like to see more experiments on more distant language pairs. This would make the paper even more interesting imho. I am also curious whether there would be a drop in performance reported conditioned on the distance/proximity between two languages in a language pair.
*I would like to see a more detailed description of the two best performing STS systems (ECNU and BIT). In what respect are these systems state-of-the-art feature engineered and mixed? I am not sure what this means without providing any additional context to the claim and description.
*How does the monolingual English STS model trained with the cross-lingual multi-task framework compare to the work of Conneau et al. (EMNLP 2017) which also used SNLI as the task on which to learn universal sentence representations. This would be a good experiment imho as it would show how much we gain from cross-lingual training and multi-tasking.
Minor:
*Page 3: Could you add a short footnote discussing how hard-negatives for the translation ranking task are selected? How do you compute similarity here?
*Do you expect performance to improve further by training MultiNLI instead of SNLI (or combining the two datasets)?
*"All hyperparameters are tuned based on preliminary experiments on a development set." -> What is used as the development set? More details needed.
*"Finally, as an additional training heuristic, we multiply the gradients to the word and character embeddings by a factor of 100." -> How is the value for the embedding gradient multiplier determined? Is there an automatic procedure to fine-tune this hyper-parameter or has this been done in a completely empirical way?
*Table 1: please define the task abbreviations before showing them in the table. It is not clear what each task is by relying only on the abbreviation.
*This dataset was not available at the time of the submission, but for the revision it would make sense to also evaluate on the new XNLI dataset of Conneau et al. (EMNLP 2018) for multilingual NLI experiments.
(After the first revision) I have raised the score after the very detailed author response (thanks for that!), but this is also conditioned on the authors making the actual revisions promised in their response. I am still quite interested to check how well the method works in a setup with distant language pairs.