The use of deep pre-trained transformers has led to remarkable progress in a number of applications . For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.
Summary: This work proposes a new transformer architecture for tasks that involve a query sequence and multiple candidate sequences. The proposed architecture, called poly-encoder, strikes a balance between a dual encoder which independently encodes the query and candidate and combines representations at the top, and a more expressive architecture which does full joint attention over the concatenated query and candidate sequences. Experiments on utterance retrieval tasks for dialog and an information retrieval task show that poly-encoders strike a good trade-off between the inference speed of the dual encoder model and the performance of the full attention model. 
Pros:
- Strong results compared to baselines on multiple dialog and retrieval tasks. 
- Detailed discussion of hyperparameter choices and good ablations.
- Paper is well written and easy to follow.
Cons:
- Limited novelty of methods. Ideas similar to the model variants discussed in this work have been considered in other work (Eg: [1]). It is also known that in-domain pre-training (i.e, pre-training on data close to the downstream task’s data distribution) helps (Eg: [2]). So this work can be considered as an application of existing ideas to dialog tasks. 
- In terms of impact, utterance retrieval has fairly limited applicability in dialog. The dialog tasks considered in this work have a maximum of 100 candidate utterances, whereas in practice, the space of possible responses is much larger. While retrieval models are useful, I am skeptical about the practical value of the improvements shown in the paper (especially the improvements over bi-encoder, which is already a decent model).
Suggestions:
One way to get around the inefficiency of the cross-encoder architecture is to first use an inexpensive scoring mechanism such as TFIDF or bi-encoder to identify a small number of promising candidates from all the possible candidates. We can then use the cross-encoder to do more precise scoring of only the promising candidates. I am curious where a pipelined model such as this compares against the variants discussed in the paper in terms of speed and performance. 
While the paper presents strong results on several dialog utterance retrieval tasks, the methods presented have limited novelty and impact. I am hence leaning towards borderline. 
References
[1] Logeswaran Lajanugen, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
[2] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Edit: I have read the author response. Based on the rebuttal, I am more convinced about the practical impact of the approach. I am raising my score and recommending accept.