Detecting Egregious Responses in Neural Sequence-to-sequence Models
In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq) model to generate egregious outputs (aggressive, malicious, attacking, etc.). And if such inputs exist, how to find them efficiently. We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them. Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users. In our experiments, we apply this approach to dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses. We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.
 # Positive aspects of this submission - This submission explores a very interesting problem that is often overlooked in sequence-to-sequence models research. - The methodology in Sections 4 and 5 is very thorough and useful. - Good comparison of last-h with attention representations, which gives good insight about the robustness of each architecture against adversarial attacks. # Criticism - In Section 3, even if the "l1 + projection" experiments seem to show that generating egregious outputs with greedy decoding is very unlikely, it doesn*t definitely prove so. It could be that your discrete optimization algorithm is suboptimal, especially given that other works on adversarial attacks for seq2seq models use different methods such as gradient regularization (Cheng et al. 2018). Similarly, the brute-force results on a simplified task in Appendix B are useful, but it*s hard to tell whether the conclusions of this experiment can be extrapolated to the original dialog task. Given that you also study "o-greedy-hit" in more detail with a different algorithm in Sections 4 and 5, I would consider removing Section 3 or moving it to the Appendix for consistency.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Detecting Egregious Responses in Neural Sequence-to-sequence Models
In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq) model to generate egregious outputs (aggressive, malicious, attacking, etc.). And if such inputs exist, how to find them efficiently. We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them. Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users. In our experiments, we apply this approach to dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses. We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 # Positive aspects of this submission - This submission explores a very interesting problem that is often overlooked in sequence-to-sequence models research. - The methodology in Sections 4 and 5 is very thorough and useful. - Good comparison of last-h with attention representations, which gives good insight about the robustness of each architecture against adversarial attacks. # Criticism - In Section 3, even if the "l1 + projection" experiments seem to show that generating egregious outputs with greedy decoding is very unlikely, it doesn*t definitely prove so. It could be that your discrete optimization algorithm is suboptimal, especially given that other works on adversarial attacks for seq2seq models use different methods such as gradient regularization (Cheng et al. 2018). Similarly, the brute-force results on a simplified task in Appendix B are useful, but it*s hard to tell whether the conclusions of this experiment can be extrapolated to the original dialog task. Given that you also study "o-greedy-hit" in more detail with a different algorithm in Sections 4 and 5, I would consider removing Section 3 or moving it to the Appendix for consistency.
assistant
 # Positive aspects of this submission - This submission explores a very interesting problem that is often overlooked in sequence-to-sequence models research. - The methodology in Sections 4 and 5 is very thorough and useful. - Good comparison of last-h with attention representations, which gives good insight about the robustness of each architecture against adversarial attacks. # Criticism - In Section 3, even if the "l1 + projection" experiments seem to show that generating egregious outputs with greedy decoding is very unlikely, it doesn*t definitely prove so. It could be that your discrete optimization algorithm is suboptimal, especially given that other works on adversarial attacks for seq2seq models use different methods such as gradient regularization (Cheng et al. 2018). Similarly, the brute-force results on a simplified task in Appendix B are useful, but it*s hard to tell whether the conclusions of this experiment can be extrapolated to the original dialog task. Given that you also study "o-greedy-hit" in more detail with a different algorithm in Sections 4 and 5, I would consider removing Section 3 or moving it to the Appendix for consistency.