Fooling Pre-trained Language Models: An Evolutionary Approach to Generate Wrong Sentences with High Acceptability Score
Large pre-trained language representation models have recently collected numerous successes in language understanding. They obtained state-of-the-art results in many classical benchmark datasets, such as GLUE benchmark and SQuAD dataset, but do they really understand the language? In this paper we investigate two among the best pre-trained language models, BERT and RoBERTa, analysing their weaknesses by generating adversarial sentences in an evolutionary approach. Our goal is to discover if and why it is possible to fool these models, and how to face this issue. This adversarial attack is followed by a cross analysis, understanding robustness and generalization proprieties of models and fooling techniques. We find that BERT can be easily fooled, but an augmentation of the original dataset with adversarial samples is enough to make it learn how not to be fooled again. RoBERTa, instead, is more resistent to this approach even if it still have some weak spots.
In this paper, the authors present a method and analysis for generating an artificial datatset to fool pre-trained language models into classifying the sentences, which are random strings of words, as being grammatical. They test their method on BERT and RoBERTa. They take (Ro)BERT(a) that is fine-tuned on CoLA and use it to select *sentences* that fool the model. The sentences they use are random strings of words. The method is evolutionary: the top 100 of the first n generations of sentences if selected to spawn off 10 new, slightly altered, sentences, where the sentences are ranked according to the probability of them being classified as grammatical. I would recommend this paper be rejected. Here are some of the issues with this paper, - The results section provides no hard numbers. The paper only descriptions of the results like *obtaining a result similar to the original score* and *the models are always fooled.* This alone is a disqualifying factor - The presented algorithm adversarially selects random strings of words that fool (Ro)BERT(a). From my reading, it seems like this is used as partial evidence that (Ro)BERT(a) is incapable of distinguishing between grammatical sentences and random sentences. The authors claim that since the sentences that are generated from the algorithm are ungrammatical strings of random words, the models can not distinguish *correct sentences and selected random collections of words* (section 6.2). I don*t think this is true. The algorithm is explicitly designed to pick out these random strings. The fact that an algorithm that does random word selection and substitution doesn*t land on grammatically sound sentences is not evidence that the models used in loop are failing at learning concepts of grammaticality. (I*d like to be clear here and say I*m not claiming that these models fully understand grammaticality, I just don*t believe this is good evidence that they don*t.) - The sentences in Table 2 don*t appear grammatical to me, though the authors say that they are. Example from table 2, *NY restaurant student Shelly dove lack.* These results are used to claim that sometimes the algorithm does converge on grammatical sentences, these conclusion seems unfounded to me. - In section 6.4 the authors claim that if a BERT model fine-tuned with some of the algorithm*s generated sentences is not *fooled,* i.e. if the algorithm with this model converges on grammatical sentences, then the *dataset was not built to deal with random sentences.* I think that this is trivially true. CoLA has no random sentences in it, so random sentences are naturally outside the dataset distribution. - In section 6.5.2 it is unclear if there is a held out test set of *fooling* sentences. The paper concludes that RoBERTa is more robust to adversarial attacks than BERT. This would be a neat conclusion but I*m afraid there is no concrete evidence of that presented in the paper. The biggest issue with the paper is the lack of empirical results, we are given no numbers to demonstrate performance making the results impossible to interpret. Smaller issues: - Figure 1 is unnecessary as it doesn*t offer additional clarification of the algorithm. - There is some odd phrasing through the paper. I*d recommend getting some edits from another native English speaker to help polish the paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fooling Pre-trained Language Models: An Evolutionary Approach to Generate Wrong Sentences with High Acceptability Score
Large pre-trained language representation models have recently collected numerous successes in language understanding. They obtained state-of-the-art results in many classical benchmark datasets, such as GLUE benchmark and SQuAD dataset, but do they really understand the language? In this paper we investigate two among the best pre-trained language models, BERT and RoBERTa, analysing their weaknesses by generating adversarial sentences in an evolutionary approach. Our goal is to discover if and why it is possible to fool these models, and how to face this issue. This adversarial attack is followed by a cross analysis, understanding robustness and generalization proprieties of models and fooling techniques. We find that BERT can be easily fooled, but an augmentation of the original dataset with adversarial samples is enough to make it learn how not to be fooled again. RoBERTa, instead, is more resistent to this approach even if it still have some weak spots.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this paper, the authors present a method and analysis for generating an artificial datatset to fool pre-trained language models into classifying the sentences, which are random strings of words, as being grammatical. They test their method on BERT and RoBERTa. They take (Ro)BERT(a) that is fine-tuned on CoLA and use it to select *sentences* that fool the model. The sentences they use are random strings of words. The method is evolutionary: the top 100 of the first n generations of sentences if selected to spawn off 10 new, slightly altered, sentences, where the sentences are ranked according to the probability of them being classified as grammatical. I would recommend this paper be rejected. Here are some of the issues with this paper, - The results section provides no hard numbers. The paper only descriptions of the results like *obtaining a result similar to the original score* and *the models are always fooled.* This alone is a disqualifying factor - The presented algorithm adversarially selects random strings of words that fool (Ro)BERT(a). From my reading, it seems like this is used as partial evidence that (Ro)BERT(a) is incapable of distinguishing between grammatical sentences and random sentences. The authors claim that since the sentences that are generated from the algorithm are ungrammatical strings of random words, the models can not distinguish *correct sentences and selected random collections of words* (section 6.2). I don*t think this is true. The algorithm is explicitly designed to pick out these random strings. The fact that an algorithm that does random word selection and substitution doesn*t land on grammatically sound sentences is not evidence that the models used in loop are failing at learning concepts of grammaticality. (I*d like to be clear here and say I*m not claiming that these models fully understand grammaticality, I just don*t believe this is good evidence that they don*t.) - The sentences in Table 2 don*t appear grammatical to me, though the authors say that they are. Example from table 2, *NY restaurant student Shelly dove lack.* These results are used to claim that sometimes the algorithm does converge on grammatical sentences, these conclusion seems unfounded to me. - In section 6.4 the authors claim that if a BERT model fine-tuned with some of the algorithm*s generated sentences is not *fooled,* i.e. if the algorithm with this model converges on grammatical sentences, then the *dataset was not built to deal with random sentences.* I think that this is trivially true. CoLA has no random sentences in it, so random sentences are naturally outside the dataset distribution. - In section 6.5.2 it is unclear if there is a held out test set of *fooling* sentences. The paper concludes that RoBERTa is more robust to adversarial attacks than BERT. This would be a neat conclusion but I*m afraid there is no concrete evidence of that presented in the paper. The biggest issue with the paper is the lack of empirical results, we are given no numbers to demonstrate performance making the results impossible to interpret. Smaller issues: - Figure 1 is unnecessary as it doesn*t offer additional clarification of the algorithm. - There is some odd phrasing through the paper. I*d recommend getting some edits from another native English speaker to help polish the paper.
assistant
In this paper, the authors present a method and analysis for generating an artificial datatset to fool pre-trained language models into classifying the sentences, which are random strings of words, as being grammatical. They test their method on BERT and RoBERTa. They take (Ro)BERT(a) that is fine-tuned on CoLA and use it to select *sentences* that fool the model. The sentences they use are random strings of words. The method is evolutionary: the top 100 of the first n generations of sentences if selected to spawn off 10 new, slightly altered, sentences, where the sentences are ranked according to the probability of them being classified as grammatical. I would recommend this paper be rejected. Here are some of the issues with this paper, - The results section provides no hard numbers. The paper only descriptions of the results like *obtaining a result similar to the original score* and *the models are always fooled. * This alone is a disqualifying factor - The presented algorithm adversarially selects random strings of words that fool (Ro)BERT(a). From my reading, it seems like this is used as partial evidence that (Ro)BERT(a) is incapable of distinguishing between grammatical sentences and random sentences. The authors claim that since the sentences that are generated from the algorithm are ungrammatical strings of random words, the models can not distinguish *correct sentences and selected random collections of words* (section 6.2). I don*t think this is true. The algorithm is explicitly designed to pick out these random strings. The fact that an algorithm that does random word selection and substitution doesn*t land on grammatically sound sentences is not evidence that the models used in loop are failing at learning concepts of grammaticality. (I*d like to be clear here and say I*m not claiming that these models fully understand grammaticality, I just don*t believe this is good evidence that they don*t.) - The sentences in Table 2 don*t appear grammatical to me, though the authors say that they are. Example from table 2, *NY restaurant student Shelly dove lack. * These results are used to claim that sometimes the algorithm does converge on grammatical sentences, these conclusion seems unfounded to me. - In section 6.4 the authors claim that if a BERT model fine-tuned with some of the algorithm*s generated sentences is not *fooled,* i.e. if the algorithm with this model converges on grammatical sentences, then the *dataset was not built to deal with random sentences. * I think that this is trivially true. CoLA has no random sentences in it, so random sentences are naturally outside the dataset distribution. - In section 6.5.2 it is unclear if there is a held out test set of *fooling* sentences. The paper concludes that RoBERTa is more robust to adversarial attacks than BERT. This would be a neat conclusion but I*m afraid there is no concrete evidence of that presented in the paper. The biggest issue with the paper is the lack of empirical results, we are given no numbers to demonstrate performance making the results impossible to interpret. Smaller issues: - Figure 1 is unnecessary as it doesn*t offer additional clarification of the algorithm. - There is some odd phrasing through the paper. I*d recommend getting some edits from another native English speaker to help polish the paper.