Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.
This paper introduces a method for generating training/test data for measuring the model*s ability of *compositional generalization* in complex compositional tasks/domains such as natural language understanding, visual understanding and other domains. The idea of the proposed method is to essentially keep the *atom* distribution unchanged between the training and test data, but maximize the divergence between the *compound* distributions between them. The authors have conducted a thorough and systematic experimentation, comparing the proposed approach with a number of other heuristic approaches for train test splits (such as random and input/output length, etc.) and using both a new large data set they generated (CFQ) and existing data set (SCAN). The experimental results verify that using their method they can obtain train test data sets with uniform atom distributions with large divergence in compound distributions, and they find that there is a surprisingly large negative correlation between the accuracy of existing state-of-the-art learning methods and the compound divergence. The data generation mechanism is systematic and involved, consisting of different categories of rules (logical form, grammar, rule application DAG*s, etc.) and it would seem that the generation method/system and the generated data would be useful as benchmark data for the community. The paper lacks technical novelty other than the training and test data generation approach, but having one available to the community with these apparently desirable characteristics as benchmark data for measuring complex, compositional generalization capabilities, and that could be invaluable to the research community.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper introduces a method for generating training/test data for measuring the model*s ability of *compositional generalization* in complex compositional tasks/domains such as natural language understanding, visual understanding and other domains. The idea of the proposed method is to essentially keep the *atom* distribution unchanged between the training and test data, but maximize the divergence between the *compound* distributions between them. The authors have conducted a thorough and systematic experimentation, comparing the proposed approach with a number of other heuristic approaches for train test splits (such as random and input/output length, etc.) and using both a new large data set they generated (CFQ) and existing data set (SCAN). The experimental results verify that using their method they can obtain train test data sets with uniform atom distributions with large divergence in compound distributions, and they find that there is a surprisingly large negative correlation between the accuracy of existing state-of-the-art learning methods and the compound divergence. The data generation mechanism is systematic and involved, consisting of different categories of rules (logical form, grammar, rule application DAG*s, etc.) and it would seem that the generation method/system and the generated data would be useful as benchmark data for the community. The paper lacks technical novelty other than the training and test data generation approach, but having one available to the community with these apparently desirable characteristics as benchmark data for measuring complex, compositional generalization capabilities, and that could be invaluable to the research community.
assistant
This paper introduces a method for generating training/test data for measuring the model*s ability of *compositional generalization* in complex compositional tasks/domains such as natural language understanding, visual understanding and other domains. The idea of the proposed method is to essentially keep the *atom* distribution unchanged between the training and test data, but maximize the divergence between the *compound* distributions between them. The authors have conducted a thorough and systematic experimentation, comparing the proposed approach with a number of other heuristic approaches for train test splits (such as random and input/output length, etc.) and using both a new large data set they generated (CFQ) and existing data set (SCAN). The experimental results verify that using their method they can obtain train test data sets with uniform atom distributions with large divergence in compound distributions, and they find that there is a surprisingly large negative correlation between the accuracy of existing state-of-the-art learning methods and the compound divergence. The data generation mechanism is systematic and involved, consisting of different categories of rules (logical form, grammar, rule application DAG*s, etc.) and it would seem that the generation method/system and the generated data would be useful as benchmark data for the community. The paper lacks technical novelty other than the training and test data generation approach, but having one available to the community with these apparently desirable characteristics as benchmark data for measuring complex, compositional generalization capabilities, and that could be invaluable to the research community.