Understanding Isomorphism Bias in Graph Data Sets
In recent years there has been a rapid increase in classification methods on graph structured data. Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance. However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set. This prevents fair competition of the algorithms and raises a question of the validity of the obtained results. We analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.
The authors discuss here the problem of isomorphism bias in graph dataset, i.e. the overfitting effect in learning networks whenever graph isomorphism features are incorporated within the model. This is a bias which jeopardises the validity and the reproducibility of several studies, and it is theoretically analogous to data leakage effects. The authors fairly discuss the problem in the introduction, with a good coverage of the related literature; the background theory is reasonably discussed, although is not very deep. The experimental part is extensive and well described, and it shows the overfitting effect very clearly. However, the novelty of the work is limited, and also the proposed solutions cannot be claimed as superior to other approaches, due to the small improvement in accuracy.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding Isomorphism Bias in Graph Data Sets
In recent years there has been a rapid increase in classification methods on graph structured data. Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance. However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set. This prevents fair competition of the algorithms and raises a question of the validity of the obtained results. We analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors discuss here the problem of isomorphism bias in graph dataset, i.e. the overfitting effect in learning networks whenever graph isomorphism features are incorporated within the model. This is a bias which jeopardises the validity and the reproducibility of several studies, and it is theoretically analogous to data leakage effects. The authors fairly discuss the problem in the introduction, with a good coverage of the related literature; the background theory is reasonably discussed, although is not very deep. The experimental part is extensive and well described, and it shows the overfitting effect very clearly. However, the novelty of the work is limited, and also the proposed solutions cannot be claimed as superior to other approaches, due to the small improvement in accuracy.
assistant
The authors discuss here the problem of isomorphism bias in graph dataset, i.e. the overfitting effect in learning networks whenever graph isomorphism features are incorporated within the model. This is a bias which jeopardises the validity and the reproducibility of several studies, and it is theoretically analogous to data leakage effects. The authors fairly discuss the problem in the introduction, with a good coverage of the related literature; the background theory is reasonably discussed, although is not very deep. The experimental part is extensive and well described, and it shows the overfitting effect very clearly. However, the novelty of the work is limited, and also the proposed solutions cannot be claimed as superior to other approaches, due to the small improvement in accuracy.