Using Deep Reinforcement Learning to Generate Rationales for Molecules
Deep learning algorithms are increasingly used in modeling chemical processes. However, black box predictions without rationales have limited used in practical applications, such as drug design. To this end, we learn to identify molecular substructures -- rationales -- that are associated with the target chemical property (e.g., toxicity). The rationales are learned in an unsupervised fashion, requiring no additional information beyond the end-to-end task. We formulate this problem as a reinforcement learning problem over the molecular graph, parametrized by two convolution networks corresponding to the rationale selection and prediction based on it, where the latter induces the reward function. We evaluate the approach on two benchmark toxicity datasets. We demonstrate that our model sustains high performance under the additional constraint that predictions strictly follow the rationales. Additionally, we validate the extracted rationales through comparison against those described in chemical literature and through synthetic experiments.
 The paper proposes a feature learning technique for molecular prediction using reinforcement learning. The predictive model is an interesting two-step approach where important atoms of the molecule are added one-by-one with a reward given by a second Q-network that learns how well we can solve the prediction problem with the given set of atoms. The overall scheme is intuitive, but The model is experimented on two small datasets of few thousand of molecules, and compared to a state-of-the-art DeepTox, and also to some basic baselines (RF/SVM/logreg). In the Tox21 dataset the proposed sparse RL-CNN method is less accurate than DeepTox or full CNN. In the hERG dataset RL-CNN is again weaker than the full CNN, but also seems to be beaten by several baseline methods. Overall the results are surprisingly weak, since e.g. with LASSO one often improves by using less features in complex problems. Both datasets should be compared to LASSO as well. It*s somewhat odd that the test performance in table 2 is often better than CV performance. This feels suspicious, especially with 79.0 vs 84.3. The table 2 does not seem reliable result, and should use more folds and more randomizations, etc. The key problem of the method is its seeming inabability to find the correct number of atoms to use. In both datasets the number of atoms were globally fixed, which is counter-intuitive. The authors should at least provide learning curves where different number of atoms are used; but ideally the method should learn the number of atoms to use for each molecule. The proposed Q+P network is interesting, but its unclear how well it works in general. There should be experiments that compare the the Q+P model with incresing number of atoms against a full CNN, to see whether the Q+P can converge to maximal performance. Overall the method is interesting and has a clear impact for molecular prediction, however the paper has limited appeal to the broader audience. Its difficult to assess how useful the Q/P-network is in general. The inability to choose the optimal number of atoms is a major drawback of the method, and the experimental section could be improved. This paper also would probably be more suitable for a chemoinformatics journal, where the rationale learning would be highly appreciated.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Using Deep Reinforcement Learning to Generate Rationales for Molecules
Deep learning algorithms are increasingly used in modeling chemical processes. However, black box predictions without rationales have limited used in practical applications, such as drug design. To this end, we learn to identify molecular substructures -- rationales -- that are associated with the target chemical property (e.g., toxicity). The rationales are learned in an unsupervised fashion, requiring no additional information beyond the end-to-end task. We formulate this problem as a reinforcement learning problem over the molecular graph, parametrized by two convolution networks corresponding to the rationale selection and prediction based on it, where the latter induces the reward function. We evaluate the approach on two benchmark toxicity datasets. We demonstrate that our model sustains high performance under the additional constraint that predictions strictly follow the rationales. Additionally, we validate the extracted rationales through comparison against those described in chemical literature and through synthetic experiments.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper proposes a feature learning technique for molecular prediction using reinforcement learning. The predictive model is an interesting two-step approach where important atoms of the molecule are added one-by-one with a reward given by a second Q-network that learns how well we can solve the prediction problem with the given set of atoms. The overall scheme is intuitive, but The model is experimented on two small datasets of few thousand of molecules, and compared to a state-of-the-art DeepTox, and also to some basic baselines (RF/SVM/logreg). In the Tox21 dataset the proposed sparse RL-CNN method is less accurate than DeepTox or full CNN. In the hERG dataset RL-CNN is again weaker than the full CNN, but also seems to be beaten by several baseline methods. Overall the results are surprisingly weak, since e.g. with LASSO one often improves by using less features in complex problems. Both datasets should be compared to LASSO as well. It*s somewhat odd that the test performance in table 2 is often better than CV performance. This feels suspicious, especially with 79.0 vs 84.3. The table 2 does not seem reliable result, and should use more folds and more randomizations, etc. The key problem of the method is its seeming inabability to find the correct number of atoms to use. In both datasets the number of atoms were globally fixed, which is counter-intuitive. The authors should at least provide learning curves where different number of atoms are used; but ideally the method should learn the number of atoms to use for each molecule. The proposed Q+P network is interesting, but its unclear how well it works in general. There should be experiments that compare the the Q+P model with incresing number of atoms against a full CNN, to see whether the Q+P can converge to maximal performance. Overall the method is interesting and has a clear impact for molecular prediction, however the paper has limited appeal to the broader audience. Its difficult to assess how useful the Q/P-network is in general. The inability to choose the optimal number of atoms is a major drawback of the method, and the experimental section could be improved. This paper also would probably be more suitable for a chemoinformatics journal, where the rationale learning would be highly appreciated.
assistant
 The paper proposes a feature learning technique for molecular prediction using reinforcement learning. The predictive model is an interesting two-step approach where important atoms of the molecule are added one-by-one with a reward given by a second Q-network that learns how well we can solve the prediction problem with the given set of atoms. The overall scheme is intuitive, but The model is experimented on two small datasets of few thousand of molecules, and compared to a state-of-the-art DeepTox, and also to some basic baselines (RF/SVM/logreg). In the Tox21 dataset the proposed sparse RL-CNN method is less accurate than DeepTox or full CNN. In the hERG dataset RL-CNN is again weaker than the full CNN, but also seems to be beaten by several baseline methods. Overall the results are surprisingly weak, since e.g. with LASSO one often improves by using less features in complex problems. Both datasets should be compared to LASSO as well. It*s somewhat odd that the test performance in table 2 is often better than CV performance. This feels suspicious, especially with 79.0 vs 84.3. The table 2 does not seem reliable result, and should use more folds and more randomizations, etc. The key problem of the method is its seeming inabability to find the correct number of atoms to use. In both datasets the number of atoms were globally fixed, which is counter-intuitive. The authors should at least provide learning curves where different number of atoms are used; but ideally the method should learn the number of atoms to use for each molecule. The proposed Q+P network is interesting, but its unclear how well it works in general. There should be experiments that compare the the Q+P model with incresing number of atoms against a full CNN, to see whether the Q+P can converge to maximal performance. Overall the method is interesting and has a clear impact for molecular prediction, however the paper has limited appeal to the broader audience. Its difficult to assess how useful the Q/P-network is in general. The inability to choose the optimal number of atoms is a major drawback of the method, and the experimental section could be improved. This paper also would probably be more suitable for a chemoinformatics journal, where the rationale learning would be highly appreciated.