Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
Generating novel graph structures that optimize given objectives while obeying some given underlying rules is fundamental for chemistry, biology and social science research. This is especially important in the task of molecular graph generation, whose goal is to discover novel molecules with desired properties such as drug-likeness and synthetic accessibility, while obeying physical laws such as chemical valency. However, designing models to find molecules that optimize desired properties while incorporating highly complex and non-differentiable rules remains to be a challenging task. Here we propose Graph Convolutional Policy Network (GCPN), a general graph convolutional network based model for goaldirected graph generation through reinforcement learning. The model is trained to optimize domain-specific rewards and adversarial loss through policy gradient, and acts in an environment that incorporates domain-specific rules. Experimental results show that GCPN can achieve 61% improvement on chemical property optimization over state-of-the-art baselines while resembling known molecules, and achieve 184% improvement on the constrained property optimization task.
The paper proposes combining graph neural nets, RL, and adversarial learning to train a molecule generation procedure optimized according to specific task requirements. RL is used to optimize non-differentiable objectives. Adversarial learning is used to encourage the policy to generate molecules that are similar to known ones. Graph neural nets are used to capture the graph structure of the molecules. Results are shown for three different tasks -- optimizing a specific property of the generated molecule, generating molecules that satisfy constraints on property scores, and generating molecules that satisfy constraints on their composition. ZINC dataset is used for training. Results show the proposed approach outperforming other learning based approaches.  Strengths: - The paper is well-written and easy to read. - Although the proposed approach is a straightforward combination of existing ideas, it is still interesting to explore the performance of such a combination on the kind of tasks considered in the paper.  Weaknesses: - In the evaluation, there is no comparison to simple baselines with no learning, like Genetic Algorithms, Hill Climbing, random search, etc. which makes it difficult to evaluate how much benefit does sophisticated learning-based approaches provide.  - In the optimization tasks, the running time, computational budget, and the number of function evaluations should all be controlled across the various algorithms (at least approximately) so that the comparisons are fair. There is no mention of these factors in the paper. Since each optimization algorithm represents a trade-off between solution quality and time/computation/evaluation budget, it is not clear where on that trade-off curve each algorithm being compared is and whether they are even comparable. (If one only considers solution quality, even brute force search will do well.) This makes it hard to decide whether the results are interesting.  Comments/Questions: - How are the potentially unstable learning dynamics of adversarial learning and RL handled? Insights into the challenges can be very useful.  - Lines 171-172 mention using "small" positive and negative rewards to discourage infeasible actions. What value is determined to be "small" and how is it determined?  - If the policy becomes deterministic, then it will always generate the same molecule. How is the policy kept stochastic so that it generates diverse molecules?  - There is a missing reference in the Supplementary Material.