summaries: The paper proposes a GNN-based extension of transformers, which have been shown to be effective for reaction prediction etc. before. In particular, the GNN-based embedding of molecules in the reaction embeddings overcomes the artificial bias inherent in the often applied sequence embeddings. The experiments show that there are sometimes increases of performance in reaction and retrosynthesis prediction - and the approach could be applied to similar problems.    This paper proposes a graph-to-sequence architecture called Graph2SMILES for the retrosynthesis and the reaction outcome prediction. Graph2SMILES uses an attention-augmented D-MPNN encoder to capture the local information and a global attention encoder with graph-aware positional embeddings to capture the global information. Experiments show that Graph2SMILES is competitive with Transformer baselines but does not outperform state-of-the-art methods on tasks of the one-step retrosynthesis and the reaction outcome prediction.  The authors proposed a new method for retrosynthesis, which does not require the mapping numbers and extracting templates from the literature.  Basically, the model consists of a graph-based encoder and a sequence based encoder. The encoder consists of local aggregation from neighbors and global attention using a new positional method. The decoder is a Transformer model with relative positional encoding.  The method achieved promising results on several retrosynthesis datasets.  This paper proposes a graph-to-SMILES framework, which incorporates several recently developed engineering techniques from the community, for synthesis planning and reaction outcome prediction tasks. The proposed method leverages graph neural networks and Transformer attention model to encode the graph inputs and then utilizes a Transformer decoder to generate the SMILES string as outputs. Experiments on benchmark retrosynthesis and reaction prediction tasks show that the proposed approach outperformed the vanilla SMILES-to-SMILES transformer baseline, but obtained inferior results than some other advanced methods. The paper is interesting, but both the technical novelty and the experimental studies are weak to me.
abstractText: Synthesis planning and reaction outcome prediction are two fundamental problems in computer-aided organic chemistry for which a variety of data-driven approaches have emerged. Natural language approaches that model each problem as a SMILES-to-SMILES translation lead to a simple end-to-end formulation, reduce the need for data preprocessing, and enable the use of well-optimized machine translation model architectures. However, SMILES representations are not an efficient representation for capturing information about molecular structures, as evidenced by the success of SMILES augmentation to boost empirical performance. Here, we describe a novel Graph2SMILES model that combines the power of Transformer models for text generation with the permutation invariance of molecular graph encoders that mitigates the need for input data augmentation. As an end-to-end architecture, Graph2SMILES can be used as a drop-in replacement for the Transformer in any task involving molecule(s)-to-molecule(s) transformations. In our encoder, an attention-augmented directed message passing neural network (D-MPNN) captures local chemical environments, and the global attention encoder allows for long-range and intermolecular interactions, enhanced by graph-aware positional embedding. Graph2SMILES improves the top-1 accuracy of the Transformer baselines by 1.7% and 1.9% for reaction outcome prediction on USPTO_480k and USPTO_STEREO datasets respectively, and by 9.8% for one-step retrosynthesis on the USPTO_50k dataset.
references.authors: Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio
references.title: Neural machine translation by jointly learning to align and translate
references.venue: In 3rd International Conference on Learning Representations,
references.authors: Javier L. Baylon Nicholas A. Cilfone Jeffrey R. Gulcher Thomas W. Chittenden
references.title: Enhancing retrosynthetic reaction prediction with deep learning using multiscale reaction classification
references.venue: Journal of Chemical Information and Modeling,
references.authors: Hangrui Bi Hengyi Wang Chence Shi Connor Coley Jian Tang Hongyu Guo
references.title: Nonautoregressive electron redistribution modeling for reaction prediction
references.venue: In Proceedings of the 38th International Conference on Machine Learning,
references.authors: Esben Jannik Bjerrum
references.title: SMILES enumeration as data augmentation for neural network modeling of molecules
references.authors: John Bradshaw Matt J. Kusner Brooks Paige Marwin H.S. Segler José Miguel
references.title: HernándezLobato. A generative model for electron paths
references.venue: In International Conference on Learning Representations,
references.authors: Shaked Brody Uri Alon Eran Yahav
references.title: How attentive are graph attention networks? CoRR, 2021
references.venue: URL https://arxiv.org/abs/2105.14491
references.authors: Deng Cai Wai Lam
references.title: Graph transformer for graph-to-sequence learning
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Benson Chen Regina Barzilay Tommi S. Jaakkola
references.title: Path-augmented graph transformer network. CoRR, 2019
references.venue: URL http://arxiv.org/abs/1905.12712
references.authors: Benson Chen Tianxiao Shen Tommi S. Jaakkola Regina Barzilay
references.title: Learning to make generalizable and diverse predictions for retrosynthesis
references.authors: Shuan Chen Yousung Jung
references.title: Deep retrosynthetic reaction prediction using local reactivity and global attention
references.venue: JACS Au,
references.authors: Kyunghyun Cho Bart van Merriënboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using RNN encoder–decoder for statistical machine translation
references.venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Connor W. Coley Luke Rogers William H. Green Klavs F. Jensen
references.title: Computer-assisted retrosynthesis based on molecular similarity
references.venue: ACS Central Science,
references.authors: Connor W. Coley Wengong Jin Luke Rogers Timothy F. Jamison Tommi S. Jaakkola William H. Green Regina Barzilay Klavs F. Jensen
references.title: A graph-convolutional neural network model for the prediction of chemical reactivity
references.venue: Chem. Sci.,
references.authors: E.J. Corey W. Todd Wipke
references.title: Computer-assisted design of complex organic syntheses
references.venue: Science, 166(3902):178–192,
references.authors: E.J. Corey Richard D. Cramer W. Jeffrey Howe
references.title: Computer-assisted synthetic analysis for complex molecules. methods and procedures for machine generation of synthetic intermediates
references.venue: Journal of the American Chemical Society,
references.authors: E.J. Corey X.M Cheng
references.title: The Logic of Chemical Synthesis
references.authors: Hanjun Dai Chengtao Li Connor Coley Bo Dai Le Song
references.title: Retrosynthesis prediction with conditional graph logic network
references.venue: Advances in Neural Information Processing Systems,
references.authors: Zihang Dai Zhilin Yang Yiming Yang Jaime Carbonell Quoc Le Ruslan Salakhutdinov
references.title: Transformer-XL: Attentive language models beyond a fixed-length context
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.authors: Kien Do Truyen Tran Svetha Venkatesh
references.title: Graph transformation policy network for chemical reaction prediction
references.venue: In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
references.authors: Hongliang Duan Ling Wang Chengyun Zhang Lin Guo Jianjun Li
references.title: Retrosynthesis with attention-based NMT model and chemical analysis of “wrong
references.venue: predictions. RSC Adv.,
references.authors: Meng-Hao Guo Junxiong Cai Zheng-Ning Liu Tai-Jiang Mu Ralph R. Martin Shi-Min Hu
references.title: PCT: point cloud transformer
references.venue: CoRR, 2020
references.authors: Dan Hendrycks Kevin Gimpel
references.title: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, 2016
references.venue: URL http://arxiv.org/abs/1606.08415
references.authors: Weihua Hu Bowen Liu Joseph Gomes Marinka Zitnik Percy Liang Vijay Pande Jure Leskovec
references.title: Strategies for pre-training graph neural networks
references.venue: In International Conference on Learning Representations,
references.authors: R. Irwin S. Dimitriadis J. He E. Bjerrum
references.title: Chemformer: A pre-trained transformer for computational chemistry, 2021
references.authors: Wengong Jin Connor Coley Regina Barzilay Tommi Jaakkola
references.title: Predicting organic reaction outcomes with Weisfeiler-Lehman network
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Wengong Jin Regina Barzilay Tommi Jaakkola
references.title: Junction tree variational autoencoder for molecular graph generation
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Eunji Kim Dongseon Lee Youngchun Kwon Min Sik Park Youn-Suk Choi
references.title: Valid, plausible, and diverse retrosynthesis using tied two-way transformers with latent variables
references.venue: Journal of Chemical Information and Modeling,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In 3rd International Conference on Learning Representations,
references.authors: Guillaume Klein Yoon Kim Yuntian Deng Jean Senellart Alexander Rush
references.title: OpenNMT: Opensource toolkit for neural machine translation
references.venue: In Proceedings of ACL 2017, System Demonstrations,
references.authors: Bumju Kwak Jeonghee Jo Byunghan Lee Sungroh Yoon
references.title: Geometry-aware transformer for molecular property prediction. CoRR, 2021
references.venue: URL https://arxiv.org/abs/2106
references.authors: Greg Landrum
references.title: Rdkit: Open-source cheminformatics software, 2016
references.venue: URL https://github
references.authors: Alpha A. Lee Qingyi Yang Vishnu Sresht Peter Bolgar Xinjun Hou Jacquelyn L. KlugMcLeod Christopher R. Butler
references.title: Molecular transformer unifies reaction prediction and retrosynthesis across pharma
references.authors: Kangjie Lin Youjun Xu Jianfeng Pei Luhua Lai
references.title: Automatic retrosynthetic route planning using template-free models
references.venue: Chem. Sci.,
references.authors: Bowen Liu Bharath Ramsundar Prasad Kawthekar Jade Shi Joseph Gomes Quang Luu Nguyen Stephen Ho Jack Sloane Paul Wender Vijay Pande
references.title: Retrosynthetic reaction prediction using neural sequence-to-sequence models
references.venue: ACS Central Science,
references.authors: D.M. Lowe
references.title: Extraction of chemical structures and reactions from the literature
references.venue: (doctoral thesis),
references.authors: Pierre Mahé Nobuhisa Ueda Tatsuya Akutsu Jean-Luc Perret Jean-Philippe Vert
references.title: Extensions of marginalized graph kernels
references.venue: In Proceedings of the Twenty-First International Conference on Machine Learning,
references.authors: Kelong Mao Xi Xiao Tingyang Xu Yu Rong Junzhou Huang Peilin Zhao
references.title: Molecular graph enhanced transformer for retrosynthesis
references.venue: prediction. Neurocomputing,
references.authors: Juno Nam Jurae Kim
references.title: Linking the neural machine translation and the prediction of organic chemistry reactions
references.authors: Greeshma Neglur Robert L. Grossman Bing Liu
references.title: Assigning unique keys to chemical compounds for data integration: Some interesting counter examples
references.venue: In Data Integration in the Life Sciences,
references.authors: Wesley Wei Qian Nathan T. Russell Claire L.W. Simons Yunan Luo Martin D. Burke Jian Peng
references.title: Integrating deep neural networks and symbolic inference for organic reactivity prediction
references.venue: ChemRxiv,
references.authors: Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu
references.title: Exploring the limits of transfer learning with a unified text-totext transformer
references.venue: Journal of Machine Learning Research,
references.authors: David Rogers Mathew Hahn
references.title: Extended-connectivity fingerprints
references.venue: Journal of Chemical Information and Modeling,
references.authors: Yu Rong Yatao Bian Tingyang Xu Weiyang Xie Ying WEI Wenbing Huang Junzhou Huang
references.title: Self-supervised graph transformer on large-scale molecular data
references.venue: Advances in Neural Information Processing Systems,
references.authors: Mikołaj Sacha Mikołaj Błaż Piotr Byrski Paweł Dąbrowski-Tumański Mikołaj Chromiński Rafał Loska Paweł Włodarczyk-Pruszyński Stanisław Jastrzębski
references.title: Molecule edit graph attention network: Modeling chemical reactions as sequences of graph edits
references.venue: Journal of Chemical Information and Modeling,
references.authors: Philippe Schwaller Théophile Gaudin Dávid Lányi Costas Bekas Teodoro Laino
references.title: Found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequenceto-sequence models
references.venue: Chem. Sci.,
references.authors: Philippe Schwaller Teodoro Laino Théophile Gaudin Peter Bolgar Christopher A. Hunter Costas Bekas Alpha A. Lee
references.title: Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction
references.venue: ACS Central Science,
references.authors: Marwin H.S. Segler Mark P. Waller
references.title: Neural-symbolic machine learning for retrosynthesis and reaction prediction
references.venue: Chemistry – A European Journal,
references.authors: Seung-Woo Seo You Young Song June Yong Yang Seohui Bae Hankook Lee Jinwoo Shin Sung Ju Hwang Eunho Yang
references.title: Gta: Graph truncated attention for retrosynthesis
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Peter Shaw Jakob Uszkoreit Ashish Vaswani
references.title: Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
references.authors: Chence Shi Minkai Xu Hongyu Guo Ming Zhang Jian Tang
references.title: A graph to graphs framework for retrosynthesis prediction
references.venue: Proceedings of the 37th International Conference on Machine Learning,
references.authors: Vignesh Ram Somnath Charlotte Bunne Connor W. Coley Andreas Krause Regina Barzilay
references.title: Learning graph models for template-free retrosynthesis
references.venue: International Conference on Machine Learning (ICML) Workshop on Graph Representation Learning and Beyond (GRL+),
references.authors: Ruoxi Sun Hanjun Dai Li Li Steven Kearnes Bo Dai
references.title: Energy-based view of retrosynthesis
references.authors: Sara Szymkuć Ewa P. Gajewska Tomasz Klucznik Karol Molga Piotr Dittwald Michał Startek Michał Bajczyk Bartosz A. Grzybowski
references.title: Computer-assisted synthetic planning: The end of the beginning
references.venue: Angewandte Chemie International Edition,
references.authors: I.V. Tetko P. Karpov R. Van Deursen
references.title: State-of-the-art augmented NLP transformer models for direct and single-step retro synthesis
references.venue: Nature Communications,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Ł ukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: Advances in Neural Information Processing Systems,
references.authors: Petar Veličković Guillem Cucurull Arantxa Casanova Adriana Romero Pietro Liò Yoshua Bengio
references.title: Graph attention networks
references.venue: In International Conference on Learning Representations,
references.authors: Jiahao Wang Shuangjia Zheng Jianwen Chen Yuedong Yang
references.title: Meta learning for low-resource molecular optimization
references.venue: Journal of Chemical Information and Modeling,
references.authors: Xiaorui Wang Yuquan Li Jiezhong Qiu Guangyong Chen Huanxiang Liu Benben Liao ChangYu Hsieh Xiaojun Yao
references.title: Retroprime: A diverse, plausible and transformer-based method for single-step retrosynthesis predictions
references.venue: Chemical Engineering Journal,
references.authors: Xing Wang Zhaopeng Tu Longyue Wang Shuming Shi
references.title: Self-attention with structural position representations
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: David Weininger
references.title: SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules
references.venue: Journal of Chemical Information and Computer Sciences,
references.authors: Kun Xu Lingfei Wu Zhiguo Wang Yansong Feng Vadim Sheinin
references.title: SQL-to-text generation with graph-to-sequence model
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Kun Xu Lingfei Wu Zhiguo Wang Yansong Feng Michael Witbrock Vadim Sheinin
references.title: Graph2seq: Graph to sequence learning with attention-based neural networks
references.venue: CoRR, abs/1804.00823,
references.authors: Chaochao Yan Qianggang Ding Peilin Zhao Shuangjia Zheng Jinyu Yang Yang Yu Junzhou Huang
references.title: Retroxpert: Decompose retrosynthesis prediction like a chemist
references.venue: Advances in Neural Information Processing Systems,
references.authors: Kevin Yang Kyle Swanson Wengong Jin Connor Coley Philipp Eiden Hua Gao Angel GuzmanPerez Timothy Hopper Brian Kelley Miriam Mathea Andrew Palmer Volker Settels Tommi Jaakkola Klavs Jensen Regina Barzilay
references.title: Analyzing learned molecular representations for property prediction
references.venue: Journal of Chemical Information and Modeling,
references.authors: Chengxuan Ying Tianle Cai Shengjie Luo Shuxin Zheng Guolin Ke Di He Yanming Shen Tie-Yan Liu
references.title: Do transformers really perform bad for graph representation
references.authors: Sanghyun Yoo Young-Seok Kim Kang Hyun Lee Kuhwan Jeong Junhwi Choi Hoshik Lee Young Sang Choi
references.title: Graph-aware transformer: Is attention all graphs need
references.authors: Shuangjia Zheng Jiahua Rao Zhongyue Zhang Jun Xu Yuedong Yang
references.title: Predicting retrosynthetic reactions using self-corrected transformer neural networks
references.venue: Journal of Chemical Information and Modeling,
references.authors: Jinhua Zhu Yingce Xia Tao Qin Wengang Zhou Houqiang Li Tie-Yan Liu
references.title: Dual-view molecule pre-training
references.authors: Kim
references.title: As an illustration of how Graph2SMILES can benefit from additional features or techniques, we briefly experiment with one of them, latent variable modeling similar to Chen et al
sections.heading: 1 INTRODUCTION
sections.text: Retrosynthetic analysis (Corey, 1988; Corey & Cheng, 1989) and its reverse problem, reaction outcome prediction (Corey & Wipke, 1969), are two fundamental problems in computer-aided organic synthesis. The former tries to propose possible reaction precursors given the desirable product, whereas the latter aims to predict the major products given reactants. Historically, they were tackled using rule-based expert systems such as LHASA (Corey et al., 1972). Recent developments in machine learning have led to a number of new template-based, graph edit-based, and translation-based methods, for which we give a detailed review in Section 3.1. For both tasks, translation-based approaches have grown popular, possibly because the end-to-end formulation is procedurally simple. Since most organic molecules can be represented as SMILES strings (Neglur et al., 2005), retrosynthesis can be cast as a translation from product SMILES to reactant SMILES (Liu et al., 2017), and so can reaction outcome prediction (Nam & Kim, 2016). Modeling these tasks as machine translation problems enables the use of neural architectures that are well-studied and well-optimized in the field of Natural Language Processing (NLP). Several of the best performing models across multiple benchmark datasets (Tetko et al., 2020; Irwin et al., 2021; Sun et al., 2020; Seo et al., 2021; Wang et al., 2021b) have used the Transformer architecture (Vaswani et al., 2017) as the backbone on SMILES representations, showing the effectiveness of translation-based formulation.
  However, SMILES representations do not provide bijective mappings to molecular structure. As a result, data augmentation with chemically-equivalent SMILES (Bjerrum, 2017) has become a common practice to improve empirical performance. Augmenting the training data with just 1 equivalent reaction SMILES by permuting both the inputs and the outputs can already produce noticeable improvements of 0.8% to 4.3% (Schwaller et al., 2019; Seo et al., 2021). Incorporating 9 (Wang et al.,
  2021b), or up to 100 (Tetko et al., 2020) equivalent SMILES can provide additional gains. While these efforts demonstrate the effectiveness of SMILES augmentation, this introduces a non-trivial choice as to how much augmentation should be done. The performance has not saturated even with 100 augmented SMILES (Tetko et al., 2020), but this may significantly complicate the model pipeline, especially during test time which typically necessitates de-duplication, ensembling, and heuristic-based scoring (Wang et al., 2021b; Tetko et al., 2020).
  In this paper, we propose a novel graph-to-sequence architecture called Graph2SMILES to solve the tasks of retrosynthesis and reaction prediction. We first design a sequential graph encoder with an attention-augmented directed message passing neural network (D-MPNN) based on Yang et al. (2019) and optionally augmented by attention-based message updates, followed by a Transformerbased global attention encoder with graph-aware positional embedding. We then pair the graph encoder with a Transformer decoder to transform molecular graph inputs into SMILES outputs, without using sequence representations of input SMILES at all. As such, we guarantee the permutation invariance of Graph2SMILES to the input, eliminating the need for input-side augmentation altogether. Our main contributions can be summarized as follows:
  1. We propose a Graph2SMILES architecture with two encoder components modeling local and global atomic interactions respectively, to address forward prediction and one-step retrosynthesis as graph-to-sequence tasks.
  2. We design a graph-aware positional embedding to further enhance performance. It is easily generalizable to graphs containing two or more molecules, while not requiring any pretraining or joint training with auxiliary tasks.
  3. We demonstrate the adequacy of graph representations alone by showing that Graph2SMILES outperforms Transformer baselines on predictive chemistry tasks without needing any input-side SMILES augmentation.
  Our Graph2SMILES architecture achieves state-of-the-art top-1 accuracy on common benchmarks among methods that do not make use of reaction templates, atom mapping, pretraining, or data augmentation strategies. We emphasize that Graph2SMILES is a backbone architecture, and hence a drop-in replacement for the Transformer model. As such, Graph2SMILES can be plugged into any method for molecular transformation tasks that uses the Transformer model, while retraining the benefits from techniques or features orthogonal to the architecture itself.
sections.heading: 2 METHODS
sections.heading: 2.1 GRAPH AND SEQUENCE REPRESENTATIONS OF MOLECULES
sections.text: There are multiple ways of representing molecular structures, such as by molecular fingerprints (Rogers & Hahn, 2010), by SMILES strings (Weininger, 1988), or as molecular graphs with atoms as nodes and bonds as edges. We represent the input molecule(s) as graphs, and the output molecule(s) as SMILES strings, thus modeling both reaction prediction and retrosynthesis as graph-to-sequence transformations.
  Formally, let Gin denotes the molecular graph input, which can contain multiple subgraphs for different molecules, with a total of N atoms. Following the convention in Somnath et al. (2020), we describe Gin = (V, E) with atoms V and bonds E . Each atom u ∈ V has a feature vector xu ∈ Ra, and each directed bond (u, v) ∈ E from atom u to v has its feature vector as xuv ∈ Rb. The details of the atom and bond features used can be found in Appendix A. We build the input molecular graphs from their SMILES strings with RDKit (Landrum, 2016). Note that all feature vectors are invariant to the order of atoms and bonds, as well as to how the original SMILES strings are written. We represent the output as a sequence of SMILES tokens Sout = {s1, s2, . . . , sn}, where the tokens {si} are obtained from the canonical SMILES using the regex tokenizer in Schwaller et al. (2019).
sections.heading: 2.2 GRAPH2SMILES
sections.text: The Graph2SMILES model is a variant of the encoder-decoder model (Cho et al., 2014) commonly used for machine translation. Figure 1 displays the architecture of Graph2SMILES with the permutation invariant graph encoding process shown within the blue dashed box. We replace the
  encoder part of the standard Transformer model (Vaswani et al., 2017) used in Molecular Transformer (Schwaller et al., 2019) with a novel attention-augmented directed message passing encoder, followed by a global attention encoder with carefully designed graph-aware positional embedding. Each module has its intuitive function: the D-MPNN captures the local chemical context, the global attention encoder allows for global-level information exchange, and the graph-aware positional embedding enables the attention encoder to make use of topological information more explicitly. The permutation invariant encoding process eliminates the need for SMILES augmentation for the input side altogether, simplifying data preprocessing and potentially saving training time.
sections.heading: 2.2.1 ATTENTION AUGMENTED DIRECTED MESSAGE PASSING ENCODER
sections.text: The first module of the graph encoder is a D-MPNN (Yang et al., 2019) with Gated Recurrent Units (GRUs) (Cho et al., 2014) used for message updates (Jin et al., 2018; Somnath et al., 2020). Unlike atom-oriented message updates in edge-aware MPNNs (Hu et al., 2020; Yan et al., 2020; Mao et al., 2021; Wang et al., 2021a), updates in D-MPNN are oriented towards directed bonds to prevent totters, or messages being passed back-and-forth between neighbors (Mahé et al., 2004; Yang et al., 2019). We optionally augment the D-MPNN with attention-based message updates inspired from Graph Attention Network (Veličković et al., 2018; Brody et al., 2021). We term this variant as Directed Graph Attention Network (D-GAT) and refer to the original D-MPNN variant used in Somnath et al. (2020) as Directed Graph Convolutional Network (D-GCN), while keeping both D-GAT and D-GCN as possible design choices for the D-MPNN.
  For D-GAT, at each message passing step t, the message mt+1uv associated with each directed bond (u, v) ∈ E is updated using
  suv = AttnSum ( xu,xuv, { mtwu } w∈N(u)\v ) (1)
  zuv = σ (Wz [xu;xuv; suv] + bz) (2) ruv = σ (Wr [xu;xuv; suv] + br) (3)
  m̃uv = tanh (W [xu;xuv] +Uruv + b) (4)
  mt+1uv = (1− zuv) suv + zuv m̃uv (5)
  where Wz , Wr, W , U and bz , br, b are the learnable weights and biases respectively. σ is the sigmoid function, ";" indicates concatenation, and is the element-wise product. AttnSum is the attention-based message aggregation defined as
  ewu = a TLeakyReLU ( Wqk [ xu;xuv;m t wu ] + bqk ) (6) awu = exp (ewu)∑
  w∈N(u)\v exp (ewu) (7)
  AttnSum ( xu,xuv, { mtwu } w∈N(u)\v ) = ∑ w∈N(u)\v awu ( Wvm t wu + bv ) (8)
  where Wqk, bqk, a are the learnable parameters for the attention scores, and Wv , bv are the parameters for the value vectors. We have omitted a self-loop in message aggregation, as it did not have any noticeable effect on performance from our experiments. In Eqn (3) we simplify the reset gate to be shared for all incoming edges, in contrast to Somnath et al. (2020) where a separate reset gate is defined for each edge.
  Lastly, after T iterations, we obtain the atom representations hu with similar attention-based aggregation (with different parameters) over the bond messages coming into each atom u, followed by a single output layer with weight Wo and GELU activation (Hendrycks & Gimpel, 2016).
  mu = AttnSum ′ ( xu, { m(T )wu } w∈N(u) ) (9)
  hu = GELU (Wo [xu;mu]) (10) We use multi-headed attention in our formulation similar to Brody et al. (2021).
sections.heading: 2.2.2 GLOBAL ATTENTION ENCODER WITH GRAPH-AWARE POSITIONAL EMBEDDING
sections.text: To capture global interactions, the atom representations coming out of the D-MPNN are fed into a global attention encoder, which is a variant of the Transformer encoder. We incorporate graphaware positional embedding, adapted from the relative positional embedding used in TransformerXL (Dai et al., 2019b) as follows. Firstly, in the standard Transformer either with sinusoidal encoding (Vaswani et al., 2017; Schwaller et al., 2019) or learnable (Devlin et al., 2019) absolute positional embedding, the attention score between atoms u and v can be decomposed as
  eabsu,v = h T uW̃ T q W̃khv + h T uW̃ T q W̃kpv + p T uW̃ T q W̃khv + p T uW̃ T q W̃kpv (11)
  where W̃q ,W̃k are weights for the keys and queries, and pu, pv are the absolute positional encoding or embedding corresponding to atoms u and v. Similar to Transformer-XL, we reparameterize the four terms. Instead of using sequence-based relative positional embedding ru−v , we use a learnable embedding term ru,v that is dependent on the shortest path length between u and v
  erelu,v = ( hTuW̃ T q + c T ) W̃khv + ( hTuW̃ T q + d T ) W̃k,Rru,v
  = ( hTuW̃ T q + c T ) W̃khv + ( hTuW̃ T q + d T ) r̃u,v (12)
  The trainable biases are renamed as c and d to avoid confusion and shared across all layers. Intuitively, the two terms in Eqn (12) model the interactions between inputs (hu and hv), and between input and relative graph position (hu and r̃u,v) respectively. Unlike Transformer-XL, we forgo the inductive bias built into the sinusoidal encoding, merging W̃k,Rru,v into a single learnable r̃u,v . This makes the relative positional embedding easily generalizable to atoms not within the same molecule, which is particular useful for reaction outcome prediction and, more generally, tasks with more than two input molecules. We also bucket the distances similar to Raffel et al. (2020) such that
  Bu,v =  distance(u, v), if distance(u, v) < 8
  8, if 8 ≤ distance(u, v) < 15 9, if 15 ≤ distance(u, v) and u, v are in the samemolecule 10, if u, v are not in the samemolecule
  Bu,v is then used to look up r̃u,v in the trainable positional embedding matrix. The rest of the global attention encoder mostly follows a standard Transformer with multi-headed self-attention (Vaswani et al., 2017), layer normalization (Ba et al., 2016), and position-wise feed forward layers, except no positional information is added to the value vectors within each Transformer layer.
sections.heading: 2.2.3 SEQUENCE DECODER
sections.text: We use a Transformer-based autoregressive decoder to decode from the atom representations after the global attention encoder. Each output token is generated by attending to all atoms with encoderdecoder attention (Bahdanau et al., 2015; Vaswani et al., 2017), while also attending to all tokens that have already been generated. Following Seo et al. (2021), we set max relative positions to 4 for the decoder, thereby enabling the usage of sequence-based relative positional embedding used in Shaw et al. (2018) and implemented by OpenNMT (Klein et al., 2017).
sections.heading: 2.3 MODEL TRAINING
sections.text: The Graph2SMILES model is then trained to maximize the conditional likelihood p (Sout|Gin) = p (s1, s2, . . . , sn|Gin) = n∏ i=1 pθ (si|s1:i−1,Gin) (13)
sections.heading: 3 RELATED WORK
sections.heading: 3.1 REACTION OUTCOME PREDICTION AND ONE-STEP RETROSYNTHESIS
sections.text: One approach to computer-aided reaction prediction and retrosynthesis is to make use of chemical reaction rules based on subgraph pattern matching that are formalized as reaction templates, as in expert systems such as LHASA (Corey et al., 1972) and SYNTHIA (Szymkuć et al., 2016). More recent efforts have used neural networks to model the two tasks as template classification (Segler & Waller, 2017; Baylon et al., 2019; Dai et al., 2019a; Chen & Jung, 2021), or template ranking based on molecular similarity (Coley et al., 2017). These template-based approaches select the top ranked templates, which can then be applied to transform the input molecules into the outputs.
  For template-based methods, there is an inevitable tradeoff between template generality and specificity. Further, these methods cannot generalize to unseen templates. As a remediation for such intrinsic limitations, a number of template-free approaches have emerged over the recent years, which can be categorized into graph edit-based and translation-based. The first category models reaction prediction or retrosynthesis as graph transformations (Jin et al., 2017; Coley et al., 2019; Do et al., 2019; Bradshaw et al., 2019; Sacha et al., 2021; Qian et al., 2020). Variants of graph edit methods include electron flow prediction (Bi et al., 2021) and semi template-based methods where reaction centers are first identified, followed by a graph or sequence recovery stage (Shi et al., 2020; Yan et al., 2020; Somnath et al., 2020; Wang et al., 2021b). Translation-based formulations, on the other hand, approach the problems as SMILES-to-SMILES translation, typically with sequence models such as Recurrent Neural Networks (Nam & Kim, 2016; Schwaller et al., 2018; Liu et al., 2017) or the Transformer (Schwaller et al., 2019; Lin et al., 2020; Lee et al., 2019; Duan et al., 2020; Tetko et al., 2020). Variants of these approaches design additional stages such as pretraining and reranking (Irwin et al., 2021; Zhu et al., 2021; Zheng et al., 2020; Sun et al., 2020), or use information about graph topology to enhance performance (Yoo et al., 2020; Seo et al., 2021; Mao et al., 2021).
  Our approach is similar to GET (Mao et al., 2021) which also solves one-step retrosynthesis with graph-enhanced encoders and sequence decoders. Unlike GET which concatenates the SMILES sequence embeddings and learned atom representations, thereby not guaranteeing permutation invariance, we do not use the sequence representation at all in our encoder. Yet Graph2SMILES yields significant improvement over GET on USPTO_50k, demonstrating the power of our graph encoder and the adequacy of graph representations alone.
sections.heading: 3.2 ADAPTING THE TRANSFORMER ENCODER FOR MOLECULAR REPRESENTATION
sections.text: The idea of modeling graphs using the Transformer architecture is not new. In the domain of molecular representation, the Molecular Attention Transformer (Maziarka et al., 2020) and GeoT (Kwak et al., 2021) inject atomic distance information into when computing attention scores. However, the computation of such distance information itself requires sampling of 3D conformers, thereby introducing an additional source of variations and breaking order invariance. An alternative is using the lengths of shortest paths between atoms. GRAT (Yoo et al., 2020) uses these lengths to parameterize the optional scale and bias for the attention score, whereas PAGTN (Chen et al., 2019)
  treats them as path features in its additive attention. Our use of pairwise shortest path lengths between atoms for our Transformer-based global attention encoder are inspired by GRAT and PAGTN. Contrary to their usage of these lengths as additional features, we explicitly separate the effect of graph topology using graph-aware relative positional embedding, considering the success of such specially designed embedding in graph representation (Ying et al., 2021) and other domains (Shaw et al., 2018; Dai et al., 2019b; Wang et al., 2019; Guo et al., 2020). Our formulation of relative positional embedding builds on top of its counterpart in Transformer-XL (Dai et al., 2019b), which we found to be empirically superior than using single learnable bias as in T5 (Raffel et al., 2020) and Graphormer (Ying et al., 2021).
sections.heading: 3.3 COMBINATION OF GRAPH NEURAL NETWORKS AND TRANSFORMER
sections.text: Combining graph encoder and Transformer encoder in a sequential manner has been explored in GET (Mao et al., 2021) and NERF (Bi et al., 2021), as well as in Wang et al. (2021a) and GROVER (Rong et al., 2020) albeit for different molecular learning tasks. Graph2SMILES does not use the sequence representation as an input as in GET, or require the output molecular graph in the encoder as in NERF. Also, none of these related works retain the explicit information about graph topology before passing the atom representations into the attention encoder like we do, which we show to be important in the ablation study in Section 4.5. Similarly, the graph-to-sequence formulation itself has been used in NLP for conditional text generation tasks such as SQL-to-text (Xu et al., 2018a;b) and AMR-to-text (Cai & Lam, 2020). Our encoder is different from these prior studies; most notably, the graph-aware positional embedding is designed to easily generalize to more than two disconnected graphs, which is typical for reaction outcome prediction.
sections.heading: 4 EXPERIMENTS
sections.heading: 4.1 DATASETS
sections.text: We evaluate model performance by top-n test accuracies on four USPTO datasets derived from reaction data originally curated by Lowe (2012). The details of these datasets are summarized in Appendix C. For reaction outcome prediction, we evaluate on the USPTO_480k_mixed and USPTO_STEREO_mixed datasets following Schwaller et al. (2019). The suffix _mixed indicates that the reactants and reagents have not been separated based on which species contribute heavy atoms to the product. While USPTO_480k has been preprocessed by Jin et al. (2017), USPTO_STEREO was filtered to a lesser extent, retaining stereochemical information and reactions forming or breaking aromatic bonds. For one-step retrosynthesis, we evaluate on the USPTO_full and USPTO_50k datasets without reaction type, both of which have been used as benchmarks for retrosynthesis. We count a prediction as correct only if it matches the ground truth output SMILES exactly, including all stereochemistry but excluding atom mapping, after canonicalization by RDKit.
sections.heading: 4.2 IMPLEMENTATION DETAILS
sections.text: For D-GAT and D-GCN, we change the hidden size to 256 from 300 used in GraphRetro (Somnath et al., 2020) to be more consistent with the global attention encoder. The number of message updating steps is set to 4. For the global attention encoder, following Molecular Transformer and GTA, we fix the embedding and hidden sizes dmodel to 256, the filter size for Transformer to 2048, the number of attention heads to 8, and the numbers of layers for both the attention encoder and the Transformer decoder to 6. We train our model using Adam optimizer (Kingma & Ba, 2015) with Noam learning rate scheduler (Vaswani et al., 2017). Similar to Schwaller et al. (2019), we group reactions with similar number of SMILES tokens together, batch by the maximal number of token count, and scale the D-MPNN outputs by √ dmodel before feeding into the attention encoder. The details of the hyperparameters used for different datasets are summarized in Appendix D. We save the model checkpoints every 5000 steps, select the best checkpoints based on the top-1 accuracy on the validation sets, and report the performance on the held-out test sets. Beam search is used to generate the output SMILES during inference with a beam size of 30. We filter out any SMILES that cannot be parsed by RDKit and keep the remaining as our final list of proposed candidates for evaluation.
sections.heading: 4.3 RESULTS ON REACTION OUTCOME PREDICTION
sections.text: Table 1 summarizes the results of Graph2SMILES and other existing works on reaction outcome prediction. We only include methods that perform evaluations on the USPTO_480k_mixed dataset in the table, and provide a brief comparison of other methods that only evaluate on the less challenging USPTO_480k_separated data in Appendix B. All excluded methods show inferior performance to the Molecular Transformer (MT), with the exception of NERF (Bi et al., 2021) which has a 0.3 point improvement in top-1 accuracy.
  For the USPTO_480k_mixed dataset, Graph2SMILES improves upon the MT baseline with 1.7, 0.5 and 0.4 point increases in top-1, 3 and 10 accuracies respectively. Similarly, for the USPTO_STEREO_mixed dataset, Graph2SMILES improves the top-1 accuracy of the MT baseline by 1.9 points, with minor improvement for top-3 accuracy. While there is still a gap between Graph2SMILES and Augmented Transformer or Chemformer, our approach does not use test-time data augmentation and ensembling as in Augmented Transformer (Tetko et al., 2020), nor have we performed pretraining as in Chemformer (Irwin et al., 2021) whose models have up to 10 times as many parameters as Graph2SMILES. Ensembling and pretraining are potential directions for improving Graph2SMILES as they are still compatible with our backbone replacement for the Transformer. For reaction outcome prediction, there is a small advantage of using D-GAT over D-GCN, with improvements of up to 0.2 points on top-n accuracy.
sections.heading: 4.4 RESULTS ON ONE-STEP RETROSYNTHESIS
sections.text: We compare the results of one-step retrosynthesis on USPTO_full of Graph2SMILES with all existing methods that report results on this dataset, to the best of our knowledge. As can be seen from Table 2, Graph2SMILES achieves higher top-1 accuracy than all methods except GTA (Seo et al., 2021), while not using any templates, atom mapping, or output-side data augmentation. These additional features or techniques, which have been demonstrated to improve the performance of Transformer variants (Appendix E), are orthogonal to the graph-to-sequence architecture itself, and can therefore potentially improve Graph2SMILES as well. For example, atom mapping can be used to enhance Graph2SMILES with graph-truncated cross-attention as in GTA.
  The much smaller USPTO_50k dataset has been benchmarked more extensively. We compare the results in Table 3, marking only the usage of the same set of features and techniques used as in Table 2 for succinctness1. The first group of rows shows that across methods that do not use reaction
  1Graph2SMILES can also benefit from other techniques such as latent variable modeling (Chen et al., 2020), which we demonstrate in Appendix E. From Table 10, using latent classes N = 2 can already boost the top-10 accuracies of the D-GCN variant from 72.9 to 79.5, and of D-GAT from 73.9 to 77.7.
  templates, atom mapping, or output SMILES augmentation, Graph2SMILES achieves the best top1 accuracy, improving the Transformer baseline (Lin et al., 2020) by 9.8 points from 43.1 to 52.9. From our experiments, we observe that it is possible to boost top-n accuracies for n > 1 at the expense of sacrificing the top-1 accuracy (even slightly), creating some room for tradeoff. To avoid over-tuning and giving overly optimistic results, however, we only report the test results for models with the highest top-1 accuracy during validation. Unlike in reaction outcome prediction, using DGAT does not have a clear advantage over D-GCN for retrosynthesis, yielding only improvement on top-5 and 10 accuracies for USPTO_50k as in Table 3.
  The second group of rows in Table 3 includes methods that use additional features or techniques. Graph2SMILES beats a number of methods in this group with a top-1 accuracy of 52.9 without using any of templates, atom mapping or output SMILES augmentation. EBM (Dual-TB) (Sun
  et al., 2020) achieves the SOTA for top-1 accuracy, and LocalRetro (Chen & Jung, 2021) for top-3, 5 and 10 accuracies respectively. Both make use of templates and atom mapping, which seem to be empirically helpful for this small dataset. Similar to how Graph2SMILES can benefit from using atom mapping as discussed earlier, we can make use of templates for potential gain, e.g. by retaining only candidates with reaction templates that have been seen in the training set.
sections.heading: 4.5 ABLATION STUDY
sections.text: We perform an ablation study to explore the effects of various components of Graph2SMILES by removing the D-MPNN, positional embedding, or the global attention encoder. We summarize the results on USPTO_50k, for which the quantitative effect is most conspicuous. From the results in Table 4, the removal of D-MPNN decreases the top-1 accuracy to 49.9. Note that this setup is architecturally similar to the Transformer baseline, but uses atom features rather than SMILES sequence as inputs. Getting rid of the graph-aware positional embedding leads to a drop in top1 accuracy of 2.1 points for D-GCN and 0.6 points for D-GAT. The effect of removing the global attention encoder is more significant, decreasing the top-1 accuracy by up to 8.3 points. We therefore conclude that all three components of our encoder in Graph2SMILES, namely the D-MPNN, the graph-aware positional embedding, and the global attention encoder, are important. They work collectively to capture a meaningful representation of the input molecular graphs.
sections.heading: 5 DISCUSSION
sections.text: Throughout our experiments, to demonstrate the advantage over a vanilla Transformer, we have focused on the baseline Graph2SMILES model, forgoing the benefits of using additional features and techniques for performance engineering. Although we have used the top-1 accuracy as a basis for comparison throughout our discussion, we recognize its limitations especially for retrosynthesis, which can have many equally plausible options. Similarly, the datasets we use for reaction outcome prediction are not perfectly detailed, with legitimate ambiguity in the identity of the major product. While Graph2SMILES shows strong performance for top-1 accuracy and can replace Transformer model with minimal modification to the pipeline, there could be cases when top-n accuracy is more relevant (e.g. in multi-step planning applications). In those cases, the aforementioned performance engineering techniques would be necessary to boost the top-n performance of Graph2SMILES.
sections.heading: 6 CONCLUSION
sections.text: In this paper, we present a novel Graph2SMILES model for template-free reaction outcome prediction and retrosynthesis. The permutation invariance of its D-MPNN and graph-aware positional embedding eliminates the need for any input-side SMILES augmentation, while achieving noticeable improvement over the Transformer baselines, especially for top-1 accuracy. Graph2SMILES is therefore an attractive drop-in replacement for any methods that use the Transformer model for molecular transformation tasks. Further gain may be possible through performance engineering tricks that are orthogonal to the architecture itself, which will be investigated in future work.
  REPRODUCIBILITY STATEMENT
  We hereby declare that all of our reported results are reproducible from our existing code base, subject to minor deviations due to hardware-level numerical uncertainties as we have observed for some GPU models (e.g. V100). We include part of our code to reproduce some specific results, with a self-explanatory README file in the supplementary materials. We will open-source all scripts to reproduce any result in the manuscript along with pretrained Graph2SMILES checkpoints after the review period, regardless of whether the manuscript is accepted.
sections.heading: A APPENDIX: ATOM AND BOND FEATURES USED
sections.text: Table 5 summarizes the atom and bond features used in Graph2SMILES. Most features were adapted from GraphRetro (Somnath et al., 2020), with the addition of chiral features (R/S and E/Z).
sections.heading: B APPENDIX: OTHER METHODS FOR REACTION OUTCOME PREDICTION
sections.text: Table 6 summarizes the results for methods excluded in Section 4.3 for reaction outcome prediction, most of which report the values on the less challenging USPTO_480k_separated dataset, in which the reagents have been heuristically separated from the reactants. Only NERF shows marginal improvement of 0.3 points for top-1 accuracy over Molecular Transformer, whereas all other methods cannot perform as well. We therefore use Molecular Transformer as our baseline in Section 4.3. Note that ELECTRO (Bradshaw et al., 2019) tests on a simpler subset of reactions with linear electron flow (LEF), and we therefore exclude it from the quantitative comparison.
sections.heading: D APPENDIX: HYPERPARAMETER SETTING
sections.heading: C APPENDIX: SUMMARY OF FOUR USPTO DATASETS USED
sections.heading: E APPENDIX: EFFECTIVENESS OF ADDITIONAL FEATURES OR TECHNIQUES
sections.text: For all groups of rows in Table 9, the first row of numbers are for the Transformer variants where features or techniques of interest are not used, and the rest of the rows otherwise. In turn, these results illustrate the effectiveness of using templates (EBM), atom-mapping (GTA), pretraining (DMP), output-side augmentation (Augmented Transformer), syntax-based reranking (SCROP) and variational inference (Latent Transformer). Inclusion of these features or techniques clearly improves the accuracies across the board over respective baselines, as highlighted in bold.
  As an illustration of how Graph2SMILES can benefit from additional features or techniques, we briefly experiment with one of them, latent variable modeling similar to Chen et al. (2020) and Kim et al. (2021), using N = 2 latent classes with a uniform prior. As in Table 10, this noticeably increases the top-n accuracies for both D-GCN and D-GAT, which can be particularly relevant for multi-step planning.
sections.heading: F APPENDIX: VISUALIZATION OF CASES WHERE GRAPH2SMILES OUTPERFORMS THE TRANSFORMER BASELINE
sections.text: We include some real cases for which Graph2SMILES gives the correct top-1 prediction but the Molecular Transformer baseline cannot, for the reaction outcome prediction task on USPTO_480k_mixed. For each row in Figure 2, the reactants are shown on the left, the ground truth product (which is also the correct prediction by Graph2SMILES) is shown in the center, and the erroneous prediction by the Molecular Transformer baseline is shown on the right.
  The first two rows represent two intramolecular ring forming reactions after an N-alkylation or amidation. As shown by the right-most molecules, without explicit knowledge of the reactant graphs, Molecular Transformer instead predicts substitution reactions at the benzylic carbon and at the phenyl carbon respectively. Both predictions seem to be based solely on local chemical environments, with the first being plausible as nucleophilic substitution on benzyl bromide, but the second nonsensical since such electrophilic aromatic substitution would not happen with the nucleophilic phenylamine. Graph2SMILES, on the other hand, correctly predicts the ring formation. We therefore hypothesize that graph representation may be more powerful at encoding beyond local contexts via its global attention mechanism, which is potentially important for such reactions. Further quantitative evaluation is left as future work, since explainability is not a trivial task for template-free models.
  The third row represents the removal of the amine group attached to the benzene ring, for which Graph2SMILES gives the correct prediction, but the Molecular Transformer predicts the formation of an aryl hydrazine, possibly by detecting the extra nitrogen in the input SMILES (from DMF) but failing to realize it as merely a spectator solvent without full recognition of its graph structure. The last row is a case where Graph2SMILES predicts the correct regioselectivity (ortho vs. para to the nitro group) for the aromatic substitution reaction. This may be interpreted as better capability of Graph2SMILES to learn longer range topological information as compared to the Transformer baseline.