summaries: This paper proposes a black-box attack where, by relying on segmentation priors, the perturbation is applied only in the salient region. This allows one to obtain reduce perceptibility with a limited number of queries and a small reduction in success rate. More specifically, once the salient region has been identified, a refining procedure is carried out to find small areas where the perturbation should be added. Experiments are performed on ImageNet and results are compared with those of some SOTA methods and their variants that work on saliency regions.

summaries: There has been a lot of interest in improving the query efficiency of black-box attacks in the recent past. However, these techniques produce examples that a human in the loop can quickly identify. The authors propose using segmentation priors to improve the black-box attacks so that the perturbations are restricted to the salient regions of the image. In addition to this, they also present a technique that improves the imperceptibility without forgoing the query efficiency.

summaries: The paper studies how to reduce the perceptibility of the perturbations to the original images produced by black-box adversarial attacks (for the $\ell_\infty$-threat model). In particular, it proposes to use a prior based on segmentation techniques to localize the changes on the subject of the image and leave the background unaltered. Moreover, the Saliency Attack is introduced to further reduce the fraction of the original image which is modified to induce misclassification.

summaries: This paper proposed a method to generate imperceptible attack in black box attack scenario by generating local perturbation blocks in salient regions. It used salient object segmentation to obtain the salient region, then applied a tree search method to find smallest blocks within the salient region that can cause the maximal change in predicted class logits. Experiments on 1000 Imagenet examples are conducted, compared to several existing baselines, showing that the proposed method can improve achieve more imperceptible attacks (where imperceptibility is measured by metric MAD).

abstractText: Deep neural networks are vulnerable to adversarial examples, even in the blackbox setting where the attacker only has query access to the model output. Recent studies have devised successful black-box attacks with high query efficiency. However, such performance often comes at the cost of the imperceptibility of adversarial attacks, which is essential for attackers. To address this issue, in this paper we propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region. We find that state-of-the-art blackbox attacks equipped with segmentation priors can achieve much better imperceptibility performance with little reduction in query efficiency and success rate. We further propose the Saliency Attack, a new gradient-free black-box attack that can further improve the imperceptibility by refining perturbations in the salient region. Experimental results show that the perturbations generated by our approach are much more imperceptible than the ones generated by other attacks, and are interpretable to some extent. Furthermore, our approach is found to be more robust to detection-based defense, which demonstrates its efficacy as well.

authors:
Moustafa Alzantot
Yash Sharma
Supriyo Chakraborty
Huan Zhang
Cho-Jui Hsieh
Mani B. Srivastava

title: Genattack: practical black-box attacks with gradient-free optimization

venue: Proceedings of the Genetic and Evolutionary Computation Conference,

authors:
Maksym Andriushchenko
Francesco Croce
Nicolas Flammarion
Matthias Hein

title: Square attack: A query-efficient black-box adversarial attack via random search

venue: Computer Vision - ECCV 2020 16th European Conference,

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations,

authors:
Wieland Brendel
Jonas Rauber
Matthias Bethge

title: Decision-based adversarial attacks: Reliable attacks against black-box machine learning models

venue: In 6th International Conference on Learning Representations,

authors:
Nicholas Carlini
David A. Wagner

title: Towards evaluating the robustness of neural networks

venue: IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA,

authors:
Pin-Yu Chen
Huan Zhang
Yash Sharma
Jinfeng Yi
Cho-Jui Hsieh

title: ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models

venue: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition

authors:
Xiaoyi Dong
Jiangfan Han
Dongdong Chen
Jiayang Liu
Huanyu Bian
Zehua Ma
Hongsheng Li
Xiaogang Wang
Weiming Zhang
Nenghai Yu

title: Robust superpixel-guided attentional adversarial attack

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Sid Ahmed Fezza
Yassine Bakhti
Wassim Hamidouche
Olivier Déforges

title: Perceptual evaluation of adversarial attacks for cnn-based image classification

venue: In 11th International Conference on Quality of Multimedia Experience QoMEX 2019,

authors:
Ian J. Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

authors:
Chuan Guo
Jared S. Frank
Kilian Q. Weinberger

title: Low frequency adversarial perturbation

venue: Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence,

title: The microbial genetic algorithm

venue: Advances in Artificial Life. Darwin Meets von Neumann - 10th European Conference,

authors:
Andrew Ilyas
Logan Engstrom
Anish Athalye
Jessy Lin

title: Black-box adversarial attacks with limited queries and information

venue: Proceedings of the 35th International Conference on Machine Learning,

authors:
Andrew Ilyas
Logan Engstrom
Aleksander Madry

title: Prior convictions: Black-box adversarial attacks with bandits and priors

venue: In 7th International Conference on Learning Representations,

authors:
Eric C. Larson
Damon M. Chandler

title: Most apparent distortion: full-reference image quality assessment and the role of strategy

venue: J. Electronic Imaging,

authors:
Nannan Li
Zhenzhong Chen

title: Toward visual distortion in black-box attacks

venue: IEEE Transactions on Image Processing,

authors:
Xurong Li
Shouling Ji
Meng Han
Juntao Ji
Zhenyu Ren
Yushan Liu
Chunming Wu

title: Adversarial examples versus cloud-based detectors: A black-box empirical study

venue: IEEE Trans. Dependable Secur. Comput.,

authors:
Yanpei Liu
Xinyun Chen
Chang Liu
Dawn Song

title: Delving into transferable adversarial examples and black-box attacks

venue: In 5th International Conference on Learning Representations,

authors:
Seungyong Moon
Gaon An
Hyun Oh Song

title: Parsimonious black-box adversarial attacks via efficient combinatorial optimization

venue: In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning,

authors:
Seyed-Mohsen Moosavi-Dezfooli
Alhussein Fawzi
Pascal Frossard

title: Deepfool: A simple and accurate method to fool deep neural networks

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Nina Narodytska
Shiva Prasad Kasiviswanathan

title: Simple black-box adversarial perturbations for deep networks. CoRR, abs/1612.06299, 2016

venue: URL http://arxiv.org/abs/1612

authors:
Chris Olah
Arvind Satyanarayan
Ian Johnson
Shan Carter
Ludwig Schubert
Katherine Ye
Alexander Mordvintsev

title: The building blocks of interpretability. Distill, 2018

authors:
Nicolas Papernot
Patrick D. McDaniel
Somesh Jha
Matt Fredrikson
Z. Berkay Celik
Ananthram Swami

title: The limitations of deep learning in adversarial settings

venue: In IEEE European Symposium on Security and Privacy,

authors:
Nicolas Papernot
Patrick D. McDaniel
Ian J. Goodfellow
Somesh Jha
Z. Berkay Celik
Ananthram Swami

title: Practical black-box attacks against machine learning

venue: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017,

authors:
Omkar M. Parkhi
Andrea Vedaldi
Andrew Zisserman

title: Deep face recognition

venue: Proceedings of the British Machine Vision Conference

authors:
Joseph Redmon
Santosh Kumar Divvala
Ross B. Girshick
Ali Farhadi

title: You only look once: Unified, real-time object detection

title: Theory of linear and integer programming

authors:
Ramprasaath R. Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In IEEE International Conference on Computer Vision, ICCV 2017,

authors:
Mahmood Sharif
Lujo Bauer
Michael K. Reiter

title: On the suitability of lp-norms for creating and preventing adversarial examples

venue: IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018,

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency maps

venue: In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations,

authors:
Jiawei Su
Danilo Vasconcellos Vargas
Kouichi Sakurai

title: One pixel attack for fooling deep neural networks

venue: IEEE Trans. Evol. Comput.,

authors:
Qi Sun
Arjun Ashok Rao
Xufeng Yao
Bei Yu
Shiyan Hu

title: Counteracting adversarial attacks in autonomous driving

venue: In IEEE/ACM International Conference On Computer Aided Design, ICCAD

authors:
Fnu Suya
Jianfeng Chi
David Evans
Yuan Tian

title: Hybrid batch attacks: Finding black-box adversarial examples with limited queries

venue: In Srdjan Capkun and Franziska Roesner (eds.), 29th USENIX Security Symposium, USENIX Security

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian J. Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jonathon Shlens
Zbigniew Wojna

title: Rethinking the inception architecture for computer vision

authors:
Chun-Chen Tu
Pai-Shun Ting
Pin-Yu Chen
Sijia Liu
Huan Zhang
Jinfeng Yi
Cho-Jui Hsieh
Shin-Ming Cheng

title: Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks

venue: In The Thirty-Third AAAI Conference on Artificial Intelligence,

authors:
Lina Wang
Kang Yang
Wenqi Wang
Run Wang
Aoshuang Ye

title: Mgaattack: Toward more query-efficient black-box attack by microbial genetic algorithm

venue: MM ’20: The 28th ACM International Conference on Multimedia,

authors:
Zhou Wang
Alan C. Bovik
Hamid R. Sheikh
Eero P. Simoncelli

title: Image quality assessment: from error visibility to structural similarity

venue: IEEE Trans. Image Process.,

authors:
Tao Xiang
Hangcheng Liu
Shangwei Guo
Tianwei Zhang
Xiaofeng Liao

title: Local black-box adversarial attacks: A query efficient approach

authors:
Weilin Xu
David Evans
Yanjun Qi

title: Under review as a conference paper at ICLR

title: Visualizing and understanding convolutional networks

authors:
Aditya Khosla
Àgata Lapedriza
Aude Oliva
Antonio Torralba

title: Learning deep

heading: 1 INTRODUCTION

text:
Deep neural networks (DNNs) have achieved significant progress in wide applications, such as image classification (Deng et al., 2009), face recognition (Parkhi et al., 2015), object detection (Redmon et al., 2016), speech recognition (4, 2012) and machine translation (Bahdanau et al., 2015). Despite their success, deep learning models have revealed vulnerability to adversarial attacks (Szegedy et al., 2014). Crafted by adding some small perturbations to benign inputs, adversarial examples (AEs) can fool DNNs into making wrong predictions, which is a critical threat especially for some security-sensitive scenarios such as autonomous driving (Sun et al., 2020).
Based on the knowledge of target model, the adversarial attacks could be divided into white-box attack and black-box attack. White-box attacks (Szegedy et al., 2014; Goodfellow et al., 2015; Papernot et al., 2016; Carlini & Wagner, 2017; Moosavi-Dezfooli et al., 2016) have full access to the architecture and parameters of the target model, and can easily generate a successful adversarial example via back propagation. By contrast, black-box attacks (Narodytska & Kasiviswanathan, 2016; Ilyas et al., 2018; 2019; Moon et al., 2019; Andriushchenko et al., 2020; Li et al., 2021; Papernot et al., 2017; Liu et al., 2017) can only query the target model to obtain the output prediction, which is a more realistic and challenging setting. Since many real-world online application programming interfaces (APIs) have time or monetary limit for user query (Ilyas et al., 2018), most current research efforts focus on how to design query-efficient attacks (Ilyas et al., 2019; Moon et al., 2019; Andriushchenko et al., 2020), which indeed achieve a huge improvement. For example, so far the state-of-the-art black-box attack (Andriushchenko et al., 2020) could succeed in untargeted attack on ImageNet dataset (Deng et al., 2009) with only tens of queries on average. However, this success has sacrificed the imperceptibility of AEs, whose global perturbations are generated with various tricks like random vertical stripes (Andriushchenko et al., 2020), and thus very obvious and perceptible to human eyes (see Figure 1 (e) and (f) for examples). In effect, imperceptibility is essential for attackers. Currently online APIs usually integrate detectors into their services to detect anomalous inputs (Li et al., 2021). The images with too visible perturbations are difficult to pass these detectors, not to mention human judgement.
To alleviate or even address this problem, we could utilize some priors to lead black-box attacks to search in small but efficient space, instead of global perturbations. Previous DNNs visualization work (Simonyan et al., 2014) proposed a type of saliency map to indicate which features should influence the output significantly. This BP saliency map 1 is generated by calculating the derivatives of the model output with respect to input. Just as show in Figure 1.(a), the brighter pixels in BP saliency maps denote they have a greater impact on the model output. The classical white-box attack JSMA (Papernot et al., 2016) just constructs such an adversarial saliency map and iteratively selects pixels from this map to perturb. But due to the black-box setting, we cannot derive this saliency map directly. Nevertheless, we could find that the region of bright pixels roughly represents the position of the main object in an image, which also accords with our intuition. Inspired by this, we propose to first generate a saliency mask to extract the regions of the main object in an image via image segmentation, and then limit the perturbation search in this smaller but more important regions, as presented in Figure 1.
Several recent studies also propose to generate similar local perturbations but applied in different attack settings with different methods (Dong et al., 2020; Xiang et al., 2021). In particular, Dong et al. (2020) produce saliency maps with the class activation mapping (CAM) (Zhou et al., 2016), which requires the internal information of the model and needs to change the model architecture. This is infeasible in black-box scenarios. Xiang et al. (2021) replace CAM with Grad-CAM (Selvaraju et al., 2017), which does not need to change the architecture of the model. However, both CAM and Grad-CAM still rely on the knowledge of the model, but in black-box setting the internal information of the target model is inaccessible. Thus, we choose salient object segmentation (Zhao & Wu, 2019), which can automatically extract salient object(s) in an image (see Figure 1.(b)). Compared with CAM and Grad-CAM, it doesn’t need any other information except for the input image.
On the other hand, these studies (Dong et al., 2020; Xiang et al., 2021) are applied to transfer-based attacks (actually a grey-box setting), which first build a substitute model to approximate the target model, then use gradient-based white-box attacks to generate adversarial examples, and finally transfer them to attack the target model. For transfer-based attack, we must either train a substitute model from scratch with huge number of queries (Papernot et al., 2017) or assume a pretrained substitute model is trained on the similar data distribution with the target model’s (Liu et al., 2017). And for the latter, the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model(s) and target model. Hence, in this paper we focus on gradient-free black-box attacks (Narodytska & Kasiviswanathan, 2016; Moon et al., 2019; Andriushchenko et al., 2020) that take no account of gradients and search for successful perturbations only according to the model output.
We add these segmentation priors to SOTA black-box attacks and find the imperceptibility of AEs is indeed enhanced but still limited due to ”global perturbation” in local region (See Figure 4 and 5). To
1To distinguish the saliency map generated by the salient object segmentation model below, here we call it as BP saliency map to indicate it is generated via back propagation.
further improve the imperceptibility, our another novelty is that even in the salient region, “saliency” could still be refined. Our inspiration also comes from some DNNs visualization works (Zhou et al., 2016; Zeiler & Fergus, 2014; Olah et al., 2018). For instance, given a dog image, CAM produces a localization heatmap and shows the dog face region is most highly lighted (Zhou et al., 2016). Zeiler & Fergus (2014) systematically cover up different portions of a dog image, and finds when the dog face is obscured, the activity in the feature map and classifier output changes dramatically. In further, by combining internal feature visualization with output prediction, Olah et al. (2018) reveal even in dog’s face region, its ears and eyes seem to be more important when distinguishing dogs. Therefore, we are inspired to assume the salient region in an image is progressive with respect to its impact on model output. If we could find smaller but more salient region, the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced. Thus, we propose our Saliency Attack, a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency.
Our main contributions can be summarized as follows:
• To our best knowledge, we are the first to use salient object segmentation to extract binary salient masks in black-box settings. Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate.
• We propose a new gradient-free black-box attack via refining in salient regions. Compared with the search methods used in other gradient-free attacks, our method is able to generate smaller but effective perturbations, which is interpretable to some extent and can further improve the imperceptibility.
• We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection-based defense like Feature Squeezing.

heading: 2 RELATED WORK

heading: 2.1 BLACK-BOX ATTACKS

text:
Gradient estimation attacks. Gradient estimation attacks first estimate the gradients by querying the target model and then apply them to run white-box attacks (Goodfellow et al., 2015; Carlini & Wagner, 2017; Madry et al., 2018). ZOO attack (Chen et al., 2017) first adopts the symmetric difference quotient to approximate the gradients and then perform Carlini-Wagner (CW) white-box attack (Carlini & Wagner, 2017). AutoZOOM (Tu et al., 2019) uses a random vector based gradient estimation to estimate gradients, which reduces the query number per iteration from 2D in ZOO to N+1 (D is the dimensionality and N is the sample size). To further enhance query efficiency, Ilyas et al. (2019) propose the “tiling trick” that updates a square of pixels simultaneously instead of a single pixel, which dramatically decreases the dimensionality by a factor of k2 (k is tile length).
Gradient-free attacks. Gradient-free attacks take no account of gradients and directly generate AEs with random search or heuristic methods according to the query result. Su et al. (2019) propose one pixel attack that adopts differential evolution algorithm to perturb the most important pixel in the image. Alzantot et al. (2019) propose GenAttack, which use genetic algorithm to generate AEs. To improve the query efficiency, Moon et al. (2019) consider a discrete surrogate optimization problem that transforms the original constraint of a continuous range [− ,+ ] to a discrete set {− ,+ }, achieving a huge reduction in the search space. This is motivated by linear program (LP) where the optimal solution is attained at an extreme point of the feasible set (Schrijver, 1998). Combing tiling trick (Ilyas et al., 2019) and discrete optimization (Moon et al., 2019), Square Attack (Andriushchenko et al., 2020) has obtained the best result on success rate and query performance so far with a randomized search scheme.
Hybrid attacks. Hybrid attacks try to combine multiple types of methods, using candidate adversarial examples generated on substitute model as starting points for further optimization with gradient estimation attacks or gradient-free attacks. Suya et al. (2020) first proposed to generate candidate AEs by white-box projected gradient descent (PGD) attack (Madry et al., 2018), and further optimize them with gradient estimation methods. Wang et al. (2020) combines a white-box attack with a
gradient-free method using microbial genetic algorithm (Harvey, 2009). Similar with transfer-based attacks, hybrid attacks also suffer from the similarity between substitute model(s) and target model.

heading: 2.2 RELATED WORK ON IMPERCEPTIBILITY

text:
The imperceptibility of AEs is vital for attackers and some attacks consider it in different manners. Guo et al. (2019) difine the imperceptibility as the smoothness in the sense of low frequency and thus search for AEs in frequency domain. While Zhang et al. (2020) regard the imperceptibility as the smoothness of visual content in an image and integrate Laplacian smoothing into optimization. Boundary Attack (Brendel et al., 2018) starts from an example that is already adversarial and approaches the original image gradually to reduce the distortion. Instead of global perturbations, some studies (Dong et al., 2020; Xiang et al., 2021) consider local perturbations to reduce the square of perturbations.
On the other hand, how to estimate the imperceptibility of AEs is still a problem. Most of the adversarial attacks use Lp norms (L0, L2 and L∞) to measure the human perceptual distance between the perturbed image and the original one. Nonetheless, Lp norms are found not suitable enough for human vision system (Sharif et al., 2018). To find a proper metric, Fezza et al. (2019) design subjective experiments to obtain the subjective scores on different AEs, and then test various image fidelity assessment (IFA) metrics including Lp norms. Among them, most apparent distortion (MAD) (Larson & Chandler, 2010) metric is found closest to subjective scores. Hence, in this paper we adopt MAD as our main imperceptibility metric to estimate the imperceptibility of AEs.

heading: 3 PROPOSED METHOD

text: In this section, we first introduce the problem formulation of crafting adversarial examples for image classification models, and then detail our approach. The overall flowchart of our proposed Saliency Attack is depicted in Figure 2.

heading: 3.1 PRELIMINARY

text:
Given a well-trained DNN classifier F : [0, 1]d → RK , where d is the dimension of the input x, and K is the number of classes. We denote Fk(x) as the predicted score that x belongs to class k. So the classifier assigns the class that maximizes Fk(x) to the input x. The goal of an untargeted attack is to find an adversarial example xadv , that results in the model misclassification from the ground-truth class y, and meanwhile keeps the distance between the adversarial and benign input smaller than a threshold :
argmax k=1,...,K
Fk(xadv) 6= y, s.t. ||xadv − x||p ≤ , xadv ∈ [0, 1]d (1)
where the other constraint indicates that the generated xadv must be an image in valid range. In this paper we focus on L∞ as our distance norm following (Moon et al., 2019; Andriushchenko et al., 2020).
Traditionally, this task of finding xadv can be rephrased as solving a constrained continuous problem: max
xadv∈[0,1]d L(F (xadv), y), s.t. ||xadv − x||∞ ≤ (2)
where L is a loss function. To improve the query efficiency, Moon et al. (2019) transform the continuous problem into a discrete surrogate problem, where the perturbation δ = xadv − x is
generated at the corner of the L∞ ball. Besides, we propose to limit δ only in the salient region. So our complete optimization problem is:
max xadv∈[0,1]d
L(F (xadv), y), s.t. ||δ||∞ ∈ {− ,+ }, δ ∈ S (3)
where S is the set of the pixels in salient region.

heading: 3.2 SALIENT OBJECT SEGMENTATION

text:
Salient object segmentation, as one changing task in image segmentation domain, aims to automatically and accurately extract salient object in an image. Specifically, given an input image, salient object segmentation model can generate a saliency map where the brighter pixels denote they have higher saliency scores, just as the saliency map shows in Figure 2. Compared with other model visualization methods like CAM (Zhou et al., 2016) and Grad-CAM (Selvaraju et al., 2017), this type of model does not require any information other than the input image, which is very suitable for the black-box setting. We adopt the Pyramid Feature Attention (PFA) network (Zhao & Wu, 2019) that achieves the state-of-the-art performance in multiple datasets. After obtaining the saliency map where all pixel values are between 0 and 1, we use a binarization threshold φ to transform the saliency map to a binary salient mask. The binarization can be expressed as
s∗i,j =
{ 0 si,j < φ
1 si,j ≥ φ (4)
where s is the saliency map and s∗ is the binary salient mask, sij and s∗i,j are the corresponding value at the position i, j. With this saliency mask s∗, we could limit the perturbation search in the salient region.
3.3 REFINING PERTURBATION IN SALIENT REGION
In this part, we design a search algorithm to refine perturbations based on a tree structure for an image by using the idea of depth-first search. As shown in Figure 3, an input image merged with its saliency mask is split into four initial blocks (a coarse grid). Then we try to add a + or − perturbation on each initial block individually and find the best block that maximizes the loss function for further perturbation. After finding the best one among all initial blocks, we can further split this block into smaller blocks (a finer grid), and again try to add a perturbation on each block individually (at this time we just flip the perturbation for convenience e.g. + to − ) to find the best smaller block. We iterate this process until the minimal block (e.g. 1 pixel) or no smaller block has a better loss. Then we go back to the last level of split blocks and use the second best block for further perturbation.
We use the loss function from CW attack (Carlini & Wagner, 2017) for untargted attack:
L(x) = −max(Z(xadv)o −max i 6=o (Z(xadv)i), 0) (5)
where Z(xadv)o is the logit with respect to the true class of the original image. In this way, the loss function is imposed to leave a margin between the true class and other classes. The refining process is presented in Algorithm 1, and the overall saliency attack is given in Algorithm 2.

heading: 4 EXPERIMENTS

text:
In this paper, we evaluate the performance of the Saliency Attack comparing against Boundary Attack (Brendel et al., 2018), TVDBA (Li & Chen, 2021) Parsimonious attack (Moon et al., 2019),
Algorithm 1: Refine Input: block b, block size k Output: adversarial example xadv
1 k′ is the block size of b; 2 {b1, b2, ..., bn} ← split b into (k′/k)2 finer blocks; 3 {bπ(1), bπ(2), ..., bπ(n)} ← sort {b1, b2, ..., bn} in descending order according to L(b), which is
calculated by using Eq. 5; 4 for each block e ∈ {bπ(1), bπ(2), ..., bπ(n)} do 5 if L(e) > L̂ then 6 xadv ← x+ δ ∪ {e}; 7 L̂← L(e); 8 if k > 1 then 9 k ← k/2;
10 Recursively call xadv ← Refine(e, k); 11 end 12 end 13 end
Algorithm 2: Saliency Attack Input: original image x, initial block size kint, query budget Output: adversarial example xadv
1 δ ← ∅ is the perturbation; 2 while kint > 1 and not exceeding query budget do 3 Running Algorithm 2: xadv ← Refine(x, kint); 4 kint ← kint/2; 5 end
Square attack (Andriushchenko et al., 2020) and their modified version equipped with segmentation priors. Among them, Boundary Attack is able to constantly reduce the distortion to a very small magnitude, TVDBA tries to minimize the distortion via integrating Structural SIMilarity (SSIM) (Wang et al., 2004) into the loss, and Square Attack can achieve the current state-of-the-art query efficiency and success rate in the black-box setting. We consider L∞ threat model on ImageNet dataset (Deng et al., 2009), set to 0.05 in [0,1] scale, and test 1000 randomly selected exmaples in all experiments. The threshold φ to produce binary salient masks is chosen to be 0.1. All parameters of the compared attacks remain consistent with those recommended in their papers. We use Inception v3 (Szegedy et al., 2016) as the target model, and different query budgets {3000, 10000, 30000} for untargeted attack. For performance metrics, we employ commonly used success rate (SR) and average number of queries (Avg. queries). To evaluate the imperceptibility of AEs, we consider L0, L2 and MAD (see the appendix for details), which is validated as closest to human vision system among existing IFA metrics (Fezza et al., 2019). All these three imperceptibility metrics are the smaller the better.

heading: 4.1 SOTA BLACK-BOX ATTACKS WITH SEGMENTATION PRIORS

text:
To verify the feasibility of our segmentation priors, we modify Parsimonious Attack and Square Attack so that the perturbations are limited in salient regions. We show the results in Table 1 and some examples in Figure 4. From the examples, we can easily find that the perturbations of original attacks are very obvious in the entire image due to global perturbations, while their segmentation versions are relatively more imperceptible since no perturbation exists in background regions.
Besides, through quantitative analysis, the attacks equipped with segmentation priors can indeed achieve much better imperceptibility performance with around one third improvement in L2 and MAD metric. L0 is also restricted to less than 50%, which implies the average square of salient regions among all examples only occupies less than half of the whole image. And this improvement only comes at little cost in SR and Avg. queries, which is still in the same order of magnitude. We also present the change of MAD scores in Figure 9 in appendix. Thus, the results demonstrate that the segmentation prior is indeed able to enhance the imperceptibility of other black-box attacks.

heading: 4.2 COMPARISON OF SALIENCY ATTACK WITH OTHER ATTACKS

text:
Although the segmentation prior is helpful to the imperceptibility, the perturbations generated by other attacks are still global, taking up almost all salient regions. Hence, we design our Saliency Attack via further refining perturbations in salient regions. Original image are resized to 256× 256 and we set the initial block size to 16. For an imperceptible AE, a low MAD value is important. So besides SR, we use a new metric SRtrue. It denotes the rate of successful AEs whose MAD values are below some threshold. Here we choose 30 as the threshold because we find the AEs with MAD ≤ 30 are basically imperceptible to human eyes (See Figure 10 in appendix for details). The comparison results are given in Table 2. We can find in all query budgets, our Saliency Attack achieves outstanding performance with a huge gap in terms of SRtrue and three imperceptibility metrics. Although TVDBA, Parsimonious and Square Attack can obtain a better SR, their successful AEs are not true imperceptible, and will be easily detected by some defenses or humans. For Boundary Attack, it can gradually make progress on SRtrue, L2 and MAD as the query budget increases, but they are still much worse than Saliency Attack. This is because it takes at least hundreds of thousands of queries for Boundary Attack to converge (Brendel et al., 2018), which is infeasible in practical. We further test more eamples to verify the statistical significance in Table 5, and present a comparison of SRtrue versus query number in Figure 11 in appendix.
From the exhabited examples in Figure 5, we can find the AEs of Boundary Attack contain obvious coarse textures because of inadequate query budget. In addition, even restricted in the same salient regions, the perturbations of Parsimonious-seg and Square seg attack are complex and irregular, while our perturbations are smaller and more important, roughtly corresponding to the bright pixels in BP saliency map. They further represent the postions of dogs’ noses or ears, which accord with our inspiration before.

heading: 4.3 HYPERPARAMETER TUNING

text:
Our Saliency Attack contains only one hyperparameter namely the initial block size k which determines the first level of split blocks. We test the effect of different k on SR, imperceptibility and query number respectively in Figure 6. As k decreases, SR and imperceptibility can be improved and SR reaches the peak when k equals 16. This is because with smaller initial blocks, Saliency Attack can search for perturbations more finely and accurately leading to higher SR and better imperceptibility performance. Meanwhile inevitably more queries are needed, especially for sorting initial blocks. That is why under a limited query budget like 10000, a turning point occurs in SR.
We also show some examples in Figure 7. It can be observed that as k decreases, the generated perturbations also become smaller but more salient. For instance, in the first row the perturbation with k = 128 roughly covers the region of the dog face, which is also the brightest region in the BP saliency map. While the perturbation with k = 32 or k = 16 focuses on smaller region of the dog ear. This indicates that our Saliency Attack could find smaller and more important perturbations progressively as the initial block size decreases. And the perturbations are interpretable to some extent like perturbing the specific regions of dog’s ears, eyes or nose.

heading: 4.4 ABLATION STUDY

text: We do ablation study of Saliency Attack, including refining in salient region, in non-salient region and without saliency (refining in the whole image). We also design a greedy search as baseline to demonstrate our Refine search. We test multiple block sizes for greedy search and use 32 as the best choice (see Table 6 in appendix). The results and some examples are given in Table 3 and Figure 8. Notice that refining in salient region and refining without saliency generate the same or almost the same perturbations, which means the salient regions indeed contain useful parts and enhance the query efficiency by limiting the search space. But for refining in non-salient region, its perturbation is more complex and visible with worse query efficiency and SR. Compared with greedy search, our Refine search has much better query efficiency and SRtrue, which proves its superiority. Hence, we can conclude that both salient region and Refine search facilitate our Saliency Attack.

heading: 4.5 ATTACKING DETECTION-BASED DEFENSE

text: To further validate the superiority of our method, we attack against one detection-based defense Feature Squeezing (Xu et al., 2018), which applies transformations to input image and judge whether it is adversarial through the stability of model output. We consider the detection rate of both successful adversarial examples (SAEs) and failure adversarial examples (FAEs). This is because if we could detect FAEs, we can raise the alarm that some potential attacker is attacking the model. From Table 4, we can find our Saliency attack is able to achieve lower detection rate in both SAEs and FAEs than other attacks, which proves the imperceptibility of our AEs from the defensive perspective.

heading: 5 CONCLUSION

text: In this paper, we propose to provide segmentation priors for black-box attacks to improve the imperceptibility of adversarial examples, whose perturbations are limited in salient regions. We utilize a salient object segmentation model to produce such saliency maps with no need for any information other than the input image. Experiments show that SOTA black-box attacks equipped with segmentation priors can indeed achieve better imperceptibility performance. Furthermore, we devise a new gradient-free black-box Saliency attack that further enhances the imperceptibility via refining in salient regions. The perturbations generated are much smaller, imperceptible and interpretable to some extent. Finally, we also attack against some detection-based defense and the results show that the adversarial examples generated by our attack are harder to be detected.

heading: A APPENDIX

text:
A.1 MAD METRIC
Most Apparent distortion (MAD) metric (Larson & Chandler, 2010) is one of the state-of-the-art full-reference image quality assessment methods. MAD attempts to merge two separate strategies for two kinds of distorted images respectively. For high-quality images with near-threshold distortion (just visible), MAD focuses on detection-based strategy to look for distortions in the presence of the image. While for low-quality images with suprathreshold distortion (clearly visible), MAD focuses on appearance-based strategy to look for image content in the presence of the distortions. MAD will control the weight of two strategies according to the type of distorted images. The calculation process of MAD can be summarized as following steps and we recommend interested readers to read the original literature.
1. Compute locations of visible distortions based on luminance images. 2. Combine the visibility map with local error image. 3. Decompose both the distorted and original images into log-Gabor subbands. 4. Calculate different statistics of each subband. 5. Calculate the adaptive blending score.
A.2 SCATTER PLOT OF MAD SCORES
Equipped with segmentation priors, 68.1% and 94.3% examples of Parsimonious-seg attack and Square-seg attack have better MAD scores compared with their original version respectively. Since Parsimonious Attack uses a greedy local search while Square attack adopts a random search, it is obvious that limiting the search in salient regions is more helpful to Square Attack.
A.3 THRESHOLD FOR MAD METRIC
A.4 TRUE SUCCESS RATE VERSUS QUERY NUMBER PLOT
Here we further present a comparison of SRtrue versus query number under different MAD thresholds. It shows that Square Attack and Parsimonious Attack can generate some successful AEs with
very few queries due to global perturbations and some tricks like initializing perturbations with random vertical stripes (Andriushchenko et al., 2020), but they lack the ability to further improve the imperceptibility. Instead, Saliency Attack conservatively select small regions to perturb, hence the query efficiency is a little lower than Square Attack and Parsimonious Attack at the beginning but soon exceeds them.
SR-query convergence plot under MAD <= 20
SR-query convergence plot under MAD <= 30
SR-query convergence plot under MAD <= 40
A.5 COMPARISON OF SALIENCY ATTACK WITH OTHER ATTACKS
We randomly choose 10,000 examples from the ImageNet validation set and divide them into 10 groups. Then we test different attacks under 3000 query budget and report their mean and standard deviation (SD). The best results are recorded in bold based on Wilcoxon signed-rank test with significance level at 0.05. We can find that our Saliency Attack is significantly better than all the baselines.
A.6 GREEDY SEARCH IN SALIENT REGION WITH DIFFERENT BLOCK SIZES
We test different block sizes for greedy search in salient region, and choose the best one to be compared in ablation study.

title: IMPERCEPTIBLE BLACK-BOX ATTACK VIA REFINING IN SALIENT REGION


summaries: The paper proposes a GNN-based extension of transformers, which have been shown to be effective for reaction prediction etc. before. In particular, the GNN-based embedding of molecules in the reaction embeddings overcomes the artificial bias inherent in the often applied sequence embeddings. The experiments show that there are sometimes increases of performance in reaction and retrosynthesis prediction - and the approach could be applied to similar problems.

summaries: This paper proposes a graph-to-sequence architecture called Graph2SMILES for the retrosynthesis and the reaction outcome prediction. Graph2SMILES uses an attention-augmented D-MPNN encoder to capture the local information and a global attention encoder with graph-aware positional embeddings to capture the global information. Experiments show that Graph2SMILES is competitive with Transformer baselines but does not outperform state-of-the-art methods on tasks of the one-step retrosynthesis and the reaction outcome prediction.

summaries: The authors proposed a new method for retrosynthesis, which does not require the mapping numbers and extracting templates from the literature.  Basically, the model consists of a graph-based encoder and a sequence based encoder. The encoder consists of local aggregation from neighbors and global attention using a new positional method. The decoder is a Transformer model with relative positional encoding.  The method achieved promising results on several retrosynthesis datasets.

summaries: This paper proposes a graph-to-SMILES framework, which incorporates several recently developed engineering techniques from the community, for synthesis planning and reaction outcome prediction tasks. The proposed method leverages graph neural networks and Transformer attention model to encode the graph inputs and then utilizes a Transformer decoder to generate the SMILES string as outputs. Experiments on benchmark retrosynthesis and reaction prediction tasks show that the proposed approach outperformed the vanilla SMILES-to-SMILES transformer baseline, but obtained inferior results than some other advanced methods. The paper is interesting, but both the technical novelty and the experimental studies are weak to me.

abstractText: Synthesis planning and reaction outcome prediction are two fundamental problems in computer-aided organic chemistry for which a variety of data-driven approaches have emerged. Natural language approaches that model each problem as a SMILES-to-SMILES translation lead to a simple end-to-end formulation, reduce the need for data preprocessing, and enable the use of well-optimized machine translation model architectures. However, SMILES representations are not an efficient representation for capturing information about molecular structures, as evidenced by the success of SMILES augmentation to boost empirical performance. Here, we describe a novel Graph2SMILES model that combines the power of Transformer models for text generation with the permutation invariance of molecular graph encoders that mitigates the need for input data augmentation. As an end-to-end architecture, Graph2SMILES can be used as a drop-in replacement for the Transformer in any task involving molecule(s)-to-molecule(s) transformations. In our encoder, an attention-augmented directed message passing neural network (D-MPNN) captures local chemical environments, and the global attention encoder allows for long-range and intermolecular interactions, enhanced by graph-aware positional embedding. Graph2SMILES improves the top-1 accuracy of the Transformer baselines by 1.7% and 1.9% for reaction outcome prediction on USPTO_480k and USPTO_STEREO datasets respectively, and by 9.8% for one-step retrosynthesis on the USPTO_50k dataset.

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: In 3rd International Conference on Learning Representations,

authors:
Javier L. Baylon
Nicholas A. Cilfone
Jeffrey R. Gulcher
Thomas W. Chittenden

title: Enhancing retrosynthetic reaction prediction with deep learning using multiscale reaction classification

venue: Journal of Chemical Information and Modeling,

authors:
Hangrui Bi
Hengyi Wang
Chence Shi
Connor Coley
Jian Tang
Hongyu Guo

title: Nonautoregressive electron redistribution modeling for reaction prediction

venue: In Proceedings of the 38th International Conference on Machine Learning,

title: SMILES enumeration as data augmentation for neural network modeling of molecules

authors:
John Bradshaw
Matt J. Kusner
Brooks Paige
Marwin H.S. Segler
José Miguel

title: HernándezLobato. A generative model for electron paths

venue: In International Conference on Learning Representations,

authors:
Shaked Brody
Uri Alon
Eran Yahav

title: How attentive are graph attention networks? CoRR, 2021

venue: URL https://arxiv.org/abs/2105.14491

authors:
Deng Cai
Wai Lam

title: Graph transformer for graph-to-sequence learning

venue: Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Benson Chen
Regina Barzilay
Tommi S. Jaakkola

title: Path-augmented graph transformer network. CoRR, 2019

venue: URL http://arxiv.org/abs/1905.12712

authors:
Benson Chen
Tianxiao Shen
Tommi S. Jaakkola
Regina Barzilay

title: Learning to make generalizable and diverse predictions for retrosynthesis

authors:
Shuan Chen
Yousung Jung

title: Deep retrosynthetic reaction prediction using local reactivity and global attention

venue: JACS Au,

authors:
Kyunghyun Cho
Bart van Merriënboer
Caglar Gulcehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using RNN encoder–decoder for statistical machine translation

venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Connor W. Coley
Luke Rogers
William H. Green
Klavs F. Jensen

title: Computer-assisted retrosynthesis based on molecular similarity

venue: ACS Central Science,

authors:
Connor W. Coley
Wengong Jin
Luke Rogers
Timothy F. Jamison
Tommi S. Jaakkola
William H. Green
Regina Barzilay
Klavs F. Jensen

title: A graph-convolutional neural network model for the prediction of chemical reactivity

venue: Chem. Sci.,

authors:
E.J. Corey
W. Todd Wipke

title: Computer-assisted design of complex organic syntheses

authors:
E.J. Corey
Richard D. Cramer
W. Jeffrey Howe

title: Computer-assisted synthetic analysis for complex molecules. methods and procedures for machine generation of synthetic intermediates

venue: Journal of the American Chemical Society,

authors:
E.J. Corey
X.M Cheng

title: The Logic of Chemical Synthesis

authors:
Hanjun Dai
Chengtao Li
Connor Coley
Bo Dai
Le Song

title: Retrosynthesis prediction with conditional graph logic network

venue: Advances in Neural Information Processing Systems,

authors:
Zihang Dai
Zhilin Yang
Yiming Yang
Jaime Carbonell
Quoc Le
Ruslan Salakhutdinov

title: Transformer-XL: Attentive language models beyond a fixed-length context

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

authors:
Kien Do
Truyen Tran
Svetha Venkatesh

title: Graph transformation policy network for chemical reaction prediction

venue: In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,

authors:
Hongliang Duan
Ling Wang
Chengyun Zhang
Lin Guo
Jianjun Li

title: Retrosynthesis with attention-based NMT model and chemical analysis of “wrong

venue: predictions. RSC Adv.,

authors:
Meng-Hao Guo
Junxiong Cai
Zheng-Ning Liu
Tai-Jiang Mu
Ralph R. Martin
Shi-Min Hu

title: PCT: point cloud transformer

authors:
Dan Hendrycks
Kevin Gimpel

title: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, 2016

venue: URL http://arxiv.org/abs/1606.08415

authors:
Weihua Hu
Bowen Liu
Joseph Gomes
Marinka Zitnik
Percy Liang
Vijay Pande
Jure Leskovec

title: Strategies for pre-training graph neural networks

authors:
R. Irwin
S. Dimitriadis
J. He
E. Bjerrum

title: Chemformer: A pre-trained transformer for computational chemistry, 2021

authors:
Wengong Jin
Connor Coley
Regina Barzilay
Tommi Jaakkola

title: Predicting organic reaction outcomes with Weisfeiler-Lehman network

venue: In Advances in Neural Information Processing Systems,

authors:
Wengong Jin
Regina Barzilay
Tommi Jaakkola

title: Junction tree variational autoencoder for molecular graph generation

venue: In Proceedings of the 35th International Conference on Machine Learning,

authors:
Eunji Kim
Dongseon Lee
Youngchun Kwon
Min Sik Park
Youn-Suk Choi

title: Valid, plausible, and diverse retrosynthesis using tied two-way transformers with latent variables

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Guillaume Klein
Yoon Kim
Yuntian Deng
Jean Senellart
Alexander Rush

title: OpenNMT: Opensource toolkit for neural machine translation

venue: In Proceedings of ACL 2017, System Demonstrations,

authors:
Bumju Kwak
Jeonghee Jo
Byunghan Lee
Sungroh Yoon

title: Geometry-aware transformer for molecular property prediction. CoRR, 2021

venue: URL https://arxiv.org/abs/2106

title: Rdkit: Open-source cheminformatics software, 2016

venue: URL https://github

authors:
Alpha A. Lee
Qingyi Yang
Vishnu Sresht
Peter Bolgar
Xinjun Hou
Jacquelyn L. KlugMcLeod
Christopher R. Butler

title: Molecular transformer unifies reaction prediction and retrosynthesis across pharma

authors:
Kangjie Lin
Youjun Xu
Jianfeng Pei
Luhua Lai

title: Automatic retrosynthetic route planning using template-free models

authors:
Bowen Liu
Bharath Ramsundar
Prasad Kawthekar
Jade Shi
Joseph Gomes
Quang Luu Nguyen
Stephen Ho
Jack Sloane
Paul Wender
Vijay Pande

title: Retrosynthetic reaction prediction using neural sequence-to-sequence models

title: Extraction of chemical structures and reactions from the literature

venue: (doctoral thesis),

authors:
Pierre Mahé
Nobuhisa Ueda
Tatsuya Akutsu
Jean-Luc Perret
Jean-Philippe Vert

title: Extensions of marginalized graph kernels

venue: In Proceedings of the Twenty-First International Conference on Machine Learning,

authors:
Kelong Mao
Xi Xiao
Tingyang Xu
Yu Rong
Junzhou Huang
Peilin Zhao

title: Molecular graph enhanced transformer for retrosynthesis

venue: prediction. Neurocomputing,

authors:
Juno Nam
Jurae Kim

title: Linking the neural machine translation and the prediction of organic chemistry reactions

authors:
Greeshma Neglur
Robert L. Grossman
Bing Liu

title: Assigning unique keys to chemical compounds for data integration: Some interesting counter examples

venue: In Data Integration in the Life Sciences,

authors:
Wesley Wei Qian
Nathan T. Russell
Claire L.W. Simons
Yunan Luo
Martin D. Burke
Jian Peng

title: Integrating deep neural networks and symbolic inference for organic reactivity prediction

venue: ChemRxiv,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J. Liu

title: Exploring the limits of transfer learning with a unified text-totext transformer

venue: Journal of Machine Learning Research,

authors:
David Rogers
Mathew Hahn

title: Extended-connectivity fingerprints

authors:
Yu Rong
Yatao Bian
Tingyang Xu
Weiyang Xie
Ying WEI
Wenbing Huang
Junzhou Huang

title: Self-supervised graph transformer on large-scale molecular data

authors:
Mikołaj Sacha
Mikołaj Błaż
Piotr Byrski
Paweł Dąbrowski-Tumański
Mikołaj Chromiński
Rafał Loska
Paweł Włodarczyk-Pruszyński
Stanisław Jastrzębski

title: Molecule edit graph attention network: Modeling chemical reactions as sequences of graph edits

authors:
Philippe Schwaller
Théophile Gaudin
Dávid Lányi
Costas Bekas
Teodoro Laino

title: Found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequenceto-sequence models

authors:
Philippe Schwaller
Teodoro Laino
Théophile Gaudin
Peter Bolgar
Christopher A. Hunter
Costas Bekas
Alpha A. Lee

title: Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction

authors:
Marwin H.S. Segler
Mark P. Waller

title: Neural-symbolic machine learning for retrosynthesis and reaction prediction

venue: Chemistry – A European Journal,

authors:
Seung-Woo Seo
You Young Song
June Yong Yang
Seohui Bae
Hankook Lee
Jinwoo Shin
Sung Ju Hwang
Eunho Yang

title: Gta: Graph truncated attention for retrosynthesis

authors:
Peter Shaw
Jakob Uszkoreit
Ashish Vaswani

title: Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)

authors:
Chence Shi
Minkai Xu
Hongyu Guo
Ming Zhang
Jian Tang

title: A graph to graphs framework for retrosynthesis prediction

venue: Proceedings of the 37th International Conference on Machine Learning,

authors:
Vignesh Ram Somnath
Charlotte Bunne
Connor W. Coley
Andreas Krause
Regina Barzilay

title: Learning graph models for template-free retrosynthesis

venue: International Conference on Machine Learning (ICML) Workshop on Graph Representation Learning and Beyond (GRL+),

authors:
Ruoxi Sun
Hanjun Dai
Li Li
Steven Kearnes
Bo Dai

title: Energy-based view of retrosynthesis

authors:
Sara Szymkuć
Ewa P. Gajewska
Tomasz Klucznik
Karol Molga
Piotr Dittwald
Michał Startek
Michał Bajczyk
Bartosz A. Grzybowski

title: Computer-assisted synthetic planning: The end of the beginning

venue: Angewandte Chemie International Edition,

authors:
I.V. Tetko
P. Karpov
R. Van Deursen

title: State-of-the-art augmented NLP transformer models for direct and single-step retro synthesis

venue: Nature Communications,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Ł ukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Petar Veličković
Guillem Cucurull
Arantxa Casanova
Adriana Romero
Pietro Liò
Yoshua Bengio

title: Graph attention networks

authors:
Jiahao Wang
Shuangjia Zheng
Jianwen Chen
Yuedong Yang

title: Meta learning for low-resource molecular optimization

authors:
Xiaorui Wang
Yuquan Li
Jiezhong Qiu
Guangyong Chen
Huanxiang Liu
Benben Liao
ChangYu Hsieh
Xiaojun Yao

title: Retroprime: A diverse, plausible and transformer-based method for single-step retrosynthesis predictions

venue: Chemical Engineering Journal,

authors:
Xing Wang
Zhaopeng Tu
Longyue Wang
Shuming Shi

title: Self-attention with structural position representations

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

title: SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

venue: Journal of Chemical Information and Computer Sciences,

authors:
Kun Xu
Lingfei Wu
Zhiguo Wang
Yansong Feng
Vadim Sheinin

title: SQL-to-text generation with graph-to-sequence model

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Kun Xu
Lingfei Wu
Zhiguo Wang
Yansong Feng
Michael Witbrock
Vadim Sheinin

title: Graph2seq: Graph to sequence learning with attention-based neural networks

authors:
Chaochao Yan
Qianggang Ding
Peilin Zhao
Shuangjia Zheng
Jinyu Yang
Yang Yu
Junzhou Huang

title: Retroxpert: Decompose retrosynthesis prediction like a chemist

authors:
Kevin Yang
Kyle Swanson
Wengong Jin
Connor Coley
Philipp Eiden
Hua Gao
Angel GuzmanPerez
Timothy Hopper
Brian Kelley
Miriam Mathea
Andrew Palmer
Volker Settels
Tommi Jaakkola
Klavs Jensen
Regina Barzilay

title: Analyzing learned molecular representations for property prediction

authors:
Chengxuan Ying
Tianle Cai
Shengjie Luo
Shuxin Zheng
Guolin Ke
Di He
Yanming Shen
Tie-Yan Liu

title: Do transformers really perform bad for graph representation

authors:
Sanghyun Yoo
Young-Seok Kim
Kang Hyun Lee
Kuhwan Jeong
Junhwi Choi
Hoshik Lee
Young Sang Choi

title: Graph-aware transformer: Is attention all graphs need

authors:
Shuangjia Zheng
Jiahua Rao
Zhongyue Zhang
Jun Xu
Yuedong Yang

title: Predicting retrosynthetic reactions using self-corrected transformer neural networks

authors:
Jinhua Zhu
Yingce Xia
Tao Qin
Wengang Zhou
Houqiang Li
Tie-Yan Liu

title: Dual-view molecule pre-training

heading: 1 INTRODUCTION

text:
Retrosynthetic analysis (Corey, 1988; Corey & Cheng, 1989) and its reverse problem, reaction outcome prediction (Corey & Wipke, 1969), are two fundamental problems in computer-aided organic synthesis. The former tries to propose possible reaction precursors given the desirable product, whereas the latter aims to predict the major products given reactants. Historically, they were tackled using rule-based expert systems such as LHASA (Corey et al., 1972). Recent developments in machine learning have led to a number of new template-based, graph edit-based, and translation-based methods, for which we give a detailed review in Section 3.1. For both tasks, translation-based approaches have grown popular, possibly because the end-to-end formulation is procedurally simple. Since most organic molecules can be represented as SMILES strings (Neglur et al., 2005), retrosynthesis can be cast as a translation from product SMILES to reactant SMILES (Liu et al., 2017), and so can reaction outcome prediction (Nam & Kim, 2016). Modeling these tasks as machine translation problems enables the use of neural architectures that are well-studied and well-optimized in the field of Natural Language Processing (NLP). Several of the best performing models across multiple benchmark datasets (Tetko et al., 2020; Irwin et al., 2021; Sun et al., 2020; Seo et al., 2021; Wang et al., 2021b) have used the Transformer architecture (Vaswani et al., 2017) as the backbone on SMILES representations, showing the effectiveness of translation-based formulation.
However, SMILES representations do not provide bijective mappings to molecular structure. As a result, data augmentation with chemically-equivalent SMILES (Bjerrum, 2017) has become a common practice to improve empirical performance. Augmenting the training data with just 1 equivalent reaction SMILES by permuting both the inputs and the outputs can already produce noticeable improvements of 0.8% to 4.3% (Schwaller et al., 2019; Seo et al., 2021). Incorporating 9 (Wang et al.,
2021b), or up to 100 (Tetko et al., 2020) equivalent SMILES can provide additional gains. While these efforts demonstrate the effectiveness of SMILES augmentation, this introduces a non-trivial choice as to how much augmentation should be done. The performance has not saturated even with 100 augmented SMILES (Tetko et al., 2020), but this may significantly complicate the model pipeline, especially during test time which typically necessitates de-duplication, ensembling, and heuristic-based scoring (Wang et al., 2021b; Tetko et al., 2020).
In this paper, we propose a novel graph-to-sequence architecture called Graph2SMILES to solve the tasks of retrosynthesis and reaction prediction. We first design a sequential graph encoder with an attention-augmented directed message passing neural network (D-MPNN) based on Yang et al. (2019) and optionally augmented by attention-based message updates, followed by a Transformerbased global attention encoder with graph-aware positional embedding. We then pair the graph encoder with a Transformer decoder to transform molecular graph inputs into SMILES outputs, without using sequence representations of input SMILES at all. As such, we guarantee the permutation invariance of Graph2SMILES to the input, eliminating the need for input-side augmentation altogether. Our main contributions can be summarized as follows:
1. We propose a Graph2SMILES architecture with two encoder components modeling local and global atomic interactions respectively, to address forward prediction and one-step retrosynthesis as graph-to-sequence tasks.
2. We design a graph-aware positional embedding to further enhance performance. It is easily generalizable to graphs containing two or more molecules, while not requiring any pretraining or joint training with auxiliary tasks.
3. We demonstrate the adequacy of graph representations alone by showing that Graph2SMILES outperforms Transformer baselines on predictive chemistry tasks without needing any input-side SMILES augmentation.
Our Graph2SMILES architecture achieves state-of-the-art top-1 accuracy on common benchmarks among methods that do not make use of reaction templates, atom mapping, pretraining, or data augmentation strategies. We emphasize that Graph2SMILES is a backbone architecture, and hence a drop-in replacement for the Transformer model. As such, Graph2SMILES can be plugged into any method for molecular transformation tasks that uses the Transformer model, while retraining the benefits from techniques or features orthogonal to the architecture itself.

heading: 2 METHODS

heading: 2.1 GRAPH AND SEQUENCE REPRESENTATIONS OF MOLECULES

text:
There are multiple ways of representing molecular structures, such as by molecular fingerprints (Rogers & Hahn, 2010), by SMILES strings (Weininger, 1988), or as molecular graphs with atoms as nodes and bonds as edges. We represent the input molecule(s) as graphs, and the output molecule(s) as SMILES strings, thus modeling both reaction prediction and retrosynthesis as graph-to-sequence transformations.
Formally, let Gin denotes the molecular graph input, which can contain multiple subgraphs for different molecules, with a total of N atoms. Following the convention in Somnath et al. (2020), we describe Gin = (V, E) with atoms V and bonds E . Each atom u ∈ V has a feature vector xu ∈ Ra, and each directed bond (u, v) ∈ E from atom u to v has its feature vector as xuv ∈ Rb. The details of the atom and bond features used can be found in Appendix A. We build the input molecular graphs from their SMILES strings with RDKit (Landrum, 2016). Note that all feature vectors are invariant to the order of atoms and bonds, as well as to how the original SMILES strings are written. We represent the output as a sequence of SMILES tokens Sout = {s1, s2, . . . , sn}, where the tokens {si} are obtained from the canonical SMILES using the regex tokenizer in Schwaller et al. (2019).

heading: 2.2 GRAPH2SMILES

text:
The Graph2SMILES model is a variant of the encoder-decoder model (Cho et al., 2014) commonly used for machine translation. Figure 1 displays the architecture of Graph2SMILES with the permutation invariant graph encoding process shown within the blue dashed box. We replace the
encoder part of the standard Transformer model (Vaswani et al., 2017) used in Molecular Transformer (Schwaller et al., 2019) with a novel attention-augmented directed message passing encoder, followed by a global attention encoder with carefully designed graph-aware positional embedding. Each module has its intuitive function: the D-MPNN captures the local chemical context, the global attention encoder allows for global-level information exchange, and the graph-aware positional embedding enables the attention encoder to make use of topological information more explicitly. The permutation invariant encoding process eliminates the need for SMILES augmentation for the input side altogether, simplifying data preprocessing and potentially saving training time.

heading: 2.2.1 ATTENTION AUGMENTED DIRECTED MESSAGE PASSING ENCODER

text:
The first module of the graph encoder is a D-MPNN (Yang et al., 2019) with Gated Recurrent Units (GRUs) (Cho et al., 2014) used for message updates (Jin et al., 2018; Somnath et al., 2020). Unlike atom-oriented message updates in edge-aware MPNNs (Hu et al., 2020; Yan et al., 2020; Mao et al., 2021; Wang et al., 2021a), updates in D-MPNN are oriented towards directed bonds to prevent totters, or messages being passed back-and-forth between neighbors (Mahé et al., 2004; Yang et al., 2019). We optionally augment the D-MPNN with attention-based message updates inspired from Graph Attention Network (Veličković et al., 2018; Brody et al., 2021). We term this variant as Directed Graph Attention Network (D-GAT) and refer to the original D-MPNN variant used in Somnath et al. (2020) as Directed Graph Convolutional Network (D-GCN), while keeping both D-GAT and D-GCN as possible design choices for the D-MPNN.
For D-GAT, at each message passing step t, the message mt+1uv associated with each directed bond (u, v) ∈ E is updated using
suv = AttnSum ( xu,xuv, { mtwu } w∈N(u)\v ) (1)
zuv = σ (Wz [xu;xuv; suv] + bz) (2) ruv = σ (Wr [xu;xuv; suv] + br) (3)
m̃uv = tanh (W [xu;xuv] +Uruv + b) (4)
mt+1uv = (1− zuv) suv + zuv m̃uv (5)
where Wz , Wr, W , U and bz , br, b are the learnable weights and biases respectively. σ is the sigmoid function, ";" indicates concatenation, and is the element-wise product. AttnSum is the attention-based message aggregation defined as
ewu = a TLeakyReLU ( Wqk [ xu;xuv;m t wu ] + bqk ) (6) awu = exp (ewu)∑
w∈N(u)\v exp (ewu) (7)
AttnSum ( xu,xuv, { mtwu } w∈N(u)\v ) = ∑ w∈N(u)\v awu ( Wvm t wu + bv ) (8)
where Wqk, bqk, a are the learnable parameters for the attention scores, and Wv , bv are the parameters for the value vectors. We have omitted a self-loop in message aggregation, as it did not have any noticeable effect on performance from our experiments. In Eqn (3) we simplify the reset gate to be shared for all incoming edges, in contrast to Somnath et al. (2020) where a separate reset gate is defined for each edge.
Lastly, after T iterations, we obtain the atom representations hu with similar attention-based aggregation (with different parameters) over the bond messages coming into each atom u, followed by a single output layer with weight Wo and GELU activation (Hendrycks & Gimpel, 2016).
mu = AttnSum ′ ( xu, { m(T )wu } w∈N(u) ) (9)
hu = GELU (Wo [xu;mu]) (10) We use multi-headed attention in our formulation similar to Brody et al. (2021).

heading: 2.2.2 GLOBAL ATTENTION ENCODER WITH GRAPH-AWARE POSITIONAL EMBEDDING

text:
To capture global interactions, the atom representations coming out of the D-MPNN are fed into a global attention encoder, which is a variant of the Transformer encoder. We incorporate graphaware positional embedding, adapted from the relative positional embedding used in TransformerXL (Dai et al., 2019b) as follows. Firstly, in the standard Transformer either with sinusoidal encoding (Vaswani et al., 2017; Schwaller et al., 2019) or learnable (Devlin et al., 2019) absolute positional embedding, the attention score between atoms u and v can be decomposed as
eabsu,v = h T uW̃ T q W̃khv + h T uW̃ T q W̃kpv + p T uW̃ T q W̃khv + p T uW̃ T q W̃kpv (11)
where W̃q ,W̃k are weights for the keys and queries, and pu, pv are the absolute positional encoding or embedding corresponding to atoms u and v. Similar to Transformer-XL, we reparameterize the four terms. Instead of using sequence-based relative positional embedding ru−v , we use a learnable embedding term ru,v that is dependent on the shortest path length between u and v
erelu,v = ( hTuW̃ T q + c T ) W̃khv + ( hTuW̃ T q + d T ) W̃k,Rru,v
= ( hTuW̃ T q + c T ) W̃khv + ( hTuW̃ T q + d T ) r̃u,v (12)
The trainable biases are renamed as c and d to avoid confusion and shared across all layers. Intuitively, the two terms in Eqn (12) model the interactions between inputs (hu and hv), and between input and relative graph position (hu and r̃u,v) respectively. Unlike Transformer-XL, we forgo the inductive bias built into the sinusoidal encoding, merging W̃k,Rru,v into a single learnable r̃u,v . This makes the relative positional embedding easily generalizable to atoms not within the same molecule, which is particular useful for reaction outcome prediction and, more generally, tasks with more than two input molecules. We also bucket the distances similar to Raffel et al. (2020) such that
Bu,v =  distance(u, v), if distance(u, v) < 8
8, if 8 ≤ distance(u, v) < 15 9, if 15 ≤ distance(u, v) and u, v are in the samemolecule 10, if u, v are not in the samemolecule
Bu,v is then used to look up r̃u,v in the trainable positional embedding matrix. The rest of the global attention encoder mostly follows a standard Transformer with multi-headed self-attention (Vaswani et al., 2017), layer normalization (Ba et al., 2016), and position-wise feed forward layers, except no positional information is added to the value vectors within each Transformer layer.

heading: 2.2.3 SEQUENCE DECODER

text: We use a Transformer-based autoregressive decoder to decode from the atom representations after the global attention encoder. Each output token is generated by attending to all atoms with encoderdecoder attention (Bahdanau et al., 2015; Vaswani et al., 2017), while also attending to all tokens that have already been generated. Following Seo et al. (2021), we set max relative positions to 4 for the decoder, thereby enabling the usage of sequence-based relative positional embedding used in Shaw et al. (2018) and implemented by OpenNMT (Klein et al., 2017).

heading: 2.3 MODEL TRAINING

text: The Graph2SMILES model is then trained to maximize the conditional likelihood p (Sout|Gin) = p (s1, s2, . . . , sn|Gin) = n∏ i=1 pθ (si|s1:i−1,Gin) (13)

heading: 3 RELATED WORK

heading: 3.1 REACTION OUTCOME PREDICTION AND ONE-STEP RETROSYNTHESIS

text:
One approach to computer-aided reaction prediction and retrosynthesis is to make use of chemical reaction rules based on subgraph pattern matching that are formalized as reaction templates, as in expert systems such as LHASA (Corey et al., 1972) and SYNTHIA (Szymkuć et al., 2016). More recent efforts have used neural networks to model the two tasks as template classification (Segler & Waller, 2017; Baylon et al., 2019; Dai et al., 2019a; Chen & Jung, 2021), or template ranking based on molecular similarity (Coley et al., 2017). These template-based approaches select the top ranked templates, which can then be applied to transform the input molecules into the outputs.
For template-based methods, there is an inevitable tradeoff between template generality and specificity. Further, these methods cannot generalize to unseen templates. As a remediation for such intrinsic limitations, a number of template-free approaches have emerged over the recent years, which can be categorized into graph edit-based and translation-based. The first category models reaction prediction or retrosynthesis as graph transformations (Jin et al., 2017; Coley et al., 2019; Do et al., 2019; Bradshaw et al., 2019; Sacha et al., 2021; Qian et al., 2020). Variants of graph edit methods include electron flow prediction (Bi et al., 2021) and semi template-based methods where reaction centers are first identified, followed by a graph or sequence recovery stage (Shi et al., 2020; Yan et al., 2020; Somnath et al., 2020; Wang et al., 2021b). Translation-based formulations, on the other hand, approach the problems as SMILES-to-SMILES translation, typically with sequence models such as Recurrent Neural Networks (Nam & Kim, 2016; Schwaller et al., 2018; Liu et al., 2017) or the Transformer (Schwaller et al., 2019; Lin et al., 2020; Lee et al., 2019; Duan et al., 2020; Tetko et al., 2020). Variants of these approaches design additional stages such as pretraining and reranking (Irwin et al., 2021; Zhu et al., 2021; Zheng et al., 2020; Sun et al., 2020), or use information about graph topology to enhance performance (Yoo et al., 2020; Seo et al., 2021; Mao et al., 2021).
Our approach is similar to GET (Mao et al., 2021) which also solves one-step retrosynthesis with graph-enhanced encoders and sequence decoders. Unlike GET which concatenates the SMILES sequence embeddings and learned atom representations, thereby not guaranteeing permutation invariance, we do not use the sequence representation at all in our encoder. Yet Graph2SMILES yields significant improvement over GET on USPTO_50k, demonstrating the power of our graph encoder and the adequacy of graph representations alone.

heading: 3.2 ADAPTING THE TRANSFORMER ENCODER FOR MOLECULAR REPRESENTATION

text:
The idea of modeling graphs using the Transformer architecture is not new. In the domain of molecular representation, the Molecular Attention Transformer (Maziarka et al., 2020) and GeoT (Kwak et al., 2021) inject atomic distance information into when computing attention scores. However, the computation of such distance information itself requires sampling of 3D conformers, thereby introducing an additional source of variations and breaking order invariance. An alternative is using the lengths of shortest paths between atoms. GRAT (Yoo et al., 2020) uses these lengths to parameterize the optional scale and bias for the attention score, whereas PAGTN (Chen et al., 2019)
treats them as path features in its additive attention. Our use of pairwise shortest path lengths between atoms for our Transformer-based global attention encoder are inspired by GRAT and PAGTN. Contrary to their usage of these lengths as additional features, we explicitly separate the effect of graph topology using graph-aware relative positional embedding, considering the success of such specially designed embedding in graph representation (Ying et al., 2021) and other domains (Shaw et al., 2018; Dai et al., 2019b; Wang et al., 2019; Guo et al., 2020). Our formulation of relative positional embedding builds on top of its counterpart in Transformer-XL (Dai et al., 2019b), which we found to be empirically superior than using single learnable bias as in T5 (Raffel et al., 2020) and Graphormer (Ying et al., 2021).

heading: 3.3 COMBINATION OF GRAPH NEURAL NETWORKS AND TRANSFORMER

text: Combining graph encoder and Transformer encoder in a sequential manner has been explored in GET (Mao et al., 2021) and NERF (Bi et al., 2021), as well as in Wang et al. (2021a) and GROVER (Rong et al., 2020) albeit for different molecular learning tasks. Graph2SMILES does not use the sequence representation as an input as in GET, or require the output molecular graph in the encoder as in NERF. Also, none of these related works retain the explicit information about graph topology before passing the atom representations into the attention encoder like we do, which we show to be important in the ablation study in Section 4.5. Similarly, the graph-to-sequence formulation itself has been used in NLP for conditional text generation tasks such as SQL-to-text (Xu et al., 2018a;b) and AMR-to-text (Cai & Lam, 2020). Our encoder is different from these prior studies; most notably, the graph-aware positional embedding is designed to easily generalize to more than two disconnected graphs, which is typical for reaction outcome prediction.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS

text: We evaluate model performance by top-n test accuracies on four USPTO datasets derived from reaction data originally curated by Lowe (2012). The details of these datasets are summarized in Appendix C. For reaction outcome prediction, we evaluate on the USPTO_480k_mixed and USPTO_STEREO_mixed datasets following Schwaller et al. (2019). The suffix _mixed indicates that the reactants and reagents have not been separated based on which species contribute heavy atoms to the product. While USPTO_480k has been preprocessed by Jin et al. (2017), USPTO_STEREO was filtered to a lesser extent, retaining stereochemical information and reactions forming or breaking aromatic bonds. For one-step retrosynthesis, we evaluate on the USPTO_full and USPTO_50k datasets without reaction type, both of which have been used as benchmarks for retrosynthesis. We count a prediction as correct only if it matches the ground truth output SMILES exactly, including all stereochemistry but excluding atom mapping, after canonicalization by RDKit.

heading: 4.2 IMPLEMENTATION DETAILS

text: For D-GAT and D-GCN, we change the hidden size to 256 from 300 used in GraphRetro (Somnath et al., 2020) to be more consistent with the global attention encoder. The number of message updating steps is set to 4. For the global attention encoder, following Molecular Transformer and GTA, we fix the embedding and hidden sizes dmodel to 256, the filter size for Transformer to 2048, the number of attention heads to 8, and the numbers of layers for both the attention encoder and the Transformer decoder to 6. We train our model using Adam optimizer (Kingma & Ba, 2015) with Noam learning rate scheduler (Vaswani et al., 2017). Similar to Schwaller et al. (2019), we group reactions with similar number of SMILES tokens together, batch by the maximal number of token count, and scale the D-MPNN outputs by √ dmodel before feeding into the attention encoder. The details of the hyperparameters used for different datasets are summarized in Appendix D. We save the model checkpoints every 5000 steps, select the best checkpoints based on the top-1 accuracy on the validation sets, and report the performance on the held-out test sets. Beam search is used to generate the output SMILES during inference with a beam size of 30. We filter out any SMILES that cannot be parsed by RDKit and keep the remaining as our final list of proposed candidates for evaluation.

heading: 4.3 RESULTS ON REACTION OUTCOME PREDICTION

text:
Table 1 summarizes the results of Graph2SMILES and other existing works on reaction outcome prediction. We only include methods that perform evaluations on the USPTO_480k_mixed dataset in the table, and provide a brief comparison of other methods that only evaluate on the less challenging USPTO_480k_separated data in Appendix B. All excluded methods show inferior performance to the Molecular Transformer (MT), with the exception of NERF (Bi et al., 2021) which has a 0.3 point improvement in top-1 accuracy.
For the USPTO_480k_mixed dataset, Graph2SMILES improves upon the MT baseline with 1.7, 0.5 and 0.4 point increases in top-1, 3 and 10 accuracies respectively. Similarly, for the USPTO_STEREO_mixed dataset, Graph2SMILES improves the top-1 accuracy of the MT baseline by 1.9 points, with minor improvement for top-3 accuracy. While there is still a gap between Graph2SMILES and Augmented Transformer or Chemformer, our approach does not use test-time data augmentation and ensembling as in Augmented Transformer (Tetko et al., 2020), nor have we performed pretraining as in Chemformer (Irwin et al., 2021) whose models have up to 10 times as many parameters as Graph2SMILES. Ensembling and pretraining are potential directions for improving Graph2SMILES as they are still compatible with our backbone replacement for the Transformer. For reaction outcome prediction, there is a small advantage of using D-GAT over D-GCN, with improvements of up to 0.2 points on top-n accuracy.

heading: 4.4 RESULTS ON ONE-STEP RETROSYNTHESIS

text:
We compare the results of one-step retrosynthesis on USPTO_full of Graph2SMILES with all existing methods that report results on this dataset, to the best of our knowledge. As can be seen from Table 2, Graph2SMILES achieves higher top-1 accuracy than all methods except GTA (Seo et al., 2021), while not using any templates, atom mapping, or output-side data augmentation. These additional features or techniques, which have been demonstrated to improve the performance of Transformer variants (Appendix E), are orthogonal to the graph-to-sequence architecture itself, and can therefore potentially improve Graph2SMILES as well. For example, atom mapping can be used to enhance Graph2SMILES with graph-truncated cross-attention as in GTA.
The much smaller USPTO_50k dataset has been benchmarked more extensively. We compare the results in Table 3, marking only the usage of the same set of features and techniques used as in Table 2 for succinctness1. The first group of rows shows that across methods that do not use reaction
1Graph2SMILES can also benefit from other techniques such as latent variable modeling (Chen et al., 2020), which we demonstrate in Appendix E. From Table 10, using latent classes N = 2 can already boost the top-10 accuracies of the D-GCN variant from 72.9 to 79.5, and of D-GAT from 73.9 to 77.7.
templates, atom mapping, or output SMILES augmentation, Graph2SMILES achieves the best top1 accuracy, improving the Transformer baseline (Lin et al., 2020) by 9.8 points from 43.1 to 52.9. From our experiments, we observe that it is possible to boost top-n accuracies for n > 1 at the expense of sacrificing the top-1 accuracy (even slightly), creating some room for tradeoff. To avoid over-tuning and giving overly optimistic results, however, we only report the test results for models with the highest top-1 accuracy during validation. Unlike in reaction outcome prediction, using DGAT does not have a clear advantage over D-GCN for retrosynthesis, yielding only improvement on top-5 and 10 accuracies for USPTO_50k as in Table 3.
The second group of rows in Table 3 includes methods that use additional features or techniques. Graph2SMILES beats a number of methods in this group with a top-1 accuracy of 52.9 without using any of templates, atom mapping or output SMILES augmentation. EBM (Dual-TB) (Sun
et al., 2020) achieves the SOTA for top-1 accuracy, and LocalRetro (Chen & Jung, 2021) for top-3, 5 and 10 accuracies respectively. Both make use of templates and atom mapping, which seem to be empirically helpful for this small dataset. Similar to how Graph2SMILES can benefit from using atom mapping as discussed earlier, we can make use of templates for potential gain, e.g. by retaining only candidates with reaction templates that have been seen in the training set.

heading: 4.5 ABLATION STUDY

text: We perform an ablation study to explore the effects of various components of Graph2SMILES by removing the D-MPNN, positional embedding, or the global attention encoder. We summarize the results on USPTO_50k, for which the quantitative effect is most conspicuous. From the results in Table 4, the removal of D-MPNN decreases the top-1 accuracy to 49.9. Note that this setup is architecturally similar to the Transformer baseline, but uses atom features rather than SMILES sequence as inputs. Getting rid of the graph-aware positional embedding leads to a drop in top1 accuracy of 2.1 points for D-GCN and 0.6 points for D-GAT. The effect of removing the global attention encoder is more significant, decreasing the top-1 accuracy by up to 8.3 points. We therefore conclude that all three components of our encoder in Graph2SMILES, namely the D-MPNN, the graph-aware positional embedding, and the global attention encoder, are important. They work collectively to capture a meaningful representation of the input molecular graphs.

heading: 5 DISCUSSION

text: Throughout our experiments, to demonstrate the advantage over a vanilla Transformer, we have focused on the baseline Graph2SMILES model, forgoing the benefits of using additional features and techniques for performance engineering. Although we have used the top-1 accuracy as a basis for comparison throughout our discussion, we recognize its limitations especially for retrosynthesis, which can have many equally plausible options. Similarly, the datasets we use for reaction outcome prediction are not perfectly detailed, with legitimate ambiguity in the identity of the major product. While Graph2SMILES shows strong performance for top-1 accuracy and can replace Transformer model with minimal modification to the pipeline, there could be cases when top-n accuracy is more relevant (e.g. in multi-step planning applications). In those cases, the aforementioned performance engineering techniques would be necessary to boost the top-n performance of Graph2SMILES.

heading: 6 CONCLUSION

text:
In this paper, we present a novel Graph2SMILES model for template-free reaction outcome prediction and retrosynthesis. The permutation invariance of its D-MPNN and graph-aware positional embedding eliminates the need for any input-side SMILES augmentation, while achieving noticeable improvement over the Transformer baselines, especially for top-1 accuracy. Graph2SMILES is therefore an attractive drop-in replacement for any methods that use the Transformer model for molecular transformation tasks. Further gain may be possible through performance engineering tricks that are orthogonal to the architecture itself, which will be investigated in future work.
REPRODUCIBILITY STATEMENT
We hereby declare that all of our reported results are reproducible from our existing code base, subject to minor deviations due to hardware-level numerical uncertainties as we have observed for some GPU models (e.g. V100). We include part of our code to reproduce some specific results, with a self-explanatory README file in the supplementary materials. We will open-source all scripts to reproduce any result in the manuscript along with pretrained Graph2SMILES checkpoints after the review period, regardless of whether the manuscript is accepted.

heading: A APPENDIX: ATOM AND BOND FEATURES USED

text: Table 5 summarizes the atom and bond features used in Graph2SMILES. Most features were adapted from GraphRetro (Somnath et al., 2020), with the addition of chiral features (R/S and E/Z).

heading: B APPENDIX: OTHER METHODS FOR REACTION OUTCOME PREDICTION

text: Table 6 summarizes the results for methods excluded in Section 4.3 for reaction outcome prediction, most of which report the values on the less challenging USPTO_480k_separated dataset, in which the reagents have been heuristically separated from the reactants. Only NERF shows marginal improvement of 0.3 points for top-1 accuracy over Molecular Transformer, whereas all other methods cannot perform as well. We therefore use Molecular Transformer as our baseline in Section 4.3. Note that ELECTRO (Bradshaw et al., 2019) tests on a simpler subset of reactions with linear electron flow (LEF), and we therefore exclude it from the quantitative comparison.

heading: D APPENDIX: HYPERPARAMETER SETTING

heading: C APPENDIX: SUMMARY OF FOUR USPTO DATASETS USED

heading: E APPENDIX: EFFECTIVENESS OF ADDITIONAL FEATURES OR TECHNIQUES

text:
For all groups of rows in Table 9, the first row of numbers are for the Transformer variants where features or techniques of interest are not used, and the rest of the rows otherwise. In turn, these results illustrate the effectiveness of using templates (EBM), atom-mapping (GTA), pretraining (DMP), output-side augmentation (Augmented Transformer), syntax-based reranking (SCROP) and variational inference (Latent Transformer). Inclusion of these features or techniques clearly improves the accuracies across the board over respective baselines, as highlighted in bold.
As an illustration of how Graph2SMILES can benefit from additional features or techniques, we briefly experiment with one of them, latent variable modeling similar to Chen et al. (2020) and Kim et al. (2021), using N = 2 latent classes with a uniform prior. As in Table 10, this noticeably increases the top-n accuracies for both D-GCN and D-GAT, which can be particularly relevant for multi-step planning.

heading: F APPENDIX: VISUALIZATION OF CASES WHERE GRAPH2SMILES OUTPERFORMS THE TRANSFORMER BASELINE

text:
We include some real cases for which Graph2SMILES gives the correct top-1 prediction but the Molecular Transformer baseline cannot, for the reaction outcome prediction task on USPTO_480k_mixed. For each row in Figure 2, the reactants are shown on the left, the ground truth product (which is also the correct prediction by Graph2SMILES) is shown in the center, and the erroneous prediction by the Molecular Transformer baseline is shown on the right.
The first two rows represent two intramolecular ring forming reactions after an N-alkylation or amidation. As shown by the right-most molecules, without explicit knowledge of the reactant graphs, Molecular Transformer instead predicts substitution reactions at the benzylic carbon and at the phenyl carbon respectively. Both predictions seem to be based solely on local chemical environments, with the first being plausible as nucleophilic substitution on benzyl bromide, but the second nonsensical since such electrophilic aromatic substitution would not happen with the nucleophilic phenylamine. Graph2SMILES, on the other hand, correctly predicts the ring formation. We therefore hypothesize that graph representation may be more powerful at encoding beyond local contexts via its global attention mechanism, which is potentially important for such reactions. Further quantitative evaluation is left as future work, since explainability is not a trivial task for template-free models.
The third row represents the removal of the amine group attached to the benzene ring, for which Graph2SMILES gives the correct prediction, but the Molecular Transformer predicts the formation of an aryl hydrazine, possibly by detecting the extra nitrogen in the input SMILES (from DMF) but failing to realize it as merely a spectator solvent without full recognition of its graph structure. The last row is a case where Graph2SMILES predicts the correct regioselectivity (ortho vs. para to the nitro group) for the aromatic substitution reaction. This may be interpreted as better capability of Graph2SMILES to learn longer range topological information as compared to the Transformer baseline.