Deformable Siamese Attention Networks for Visual Object Tracking

Siamese-based trackers have achieved excellent performance on visual object tracking. However, the target template is not updated online, and the features of the target template and search image are computed independently in a Siamese architecture. In this paper, we propose Deformable Siamese Attention Networks, referred to as SiamAttn, by introducing a new Siamese attention mechanism that computes deformable self-attention and cross-attention. The self attention learns strong context information via spatial attention, and selectively emphasizes interdependent channel-wise features with channel attention. The cross-attention is capable of aggregating rich contextual inter-dependencies between the target template and the search image, providing an implicit manner to adaptively update the target template. In addition, we design a region refinement module that computes depth-wise cross correlations between the attentional features for more accurate tracking. We conduct experiments on six benchmarks, where our method achieves new state of-the-art results, outperforming the strong baseline, SiamRPN++ [24], by 0.464->0.537 and 0.415->0.470 EAO on VOT 2016 and 2018.

Introduction
Visual object tracking aims to track a given target object at each frame over a video sequence. It is a fundamental task in computer vision [17,16,20], and has numerous practical applications, such as automatic driving [23], human-computer interaction [28], robot sensing, etc. Recent efforts have been devoted to improving the performance of visual object trackers. However, developing a fast, accurate and robust tracker is still highly challenging due to the vast amount of deformations, motions and occlusions that often occur on video objects with complex background [38,22,10].
Deep learning technologies have significantly advanced the task of visual object tracking, by providing the strong Figure 1: Tracking results of our deformable Siamese attention networks (SiamAttn) with three state-of-the-art trackers. Our results are more accurate, and are robust to appearance changes, complex background and close distractors with occlusions. Fig. 4 further shows the strong discriminative features learned by our Siamese attention module. capacity of learning powerful deep features. For example, Bertinetto et al. [1] first introduced Siamese networks for visual tracking. Since then, object trackers built on Siamese networks and object detection frameworks have achieved the state-of-the-art performance, such as SiamRPN [25], SiamRPN++ [24], and SiamMask [36]. The Siamese-based trackers formulate the problem of visual object tracking as a matching problem by computing cross-correlation similarities between a target template and a search region, which transforms the tracking problem into finding the target object from an image region by computing the highest visual similarity [1,25,24,36,44]. Therefore, it casts the tracking problem into a Region Proposal Network (RPN) [13] based detection framework by leveraging Siamese networks, which is the key to boost the performance of recent deep trackers.
Siamese-based trackers are trained completely offline by using massive frame pairs collected from videos, and thus the target template can not be updated online. This makes it difficult to precisely track the targets with large appearance variations, significant deformations, or occlusions, which inevitably increase the risk of tracking drift. Furthermore, the convolutional features of the target object and the search image are computed independently in Siamese architecture, where background context information is completely discarded in target features, but is of great importance to distinguish the target from close distractors and complex backgrounds. Recent work attempted to enhance the target representation by integrating the features of previous targets [41,14], but the discriminative context information from the background is ignored. Alternatively, we introduce a new Siamese attention mechanism that encodes rich background context into the target representation by computing cross-attention in the Siamese networks.
Recently, the attention mechanism was introduced to visual object tracking in [35,45], which inspired the current work. However, the attentions and deep features of the target template and the search image are computed separately in [35,45], which limits the potential performance of the Siamese architecture. In this work, we propose Deformable Siamese Attention Networks, referred as SiamAttn, to improve the feature learning capability of Siamese-based trackers. We present a new deformable Siamese attention which can improve the target representation with strong robustness to large appearance variations, and also enhance the target discriminability against distractors and complex backgrounds, resulting in more accurate and stable tracking, as shown in Fig. 1

Related Work
Correlation filter based trackers have been widely used since MOSSE [3], due to their efficiency and expansibility. However, the tracking object can be continuously improved online, which inevitably limits the representation ability of such trackers. Deep learning technologies provide a powerful tool with the ability to learn strong deep representations, and recent work attempted to incorporate the correlation filter framework with such features learning capability, such as MDNet [31], C-COT [8], ECO [7] and GFS-DCF [40].
There is another trend to build trackers on Siamese networks, by learning from massive data offline. Bertinetto et al. [1] first introduced SiamFC for visual tracking, by using Siamese networks to measure the similarity between target and search image. Then Li et al. [25] applied a region proposal network (RPN) [13] into Siamese networks, referred as SiamRPN. Zhu et al. [44] extended the SiamRPN by developing distractor-aware training. Recently, SiamDW-RPN [43] and SiamRPN++ [24] were proposed, which allow the Siamese-based trackers to explore deeper networks, while Wang et al. [36] developed a SiamMask that incorporates instance segmentation into tracking. Our work is related to that of [11] where a C-RPN was developed to progressively refine the location of target with a sequence of RPNs, but we design a new module that only refines a single output region, which is particularly lightweight and can be integrated into very deep Siamese networks.
However, Siamese-based trackers can be affected by distractors with complex backgrounds. Recent work attempted to design various strategies to update the template online, in an effort to improve the target-discriminability of Siamesebased trackers, such as MLT [4], UpdateNet [42] and Grad-Net [26]. An alternative solution is to extend existing online discriminative framework with deep networks for endto-end learning, e.g., ATOM [6] and DiMP [2]. In addition, Zhu et al. [45] exploited motion information in Siamese networks to improve the feature representation.
Recently, the attention mechanism has been widely applied in various tasks. Hu et al. [18] proposed a SENet to enhance the representational power of the networks by modeling channel-wise relationships via attentions. Wang et al. [37] developed a non-local operation in the space-time dimension to guide the aggregation of contextual information. In [12], a self-attention mechanism was introduced to harvest the contextual information for semantic segmentation. Particularly, Wang et al. [35] proposed a RASNet by developing an attention mechanism for Siamese trackers, but it only utilizes the template information, which might limit its representation ability. To better explore the potentials of feature attentions in Siamese networks, we compute both self-attention and cross-branch attention jointly with deformable operations to enhance the discriminative representation of target.

Deformable Siamese Attention Networks
We describe the details of our Deformable Siamese Attention Networks (SiamAttn). As shown in Fig. 2, it consists of three main components: a deformable Siamese attention (DSA) module, Siamese region proposal networks (Siamese RPN), and a region refinement module.
Overview. We use a five-stage ResNet-50 as the backbone of Siamese networks, which computes increasingly highlevel features as the layers become deeper. The features of the last three stages on both Siamese branches can be modulated and enhanced by the proposed DSA module, generating two-stream attentional features. Then we apply three Siamese RPN blocks described in [24] to the attentional features, generating dense response maps, which are further processed by a classification head and a bounding box regression head to predict a single tracking region. Finally, the generated tracking region is further refined by a region refinement module, where depth-wise cross correlations are computed on the two-stream attentional features. The correlated features are further fused and enhanced, and then are used for refining the tracking region via joint bounding box regression and target mask prediction.

Siamese-based Trackers
Bertinetto et al. [1] introduced Siamese networks for visual object tracking, which formulates visual object tracking as a similarity learning problem. Siamese networks consist of a pair of CNN branches with sharing parameters φ, which are used to project the target image (z) and the search image (x) into a common embedding space, where a similarity metric g can be computed to measure the similarity between them, g(φ(x), φ(z)). Li et al. [25] applied region proposal networks (RPN) [13] with Siamese networks for visual object tracking (referred as SiamRPN), where the computed features φ(x) and φ(z) are fed into the RPN framework using an up-channel cross-correlation operation. This generates dense response maps where RPN-based detection can be implemented, leading to significant performance improvements.
SiamRPN++. In [24], SiamRPN++ was introduced to improve the performance of SiamRPN, by exploring the power of deeper networks. A spatial aware sampling strategy was developed to address a key limitation of the Siamese-based trackers, allowing them to benefit from a deeper backbone likes ResNet-50. Furthermore, SiamRPN++ adopts a depthwise cross correlation to replace the up-channel cross correlation, which reduces the number of parameters and accelerates the training process. Moreover, it aggregates multilayer features to predict the target more accurately. Similarly, we use ResNet-50 as backbone, with depth-wise cross correlation and multi-layer aggregation strategy, by following SiamRPN++ [24]. But we introduce a new Siamese attention module that enhances the learned discriminative representations of the target object and the search image, which is the key to improve the tracking performance on both accuracy and robustness.

Deformable Siamese Attention Module
As illustrated in Fig. 3, the proposed DSA module takes a pair of convolutional features computed from Siamese networks as inputs, and outputs the modulated features by applying the Siamese attention mechanism. The DSA module consists of two sub-modules: a self-attention sub-module and a cross-attention sub-module. We denote the feature maps of the target and the search image as Z and X, with feature shapes of C × h × w and C × H × W . Self-Attention. Inspired by [12], our self-attention submodule attends to two aspects, namely channels and special positions. Unlike the classification or detection task where object classes are pre-defined, visual object tracking is a class-agnostic task and the class of object is fixed during the whole tracking process. As observed in [24], each channel map of the high-level convolutional features usually responses for a specific object class. Equally treating the features across all channels will hinder the representation ability. Similarly, as limited by receptive fields, the features computed at each spatial position of the maps can only capture the information from a local patch. Therefore, it is crucial to learn the global context from the whole image.
Specifically, self-attention is computed separately on the target branch and the search branch, and both channel selfattention and spatial self-attention are calculated at each branch. Taking the spatial self-attention for example. Sup-pose the input features are X ∈ R C×H×W , we first apply two separate convolution layers with 1 × 1 kernels on X to generate query features Q and key features K respectively, where Q, K ∈ R C ×H×W and C = 1 8 C is the reduced channel number. The two features are then reshaped toQ,K ∈ R C ×N where N = H × W . We can generate a spatial self-attention map A s s ∈ R N ×N via matrix multiplication and column-wise softmax operations as, Meanwhile, a 1×1 convolution layer with a reshape operation is applied to the features X to generate value features V ∈ R C×N , which are multiplied with the attention map and then are added to the reshaped featuresX ∈ R C×N with a residual connection as, where α is a scalar parameter. The outputs are then reshaped back to the original size as X s s ∈ R C×H×W . We can compute channel self-attention A s c and the channel-wise attentional features X s c in a similar manner. Notice that on computing the channel self-attention and the corresponding attentional features, the query, key and value features are the original convolutional features computed directly from the Siamese networks, without implementing 1 × 1 convolutions. The final self-attentional features X s are generated by simply combining the spatial and channel-wise attentional features using element-wise sum.
Cross-Attention. Siamese networks usually make predictions in the last stage, while the features from two branches are computed separately, but may compensate each other. It is common that multiple objects appear at the same time, even with occlusions during tracking. Therefore, it is of great importance for the search branch to learn the target information, which enables it to generate a more discriminative representation that helps identify the target more accurately. Meanwhile, the target representation can be more meaningful when the contextual information from the search image is encoded. To this end, we propose a cross-attention sub-module to learn such mutual information from two Siamese branches, which in turn allows the two branches to work more collaboratively.
Specifically, we use Z ∈ R C×h×w and X ∈ R C×H×W to denote template features and search features, respectively. Taking the search branch for example, we first reshape the target features Z toZ ∈ R C×n where n = h × w. Then we compute the cross-attention from the target branch by performing similar operations as channel self-attention, where row-wise softmax is implemented on the computed matrix. Then the cross-attention computed from the target branch is encoded into the search features X as, where γ is a scalar parameter, and the reshaped features X c ∈ R C×H×W are the output of the sub-module. Finally, the self-attentional features X s and the crossattentional features X c are simply combined with an element-wise sum, generating the attentional features for the search image. The attentional features for target image can be computed in a similar manner.
Deformable Attention. The building units in CNNs, such as convolution or pooling units, often have fixed geometric structures, by assuming the objects are rigid. For object tracking, it is of importance to model complex geometric transformations because the tracking objects usually have large deformations due to various factors, such as viewpoint, pose, occlusion and so on. The proposed attention mechanism can handle such challenges to some extent. We further introduce deformable attention to enhance the capability for handling such geometric transformations.
The deformable attention can sample the input feature maps at variable locations instead of the fixed ones, making them attend to the content of objects with deformations. Therefore, it is particularly suitable for object tracking, where the visual appearance of a target can be changed significantly over time. Specifically, a 3 × 3 deformable convolution [5] is further applied to the computed attentional features, generating the final attentional features which are more accurate, discriminative and robust. As shown in Fig. 4, with our DSA module, the confidence maps of the attentional features focus more accurately on the interested objects, making the objects more discriminative against distractors and background.
Region Proposals. The DSA module outputs Siamese attentional features for both target image and search image. Then we apply three Siamese RPN blocks on the attentional features for generating a set of target proposals, with corresponding bounding boxes and class scores, as shown in Fig. 2. Specifically, a Siamese RPN block is a combination of multiple fully convolutional layers, depth-wise cross correlation, with a regression head and a classification head on top, as described in [25]. It takes a pair of convolutional features computed from the two branches of Siamese networks, and outputs dense prediction maps. By following [24], we apply three Siamese RPN blocks for the Siamese features computed from the last three stages, generating three prediction maps which are further combined by a weighted sum. Each spatial position of the combined maps predicts a set of region proposals, corresponding to the pre-defined anchors. Then the predicted proposal with the highest classification score is selected as the output tracking region.

Region Refinement Module
We further develop a region refinement module to improve the localization accuracy of the predicted target region. We first apply depth-wise cross correlations between the two attentional features across multiple stages, generating multiple correlation maps. Then the correlation maps are fed into a fusion block, where the feature maps with different sizes are aligned in both spatial and channel domains, e.g., by using up-sampling or down-sampling, with 1 × 1 convolution. Then the aligned features are further fused (with element-wise sum) for predicting both bounding box and mask of the target. Besides, we further perform two additional operations: (1) we combine the convolutional features of the first two-stages into the fused features, which encodes richer local detailed information for mask prediction; (2) a deformable RoI pooling [5] is applied to compute target features more accurately. Bounding box regression and mask prediction often require different levels of convolutional features. Thus we generate the convolutional features with spatial resolutions of 64 × 64 for mask prediction and 25 × 25 for bounding box regression.
Notice that the classification head is not applied since visual object tracking is a class-agnostic task. The input resolution for the bounding box head is 4 × 4. By using two fully-connected layers with 512 dimensions, the bounding box head predicts a 4-tuple t = (t x , t y , t w , t h ). Similarly, the input of mask prediction head has a spatial resolution of 16 × 16. By using four convolutional layers and a de-convolutional layer, the mask head predicts a classagnostic binary mask with 64 × 64 for the tracking object. Compared to ATOM [6] and SiamMask [36] which predict bounding boxes and masks densely, our refinement module uses light-weight convolutional heads to predict a bounding box and a mask for a single tracking region, which is much more computationally efficient.

Training Loss
Our model is trained in an end-to-end fashion, where the training loss is a weighted combination of multiple functions from Siamese RPN and region refinement module: where L rpn-cls and L rpn-reg refer to a classification loss and a regression loss in Siamese RPN. We employ a negative log-likelihood loss and a smooth L1 loss correspondingly. Similarly, L refine-box and L refine-mask indicate a smooth L1 loss for bounding box regression and a binary cross-entropy loss for mask segmentation in region refinement module. The weight parameters λ 1 , λ 2 and λ 3 are used to balance different tasks, which are empirically set to 0.2, 0.2 and 0.1 in our implementation.

Datasets
OTB-2015 [38]. OTB-2015 is one of the most commonly used benchmarks for visual object tracking. It has 100 fully annotated video sequences using two evaluation metrics, a precision score and an area under curve (AUC) of success plot. The precision score is the percentage of frames in which the distance between the center of tracking results and ground-truth is under 20 pixels. The success plot shows the ratios of successfully tracked frames with various thresholds. VOT2016 [21] & VOT2018 [22]. VOT2016 and VOT2018 are widely-used benchmarks for visual object tracking. VOT2016 contains 60 sequences with various challenging factors while VOT2018 has 10 different sequences with VOT2016. The two datasets are annotated with the rotated bounding boxes, and a reset-based methodology is applied for evaluation. For both benchmarks, trackers are measured in terms of accuracy (A), robustness (R), and expected average overlap (EAO). UAV123 [29]. UAV123 contains 123 sequences captured from low-altitude UAVs. Unlike other tracking datasets, the viewpoint of UAV123 is aerial and the targets to be tracked are usually small. LaSOT [10]. LaSOT is a large-scale dataset with 1400 sequences in total, and 280 sequences in test set. high-quality dense annotations are provided, and deformation and occlusion are very common in LaSOT. The average sequence length of LaSOT is 2500 frames, demonstrating long-term performance of the evaluated trackers. TrackingNet [30]. TrackingNet contains 30000 sequences with 14 million dense annotations and 511 sequences in the test set. It covers diverse object classes and scenes, requiring trackers to have both discriminative and generative capabilities.

Implementation Details
We use ResNet-50, pre-trained on ImageNet [9], as the backbone, and the whole networks are then fine-tuned on the training sets of COCO [27], YouTube-VOS [39], La-SOT [10] and TrackingNet [30]. We apply stochastic gradient descent (SGD) with a momentum of 0.9 and a weight decay of 10 −5 for optimization. By following SiamFC [1], we adopt an exemplar image of 127 × 127 and a search image of 255 × 255 for training and testing.
Our model is trained for 20 epochs. By following SiamRPN++ [24], we use a warm-up learning rate of 10 −3     for the first 5 epochs which decays exponentially from 5 × 10 −3 to 5 × 10 −4 for the last 15 epochs. The weights of backbone are frozen for the first 10 epochs, then the whole networks are trained end-to-end for the last 10 epochs. In particular, the learning rate of backbone is 20 times smaller than the other parts. Batch size is set to 12. Our anchor boxes have 5 aspect ratios, [0.33, 0.5, 1, 2, 3]. In Siamese-RPN blocks, an anchor box is labelled as positive when it has an IoU> 0.6, or as negative when IoU< 0.3. Other patches whose IoU overlap falls in between are ignored.
In addition, we sample 16 regions from each image with IoU> 0.5 for training our region refinement module. For the backbone networks, we employ dilated convolutions for the last two blocks to increase the receptive fields, and the effective strides at these two blocks are reduced from 16 or 32 pixels to 8 pixels. We also reduce the number of feature channels to 256 for the last three blocks of the backbone networks via a 1 × 1 convolution layer. During inference, cosine window penalty, scale change penalty and linear interpolation update strategy [25] are applied. Only one single region with the highest score predicted by Siamese RPN blocks is fed into our region refinement module. Our method is implemented using PyTorch, and we use NVIDIA GeForce RTX 2080Ti GPU.

Ablation Study
We study the impact of individual components in SiamAttn, and conduct ablation study on VOT2016.

Model Architecture.
We use SiamRPN++ [24] as baseline. As shown in Tab. 5, SiamRPN++ achieves an EAO of 0.464. By adding mask learning layers to SiamRPN++, the EAO can be improved to 0.477. With our region refinement module, the EAO score is further increased by +2.2%. Compared with the baseline, the accuracy score improves from 0.64 to 0.67, with comparable robustness. Our Siamese attention consists of both self-attention and cross-attention, each of which can further improve the EAO by +4.7% or +4.9% respectively. This suggests that the proposed cross-attention is critical to the tracking results, and even has a more significant impact than the self-attention. Jointly exploring both selfattention and cross-attention makes our method not only robust, but also more accurate. This results in a high EAO of 0.537, surpassing the baseline by a large margin of 7.3%.

Deformable Layers.
In this study, we evaluate the impact of deformable operations by replacing them with a regular convolution. As shown in Tab. 6, this results in slight performance drops, e.g., 0.537→0.520 EAO with deformable convolution and 0.537→0.531 EAO with   On Training Data. In this study, we investigate the impact of training with different training sets. Our current results are achieved by using a combination of multiple training sets from recent LaSOT [10] and TrackingNet [30], with COCO [27] and YouTube-VOS [39], mainly following [6] with an additional YouTube-VOS [39] for providing mask annotations. We also report the results on a different training combination applied by [36], including COCO [27], YouTube-VOS [39], YouTube-BoundingBox [32], ImageNet-VID, and ImageNet-Det [33]. Results are reported in Tab. 7. Using the recent large-scale tracking datasets can improve the results with 1.2% EAO on VOT2016, while our approach can still achieve the state-of-the-art performance using a different choice of the training sets.
Speed Analysis. On OTB-2015, UAV, LaSOT and Track-ingNet benchmarks, our model predicts axis-aligned bounding box, without mask head. It can achieve an inference speed of 45 fps. On VOT benchmarks, our model generates the rotated boxes from the predicted masks, which reduces the inference speed to 33 fps.

Conclusion
We have presented new Deformable Siamese Attention Networks for visual object tracking. We introduce a deformable Siamese attention mechanism consisting of both self-attention and cross-attention. The new Siamese attention can strongly enhance target discriminability, and at the same time, improve the robustness against large appearance variations, complex backgrounds and distractors. Additionally, a region refinement module is designed to further increase the tracking accuracy. Extensive experiments are conducted on six benchmarks, where our method obtains new state-of-the-art results, with real-time running speed.


extfieldsofstudy:
Computer Science
Mathematics

text:
Graph Structure Learning for Robust Graph Neural Networks

Graph Neural Networks (GNNs) are powerful tools in representation learning for graphs. However, recent studies show that GNNs are vulnerable to carefully-crafted perturbations, called adversarial attacks. Adversarial attacks can easily fool GNNs in making predictions for downstream tasks. The vulnerability to adversarial attacks has raised increasing concerns for applying GNNs in safety-critical applications. Therefore, developing robust algorithms to defend adversarial attacks is of great significance. A natural idea to defend adversarial attacks is to clean the perturbed graph. It is evident that real-world graphs share some intrinsic properties. For example, many real-world graphs are low-rank and sparse, and the features of two adjacent nodes tend to be similar. In fact, we find that adversarial attacks are likely to violate these graph properties. Therefore, in this paper, we explore these properties to defend adversarial attacks on graphs. In particular, we propose a general framework Pro-GNN, which can jointly learn a structural graph and a robust graph neural network model from the perturbed graph guided by these properties. Extensive experiments on real-world graphs demonstrate that the proposed framework achieves significantly better performance compared with the state-of-the-art defense methods, even when the graph is heavily perturbed. We release the implementation of Pro-GNN to our DeepRobust repository for adversarial attacks and defenses (footnote: https://github.com/DSE-MSU/DeepRobust). The specific experimental settings to reproduce our results can be found in https://github.com/ChandlerBang/Pro-GNN.

(the Facebook friend network). With their prevalence, it is particularly important to learn effective representations of graphs and then apply them to solve downstream tasks. Recent years have witnessed great success from Graph Neural Networks (GNNs) [15,17,21,29] in representation learning of graphs. GNNs follow a message-passing scheme [14], where the node embedding is obtained by aggregating and transforming the embeddings of its neighbors. Due to the good performance, GNNs have been applied to various analytical tasks including node classification [17], link prediction [18], and recommender systems [34].
Although promising results have been achieved, recent studies have shown that GNNs are vulnerable to adversarial attacks [10,16,31,38,39]. In other words, the performance of GNNs can greatly degrade under an unnoticeable perturbation in graphs. The lack of robustness of these models can lead to severe consequences for critical applications pertaining to the safety and privacy. For example, in credit card fraud detection, fraudsters can create several transactions with only a few high-credit users to disguise themselves, thus escaping from the detection based on GNNs. Hence, developing robust GNN models to resist adversarial attacks is of significant importance. Modifying graph data can perturb either node features or graph structures. However, given the complexity of structural information, the majority of existing adversarial attacks on graph data have focused on modifying graph structure especially adding/deleting/rewiring edges [33]. Thus, in this work, we aim to defend against the most common setting of adversarial attacks on graph data, i.e., poisoning adversarial attacks on graph structure. Under this setting, the graph structure has already been perturbed by modifying edges before training GNNs while node features are not changed.
One perspective to design an effective defense algorithm is to clean the perturbed graph such as removing the adversarial edges and restoring the deleted edges [27,37]. The key challenge from this perspective is what criteria we should follow to clean the perturbed graph. It is well known that real-world graphs often share certain properties. First, many real-world clean graphs are low-rank and sparse [36]. For instance, in a social network, most individuals are connected with only a small number of neighbors and there are only a few factors influencing the connections among users [13,36]. Second, connected nodes in a clean graph are likely to share similar features or attributes (or feature smoothness) [24]. For example, in a citation network, two connected publications often share similar topics [17]. Figure 1 demonstrates these properties of clean and poisoned graphs. Specifically, we apply the state-of-the-art graph poisoning attack, metattack [39], to perturb the graph data and visualize the graph properties before and after mettack. As shown in Figure 1a, metattack enlarges the singular values of the adjacency matrix and Figure 1b illustrates that metattack quickly increases the rank of adjacency matrix. Moreover, when we remove the adversarial and normal edges from the perturbed graph respectively, we observe that removing adversarial edges reduces the rank faster than removing normal edges as demonstrated in Figure 1c. In addition, we depict the density distribution of feature difference of connected nodes of the attacked graph in Figure 1d. It is observed that metattack tends to connect nodes with large feature difference. Observations from Figure 1 indicate that adversarial attacks could violate these properties. Thus, these properties have the potential to serve as the guidance to clean the perturbed graph. However, work of exploring these properties to build robust graph neural networks is rather limited.
In this paper, we target on exploring graph properties of sparsity, low rank and feature smoothness to design robust graph neural networks. Note that there could be more properties to be explored and we would like to leave it as future work. In essence, we are faced with two challenges: (i) how to learn clean graph structure from poisoned graph data guided by these properties; and (ii) how to jointly learn parameters for robust graph neural network and the clean structure. To solve these two challenges, we propose a general framework Property GNN (Pro-GNN) to simultaneously learn the clean graph structure from perturbed graph and GNN parameters to defend against adversarial attacks. Extensive experiments on a variety of real-world graphs demonstrate that our proposed model can effectively defend against different types of adversarial attacks and outperforms the state-of-the-art defense methods.
The rest of the paper is organized as follows. In Section 2, we review some of the related work. In Section 3, we introduce notations and formally define the problem. We explain our proposed framework in Section 4 and report our experimental results in Section 5. Finally, we conclude the work with future directions in Section 6.

RELATED WORK
In line with the focus of our work, we briefly describe related work on GNNs, and adversarial attacks and defense for graph data.

Graph Neural Networks
Over the past few years, graph neural networks have achieved great success in solving machine learning problems on graph data. To learn effective representation of graph data, two main families of GNNs have been proposed, i.e., spectral methods and spatial methods. The first family learns node representation based on graph spectral theory [6,11,17]. Bruna et al. [6] generalize the convolution operation from Euclidean data to non-Euclidean data by using the Fourier basis of a given graph. To simplify spectral GNNs, Defferrard et al. [11] propose ChebNet and utilize Chebyshev polynomials as the convolution filter. Kipf et al. [17] propose GCN and simplify ChebNet by using its first-order approximation. Further, Simple Graph Convolution (SGC) [30] reduces the graph convolution to a linear model but still achieves competitive performance. The second family of models define graph convolutions in the spatial domain as aggregating and transforming local information [14,15,29]. For instance, DCNN [2] treats graph convolutions as a diffusion process and assigns a certain transition probability for information transferred from one node to the adjacent node. Hamilton et al. [15] propose to learn aggregators by sampling and aggregating neighbor information. Veličković et al. [29] propose graph attention network (GAT) to learn different attention scores for neighbors when aggregating information. To further improve the training efficiency, FastGCN [8] interprets graph convolutions as integral transforms of embedding functions under probability measures and performs importance sampling to sample a fixed number of nodes for each layer. For a thorough review, we please refer the reader to recent surveys [3,32,35].

Adversarial Attacks and Defense for GNNs
Extensive studies have demonstrated that deep learning models are vulnerable to adversarial attacks. In other words, slight or unnoticeable perturbations to the input can fool a neural network to output a wrong prediction. GNNs also suffer this problem [5,10,16,22,23,31,38,39]. Different from image data, the graph structure is discrete and the nodes are dependent of each other, thus making it far more challenging. The nettack [38] generates unnoticeable perturbations by preserving degree distribution and imposing constraints on feature co-occurrence. RL-S2V [10] employs reinforcement learning to generate adversarial attacks. However, both of the two methods are designed for targeted attack and can only degrade the performance of GNN on target nodes. To perturb the graph globally, metattack [39] is proposed to generate poisoning attacks based on meta-learning. Although increasing efforts have been devoted to developing adversarial attacks on graph data, the research about improving the robustness of GNNs has just started recently [27,31,37,40]. One way to solve the problem is to learn a robust network by penalizing the attention scores of adversarial edges. RGCN [37] is to model Gaussian distributions as hidden layers to absorb the effects of adversarial attacks in the variances. PA-GNN [27] leverages supervision knowledge from clean graphs and applies a meta-optimization way to learn attention scores for robust graph neural networks. However, it requires additional graph data from similar domain. The other way is to preprocess the perturbed graphs to get clean graphs and train GNNs on the clean ones. Wu et. al [31] have found that attackers tend to connect to nodes with different features and they propose to remove the links between dissimilar nodes. Entezari et al. [12] have observed that nettack results in changes in high-rank spectrum of the graph and propose to preprocess the graph with its low-rank approximations. However, due to the simplicity of two-stage preprocessing methods, they may fail to counteract complex global attacks.
Different from the aforementioned defense methods, we aim to explore important graph properties to recover the clean graph while learning the GNN parameters simultaneously, which enables the proposed model to extract intrinsic structure from perturbed graph under different attacks.

PROBLEM STATEMENT
Before we present the problem statement, we first introduce some notations and basic concepts. The Frobenius norm of a matrix S is defined by ||S|| 2 F = Σ i j S 2 i j . The ℓ 1 norm of a matrix S is given by ||S|| 1 = Σ i j |S i j | and the nuclear norm of a matrix S is defined (S) + denotes the element-wise positive part of matrix S where S i j = max{S i j , 0} and sдn(S) indicates the sign matrix of S where sдn(S) i j = 1, 0, or −1 if S i j >0, =0, or <0, respectively. We use ⊙ to denote Hadamard product of matrices. Finally, we use tr (S) to indicate the trace of matrix S, i.e., tr (S) = i S ii .
Let G = (V, E) be a graph, where V is the set of N nodes {v 1 , v 2 , ..., v N } and E is the set of edges. The edges describe the relations between nodes and can also be represented by an adjacency matrix A ∈ R N ×N where A i j denotes the relation between nodes v i and v j . Furthermore, we use X = [x 1 , x 2 , . . . , x N ] ∈ R N ×d to denote the node feature matrix where x i is the feature vector of the node v i . Thus a graph can also be denoted as G = (A, X). Following the common node classification setting, only a part of nodes V L = {v 1 , v 2 , ..., v l } are associated with corresponding labels Y L = {y 1 , y 2 , ..., y l } where y i denotes the label of v i .
Given a graph G = (A, X) and the partial labels Y L , the goal of node classification for GNN is to learn a function f θ : V L → Y L that maps the nodes to the set of labels so that f θ can predict labels of unlabeled nodes. The objective function can be formulated as where θ is the parameters of f θ , f θ (X, A) i is the prediction of node v i and ℓ(·, ·) is to measure the difference between prediction and true label such as cross entropy. Though there exist a number of different GNN methods, in this work, we focus on Graph Convolution Network (GCN) in [17]. Note that it is straightforward to extend the proposed framework to other GNN models. Specifically, a two-layer GCN with θ = (W 1 , W 2 ) implements f θ as Update GNN Parameters "#$ = " − ∇ *+ ℒ -.. whereÂ

Low-rank and
σ is the activation function such as ReLU.
With aforementioned notations and definitions, the problem we aim to study in this work can be formally stated as: Given G = {A, X} and partial node label V L with A being poisoned by adversarial edges and feature matrix X unperturbed, simultaneously learn a clean graph structure with the graph adjacency matrix S ∈ S = [0, 1] N ×N and the GNN parameters θ to improve node classification performance for unlabeled nodes.

THE PROPOSED FRAMEWORK
Adversarial attacks generate carefully-crafted perturbation on graph data. We refer to the carefully-crafted perturbation as adversarial structure. Adversarial structure can cause the performance of GNNs to drop rapidly. Thus, to defend adversarial attacks, one natural strategy is to eliminate the crafted adversarial structure, while maintaining the intrinsic graph structure. In this work, we aim to achieve the goal by exploring graph structure properties of low rank, sparsity and feature smoothness. The illustration of the framework is shown in Figure 2, where edges in black are normal edges and edges in red are adversarial edges introduced by an attacker to reduce the node classification performance. To defend against the attacks, Pro-GNN iteratively reconstructs the clean graph by preserving the low rank, sparsity, and feature smoothness properties of a graph so as to reduce the negative effects of adversarial structure. Meanwhile, to make sure that the reconstructed graph can help node classification, Pro-GNN simultaneously updates the GNN parameters on the reconstructed graph by solving the optimization problem in an alternating schema. In the following subsections, we will give the details of the proposed framework.

Exploring Low rank and Sparsity Properties
Many real-world graphs are naturally low-rank and sparse as the entities usually tend to form communities and would only be connected with a small number of neighbors [36]. Adversarial attacks on GCNs tend to add adversarial edges that link nodes of different communities as this is more efficient to reduce node classification performance of GCN. Introducing links connecting nodes of different communities in a sparse graph can significantly increase the rank of the adjacency matrix and enlarge the singular values, thus damaging the low rank and sparsity properties of graphs, which is verified in Figure 1a and Figure 1b. Thus, to recover the clean graph structure from the noisy and perturbed graph, one potential way is to learn a clean adjacency matrix S close to the adjacency matrix of the poisoned graph by enforcing the new adjacency matrix with the properties of low rank and sparsity. As demonstrated in Figure1c, the rank decreases much faster by removing adversarial edges than by removing normal edges. This implies that the low rank and sparsity constraint can remove the adversarial edges instead of normal edges. Given the adjacency matrix A of a poisoned graph, we can formulate the above process as follows: arg min Since adversarial attacks target on performing unnoticeable perturbations to graphs, the first term ∥A − S∥ 2 F ensures that the new adjacency matrix S should be close to A. As we assume that the graph are undirected, the new adjacency matrix should be symmetric, i.e., S = S ⊤ . R(S) denotes the constraints on S to enforce the properties of low rank and sparsity. According to [7,19,26], minimizing the ℓ 1 norm and the nuclear norm of a matrix can force the matrix to be sparse and low-rank, respectively. Hence, to ensure a sparse and low-rank graph, we want to minimize the ℓ 1 norm and the nuclear norm of S. Eq. (3) can be rewritten as: where α and β are predefined parameters that control the contributions of the properties of sparsity and low rank, respectively. One important benefit to minimize the nuclear norm ∥S∥ * is that we can reduce every singular value, thus alleviating the impact of enlarging singular values from adversarial attacks.

Exploring Feature Smoothness
It is evident that connected nodes in a graph are likely to share similar features. In fact, this observation has been made on graphs from numerous domains. For example, two connected users in a social graph are likely to share similar attributes [24], two linked web pages in the webpage graph tend to have similar contents [28] and two connected papers in the citation network usually have similar topics [17]. Meanwhile, recently it is demonstrated that adversarial attacks on graphs tend to connect nodes with distinct features [31]. Thus, we aim to ensure the feature smoothness in the learned graph. The feature smoothness can be captured by the following term L s : where S is the new adjacency matrix, S i j indicates the connection of v i and v j in the learned graph and (x i − x j ) 2 measures the feature difference between v i and v j . L s can be rewritten as: where L = D − S is the graph Laplacian matrix of S and D is the diagonal matrix of S. In this work, we use normalized Laplacian matrixL = D −1/2 LD −1/2 instead of L to make feature smoothness independent on the degrees of the graph nodes [1], i.e., where d i denotes the degree of v i in the learned graph. In the learned graph, if v i and v j are connected (i.e., S i j 0), we expect that the feature difference (x i −x j ) 2 should be small. In other words, if the features between two connected node are quite different, L s would be very large. Therefore, the smaller L s is, the smoother features X are on the graph S. Thus, to fulfill the feature smoothness in the learned graph, we should minimize L s . Therefore, we can add the feature smoothness term to the objective function of Eq. (4) to penalize rapid changes in features between adjacent nodes as: where λ is a predefined parameter to control the contribution from feature smoothness.

Objective Function of Pro-GNN
Intuitively, we can follow the preprocessing strategy [12,31] to defend against adversarial attacks -we first learn a graph from the poisoned graph via Eq. (8) and then train a GNN model based on the learned graph. However, with such a two-stage strategy, the learned graph may be suboptimal for the GNN model on the given task. Thus, we propose a better strategy to jointly learn the graph structure and the GNN model for a specific downstream task. We empirically show that jointly learning GNN model and the adjacency matrix is better than two stage one in Sec 5.4.2. The final objective function of Pro-GNN is given as arg min where L G N N is a loss function for the GNN model that is controlled by a predefined parameter γ . Another benefit of this formulation is that the information from L G N N can also guide the graph learning process to defend against adversarial attacks since the goal of graph adversarial attacks is to maximize L G N N .

An Optimization Algorithm
Jointly optimizing θ and S in Eq.(9) is challenging. The constraints on S further exacerbate the difficulty. Thus, in this work, we use an alternating optimization schema to iteratively update θ and S.
Update θ . To update θ , we fix S and remove terms that are irrelevant to θ , then the objective function in Eq.(9) reduces to: which is a typical GNN optimization problem and we can learn θ via stochastic gradient descent.
Update S. Similarly, to update S, we fix θ and arrive at where L(S, A) is defined as Note that both ℓ 1 norm and nuclear norm are non-differentiable. For optimization problem with only one non-diffiential regularizer Algorithm 1: Pro-GNN Data: Adjacency matrix A, Attribute matrix X, Labels Y L , Hyper-parameters α, β, γ , λ, τ , Learning rate η, η ′ Result: Learned adjacency S, GNN parameters θ 1 Initialize S ← A 2 Randomly initialize θ 3 while Stopping condition is not met do Return S, θ R(S), we can use Forward-Backward splitting methods [9]. The idea is to alternate a gradient descent step and a proximal step as: where η is the learning rate, prox R is the proximal operator as: In particular, the proximal operator of ℓ 1 norm and nuclear norm can be represented as [4,26], where Z = U diaд(σ 1 , ..., σ n )V ⊤ is the singular value decomposition of Z. To optimize objective function with two non-differentiable regularizers, Richard et al. [25] introduce the Incremental Proximal Descent method based on the introduced proximal operators. By iterating the updating process in a cyclic manner, we can update S as follows, After we learn a relaxed S, we project S to satisfy the constraints. For the symmetric constraint, we let S = S+S ⊤ 2 . For the constraint S i j ∈ [0, 1], we project S i j < 0 to 0 and S i j > 1 to 1. We denote these projection operations as P S (S).
Training Algorithm. With these updating and projection rules, the optimization algorithm is shown in Algorithm 1. In line 1, we first initialize the estimated graph S as the poisoned graph A. In line 2, we randomly initialize the GNN parameters. From lines 3 to 10, we update S and the GNN parameters θ alternatively and iteratively. Specifically, we train the GNN parameters in each iteration while training the graph reconstruction model every τ iterations.

EXPERIMENTS
In this section, we evaluate the effectiveness of Pro-GNN against different graph adversarial attacks. In particular, we aim to answer the following questions: • RQ1 How does Pro-GNN perform compared to the state-of-theart defense methods under different adversarial attacks? • RQ2 Does the learned graph work as expected?
• RQ3 How do different properties affect performance of Pro-GNN.
Before presenting our experimental results and observations, we first introduce the experimental settings.

Experimental settings
5.1.1 Datasets. Following [38,39], we validate the proposed approach on four benchmark datasets, including three citation graphs, i.e., Cora, Citeseer and Pubmed, and one blog graph, i.e., Polblogs. The statistics of the datasets are shown in Table 1. Note that in the Polblogs graph, node features are not available. In this case, we set the attribute matrix to N × N identity matrix.

Baselines.
To evaluate the effectiveness of Pro-GNN, we compare it with the state-of-the-art GNN and defense models by using the adversarial attack repository DeepRobust [20]: • GCN [17]: while there exist a number of different Graph Convolutional Networks (GCN) models, we focus on the most representative one [17]. • GAT [29]: Graph Attention Netowork (GAT) is composed of attention layers which can learn different weights to different nodes in the neighborhood. It is often used as a baseline to defend against adversarial attacks. • RGCN [37]: RGCN models node representations as gaussian distributions to absorb effects of adversarial attacks. It also employs attention mechanism to penalize nodes with high variance. • GCN-Jaccard [31]: Since attackers tend to connect nodes with dissimilar features or different labels, GCN-Jaccard preprocesses the network by eliminating edges that connect nodes with jaccard similarity of features smaller than threshold τ . Note that this method only works when node features are available. • GCN-SVD [12]: This is another preprocessing method to resist adversarial attacks. It is noted that nettack is a high-rank attack, thus GCN-SVD proposes to vaccinate GCN with the low-rank approximation of the perturbed graph. Note that it originally targets at defending against nettack, however, it is straightforward to extend it to non-targeted and random attacks.
In addition to representative baselines, we also include one variant of the proposed framework, Pro-GNN-fs, which is the variant by eliminating the feature smoothness term (or setting λ = 0).

Defense Performance
To answer the first question, we evaluate the node classification performance of Pro-GNN against three types of attacks, i.e., nontargeted attack, targeted attack and random attack: • Targeted Attack: Targeted attack generates attacks on specific nodes and aims to fool GNNs on these target nodes. We adopt nettack [38] for the targeted attack method, which is the stateof-the-art targeted attack on graph data. • Non-targeted Attack: Different from targeted attack, the goal of non-targeted attack is to degrade the overall performance of GNNs on the whole graph. We adopt one representative nontargeted attack, metattack [39] . • Random Attack: It randomly injects fake edges into the graph.
It can also be viewed as adding random noise to the clean graph.
We first use the attack method to poison the graph. We then train Pro-GNN and baselines on the poisoned graph and evaluate the node classification performance achieved by these methods.

Against Non-targeted Adversarial Attacks.
We first evaluate the node classification accuracy of different methods against nontargeted adversarial attack. Specifically, we adopt metattack and keep all the default parameter settings in the authors' original implementation. The metattack has several variants. For Cora, Citeseer and Polblogs datasets, we apply Meta-Self since it is the most destructive attack variant; while for Pubmed, the approximate version of Meta-Self, A-Meta-Self is applied to save memory and time. We vary the perturbation rate, i.e., the ratio of changed edges, from 0 to 25% with a step of 5%. As mentioned before, all the experiments are conducted 10 times and we report the average accuracy with standard deviation in Table 2. The best performance is highlighted in bold. From the table, we make the following observations: • Our method consistently outperforms other methods under different perturbation rates. For instance, on Polblogs dataset our model improves GCN over 20% at 5% perturbation rate. Even under large perturbation, our method outperforms other baselines by a larger margin. Specifically, under the 25% perturbation rate on the three datasets, vanilla GCN performs very poorly and our model improves GCN by 22%, 12% and 14%, respectively. • Although GCN-SVD also employs SVD to get low-rank approximation of the graph, the performance of GCN-SVD drops rapidly. This is because GCN-SVD is designed for targeted attack, it cannot adapt well to the non-targeted adversarial attack. Similarly, GCN-Jaccard does not perform as well as Pro-GNN under different perturbation rates. This is because simply preprocessing the perturbed graph once cannot recover the complex intrinsic graph structure from the carefully-crafted adversarial noises. On the contrary, simultaneously updating the graph structure and GNN parameters with the low rank, sparsity and feature smoothness constraints helps recover better graph structure and learn robust GNN parameters. • Pro-GNN achieves higher accuracy than Pro-GNN-fs especially when the perturbation rate is large, which demonstrates the effectiveness of feature smoothing in removing adversarial edges.

Against Targeted Adversarial Attack.
In this experiment, nettack is adopted as the targeted-attack method and we use the default parameter settings in the authors' original implementation. Following [37], we vary the number of perturbations made on every targeted node from 1 to 5 with a step size of 1. The nodes in test set with degree larger than 10 are set as target nodes. For Pubmed dataset, we only sample 10% of them to reduce the running time of nettack while in other datasets we use all the target nodes. The node classification accuracy on target nodes is shown in Figure 3. From the figure, we can observe that when the number of perturbation increases, the performance of our method is better than other methods on the attacked target nodes in most cases. For instance, on Citeseer dataset at 5 perturbation per targeted node, our model improves vanilla GCN by 23% and outperforms other defense methods by 11%. It demonstrates that our method can also resist the targeted adversarial attack.

Against Random Attack.
In this subsection, we evaluate how Pro-GNN behaves under different ratios of random noises from 0% to 100% with a step size of 20%. The results are reported in Figure 4. The figure shows that Pro-GNN consistently outperforms all other baselines and successfully resists random attack. Together with observations from Sections 5.2.1 and 5.2.2, we can conclude that Pro-GNN is able to defend various types of adversarial attacks. This is a desired property in practice since attackers can adopt any kinds of attacks to fool the system.

Importance of Graph Structure Learning
In the previous subsection, we have demonstrated the effectiveness of the proposed framework. In this section, we aim to understand the graph we learned and answer the second question.

Normal Edges Against Adversarial Edges.
Based on the fact that adversary tends to add edges over delete edges [31,39], if the model tends to learn a clean graph structure, the impact of the adversarial edges should be mitigated from the poisoned graph. Thus, we investigate the weights of normal and adversarial edges in the learned adjacency matrix S. We visualize the weight density distribution of normal and perturbed edges of S in Figure 5. Due to the limit of space, we only show results on Pubmed and Polblogs under metattack. As we can see in the figure, in both datasets, the weights of adversarial edges are much smaller than those of normal edges, which shows that Pro-GNN can alleviate the effect of adversarial edges and thus learn robust GNN parameters. In this subsection, we study the performance when the graph is heavily poisoned. In particular, we poison the graph with 25% perturbation by metattack . If a graph is heavily poisoned, the performance of GCN will degrade a lot. One straightforward solution is to remove the poisoned graph structure. Specifically, when removing the graph structure, the adjacency matrix will be all zeros and GCN normalizes the zero matrix into identity matrix and then makes prediction totally by node features. Under this circumstance, GCN actually becomes a feed-forward neural network. We denote it as GCN-NoGraph. We report the performance of GCN, GCN-NoGraph and Pro-GNN when the graph is heavily poisoned in Table 3.
From the table, we first observe that when the graph structure is heavily poisoned, by removing the graph structure, GCN-NoGraph outperforms GCN. This observation suggests the necessity to defend poisoning attacks on graphs because the poisoned graph structure are useless or even hurt the prediction performance. We also note that Pro-GNN obtains much better results than GCN-NoGraph. This observation suggests that Pro-GNN can learn useful graph structural information even when the graph is heavily poisoned.

Ablation Study
To get a better understanding of how different components help our model defend against adversarial attacks, we conduct ablation studies and answer the third question in this subsection.

Regularizers.
There are four key predefined parameters, i.e., α, β, γ and λ, which control the contributions for sparsity, low rank, GNN loss and feature smoothness, respectively. To understand the impact of each component, we vary the values of one parameter and set other parameters to zero, and then check how the performance changes. Correspondingly, four model variants are created: Pro-GNN-α, Pro-GNN-β, Pro-GNN-γ and Pro-GNN-λ. For example, Pro-GNN-α denotes that we vary the values of α while setting β, γ and λ to zero. We only report results on Cora and Citeseer, since similar patterns are observed in other cases, shown in Figure 6.
From the figure we can see Pro-GNN-α does not boost the model's performance too much with small perturbations. But when the perturbation becomes large, Pro-GNN-α outperforms vanilla GCN because it can learn a graph structure better than a heavily poisoned adjacency graph as shown in Section 5.3.2. Also, Pro-GNN-β and Pro-GNN-λ perform much better than vanilla GCN. It is worth noting that, Pro-GNN-β outperforms all other variants except Pro-GNN, indicating that nuclear norm is of great significance in reducing the impact of adversarial attacks. It is in line with our observation that adversarial attacks increase the rank of the graph and enlarge the singular values. Another observation from the figure is that, Pro-GNN-γ works better under small perturbation and when the perturbation rate increases, its performance degrades. From the above observations, different components play different roles in defending adversarial attacks. By incorporating these components, Pro-GNN can explore the graph properties and thus consistently outperform state-of-the-art baselines.

Two-Stage vs One-Stage.
To study the contribution of jointly learning structure and GNN parameters, we conduct experiments with the variant Pro-GNN-two under metattack . Pro-GNN-two is the two stage variant of Pro-GNN where we first obtain the clean graph and then train a GNN model based on it. We only show the results on Cora in Table 4 due to the page limitation. We can observe from the results that although Pro-GNN-two can achieve good performance under large perturbation, it fails to defend the attacks when the perturbation rate is relatively low. The results demonstrate that jointly learning structure and GNN parameters can actually help defend attacks.

Parameter Analysis
In this subsection, we explore the sensitivity of hyper-parameters α, β, γ and λ for Pro-GNN. In the experiments, we alter the value of α, β, γ and λ to see how they affect the performance of our model. More specifically, we vary α from 0.00025 to 0.064 in a log scale of base 2, β from 0 to 5, γ from 0.0625 to 16 in a log scale of base 2 and λ from 1.25 to 320 in a log scale of base 2. We only report the results on Cora dataset with the perturbation rate of 10% by metattack since similar observations are made in other settings. The performance change of Pro-GNN is illustrated in Figure 7. As we can see, the accuracy of Pro-GNN can be boosted when choosing appropriate values for all the hyper-parameters. Different from γ , appropriate values of α and λ can boost the performance but large values will greatly hurt the performance. This is because focusing on sparsity and feature smoothness will result in inaccurate estimation on the graph structure. For example, if we set α and λ to +∞, we will get a trivial solution of the new adjacency matrix, i.e, S = 0. It is worth noting that, appropriate value of β can greatly increase the model's performance (more than 10%) compared with the variant without β, while too large or too small value of β will hurt the performance. This is also consistent with our observation in Section 5.4.1 that the low rank property plays an important role in defending adversarial attacks.

CONCLUSION
Graph neural networks can be easily fooled by graph adversarial attacks. To defend against different types of graph adversarial  Figure 7: Results of parameter analysis on Cora dataset attacks, we introduced a novel defense approach Pro-GNN that learns the graph structure and the GNN parameters simultaneously. Our experiments show that our model consistently outperforms state-of-the-art baselines and improves the overall robustness under various adversarial attacks. In the future, we aim to explore more properties to further improve the robustness of GNNs.


The Audit Investigation and Accounting Forensicin Detecting Fraud in Digital Environment

A forensic accountant helps organizations or individuals primarily to provide management support in the form of reports to detect fraud and support litigation, primarily through expert witness testimony. The objectives to be achieved in this study are to explore how investigative and forensic accounting audits are applied in detecting fraud in the digital environment and to find out why investigative audits and forensic accounting in detecting fraud in the digital environment need to be applied. Researchers conducted a library study by collecting several economic journals and books relating to the problem under study. The data collection method in this research is done by interpreting and analyzing qualitative data. The results of this study indicate that forensic accounting in detecting fraud in the digital environment can be done by computer forensic and investigations that must be done by making copies of the entire log data, making fingerprints from numerical data, making fingerprints from copies, making master list hashes and documenting data that has been done.

Introduction
The high level of corruption has become a powerful driver for the development of forensic accounting practices in Indonesia. Forensic accounting is needed because of the potential for fraud that is capable of destroying the government, business, education, departments and other sectors. Fraud occurs because of low corporate governance, weak enforcement, weaknesses in the field of law enforcement, accounting standards and others consistent with the level of corruption and weaknesses in the administration of the country. Lately, cases of fraud that often occur in Indonesia, namely cybercrime, or commonly called as white-collar crime. Lately, cybercrime has increasingly increased. The handling of cybercrime cases is still quite tricky because the technology in Indonesia is still inadequate, and the lack of knowledge of digital techniques. The methods and methods used to manipulate companies are numerous, and the possibility of detecting all fraud through a computer is just a dream. Only a small number of cases that occur can be revealed while cases that occur, the number is astonishing.
In addition to cybercrime, white-collar crime is a crime that is rife in Indonesia. White-collar crime is limited to crimes committed within the scope of their office and therefore, does not include crimes of murder, adultery, rape, and others. That is generally not within the scope of the activities of white-collar criminals. Some of the examples of a white-collar crime that occurred in Indonesia is the Century Bank bailout case that began in October 2008 in which the perpetrators of these crimes are officials who have authority in Century Bank. Another example that has recently happened in the case of Jiwasraya and Asabri Insurance which has a systemic impact. Furthermore, until now, these cases have not been resolved. One of the causes of the problematic detection of fraud in Indonesia is due to the continued development of forensic accounting and the absence of experts who can disclose the fraud so that handling is delicate. Accounting graduates who work as accountants or auditors, like it or not, must understand forensic accounting. Therefore, the discipline of accounting is required to make changes and follow the trend of current problems, especially those related to fraud issues. That way, academics can be more responsive to fraud cases both in the digital environment and outside the digital environment that often occurs as an indication of corruption in this country.

Audit Investigation
Over time, the development of forensic accounting has become more sophisticated, involving one more area, namely auditing. The growing complexity of business and the increasingly opening up of business and investment opportunities leads to a higher risk of fraud. Referring to various cases, both inside and outside the country shows that fraud can occur anywhere. In order to minimize losses due to fraud and improve the control system, if there are strong indications of fraud, the company is expected to take appropriate steps to conduct an investigative audit (Scotland, 2017). Conducting an investigative audit is based more on the mindset that in order to expose a fraud the auditor must think like the perpetrators of fraud themselves, by basing the implementation of procedures established at the planning, implementation, reporting and follow-up stages of the audit (Rapp, 2010).
Auditors must have the ability to prove the existence of fraud that has occurred and previously indicated by various parties. The auditor must be sensitive to all things that are unnatural whether it is felt too big, too small, too often, too low, too much, too little, or an odd impression. Auditors must be able to communicate in their "language". The auditor must also have the technical ability to understand financial concepts and the ability to conclude. Furthermore, it is also imperative for auditors to simplify financial concepts so people, in general, can understand what they mean. Investigative auditors are "a combination of lawyers, accountants, criminologists and detectives (or investigators)" (Justenhoven, Sechser, & Loitz, n.d.).

Definition of Investigative Audit
An investigative audit is a form of audit that aims to identify and uncover fraud or crime using approaches, procedures or techniques commonly used in an investigation or investigation of a crime (Imoniana, 2013).
An investigative audit is similar to the term Fraud Examination as referred to in the Fraud Examination Manual published by the Association of Certified Fraud Examiners (ACFE). According to the manual of the fraud examiners, what is meant by an investigative audit is the methodology for resolving fraud allegations from inception to disposition. More specifically, fraud examination involves obtaining evidence and taking statements, writing reports, testifying findings and assisting in the detection and prevention of fraud (Reurink, n.d.).

Difference between Financial Audit and Investigative Audit
There are several differences between a financial audit and an audit investigation, namely: 1. Basis of Audit Implementation; the basis of conducting an investigative audit is a request from the investigator to detect possible fraud. Besides, an investigative audit can also be conducted based on complaints from the public about suspicion of fraud and from audit findings that lead to the possibility of fraud obtained from previous financial audits. 2. Auditor's responsibility; in financial audits, audits are responsible on behalf of the audit institution or the Public Accounting Firm (KAP) where the auditor works. In an investigative audit, the auditor is responsible for the designated personal name, because if the statement in a court hearing is a false statement, the auditor in question will be subject to sanctions. 3. Audit Objectives; the purpose of the financial audit is to find out the client's financial statements by generally accepted accounting principles. An investigative audit aims to help the investigator to make the case clear by looking for the evidence needed to support the prosecutor's indictment. 4. Audit Techniques and Procedures; in financial audits, audit procedures and techniques used refer only to auditing standards, while investigative audits refer to auditing standards as well as the investigator's authority so that more extensive audit techniques can be used. 5. Implementation of the Audit Planning and Implementation Principle; the financial audit uses scepticism professionalism, while investigative audits besides using scepticism professionalism also use the principle of presumption of innocence. 6. Requirements of the Audit Team; auditors, must master accounting and auditing issues, while in investigative audits, auditors must also know the applicable legal provisions in addition to mastering accounting and auditing issues. 7. Audit Report; the auditor's opinion regarding the suitability of the financial statements with accounting principles generally applies. In an investigative audit, states who is responsible and involved in fraud cases handled, but still applies the principle of presumption of innocence.

The purpose of the investigation
The purpose of the investigation are some of them (Beneish et al., 2011): 1. Dismiss management. The aim is as a stern rebuke that management is unable to account for its fiduciary obligations. 2. Examine, collect and assess enough and relevant evidence. The aim will be to emphasize the acceptance of evidence as evidence to convince a judge in court. 3. Protect the reputation of innocent employees. 4. It is finding and securing relevant documents for investigation. 5. Find the embezzled assets and seek recovery from the losses incurred. 6. They are ensuring that the perpetrators of crime cannot escape their actions. 7. It wiped out all employees of a criminal. 8. It is ensuring that the company is no longer the target of pillaging. 9. Determine how the investigation will proceed.

Principles of Investigative Audit
The following principles based on experience and practice can be used as guidelines for investigators in each situation as follows (Beneish et al., 2011): 1. The investigation is the act of searching for the truth by paying attention to justice and based on the provisions of the applicable legislation. 2. Investigation activities include the use of evidence sources that can support the facts in question. 3. The investigator collects facts in such a way that the evidence obtained can provide his conclusions. 4. Information is the breath and blood of the investigation so that the investigator must consider all possibilities to obtain information. 5. Observations, information and interviews are an essential part of an investigation. 6. The perpetrators of crime are humans; therefore if they are treated as befits humans, then they will also respond like humans.

Axiom of Investigation Audit
There are three axioms in conducting an investigative audit (Reurink, n.d.): 1. Fraud is always hidden. Fraud, in this case, hides all aspects that might be able to direct other parties to find the occurrence of fraud. The parties involved closed their rotten meetings. The method of concealing fraud is so neat that even an experienced fraud inspector or investigator can be fooled. 2. Perform interactive proof. An auditor must consider whether there is evidence that could incriminate a suspect who has never committed fraud. Moreover, conversely, the auditor must also be able to consider whether evidence that does not incriminate a person has committed fraud. 3. Fraud happens to be the court's authority to decide. In investigating fraud, the investigator only makes guesses about whether a person is guilty or not based on the evidence he has collected. However, the fraud existence that occurs can be ascertained if a panel of judges and jury has decided it in court.

Audit Investigation Methodology
According to the internal audit methodology, a fraud auditor can test or examine several matters relating to the subject of the audit or the work procedures and organization where the fraud is suspected and the person concerned (Justenhoven et al., n.d.). To find answers to fraud without complete evidence, the auditor needs to make certain assumptions. The methodology emphasizes when and how to carry out an investigative of cases that have indications of fraud and have implications for the legal aspects, how to proceed. Investigation checks conducted to reveal the existence of fraud consists of many steps. Because the conduct of an investigative investigation of fraud is related to the individual rights of other parties, the investigation must be carried out after a very adequate and strong reason has been termed as predication.
Prediction is an overall condition that directs or shows the existence of a strong belief based on professionalism and caution from the auditor who has been equipped with training and understanding of fraud, that fraud has occurred, is happening, or will occur. Without prediction, an investigative examination cannot be carried out. This causes dissatisfaction from various circles who think that if an audit institution finds an indication of irregularities in carrying out its financial audit, then the institution can conduct an investigative audit.
The investigation is not necessarily directly carried out because the indications found are generally still very premature so that it requires a little deepening in order to obtain evidence that is strong enough to be carried out investigation investigations. The overall outline of the investigative audit process, from beginning to end, is broken down as follows (News, 2017):

Preliminary Information Review
The examiner carries out: gathering additional information, compiling facts and process events, determining and calculating tentative financial losses, determining tentative irregularities, and preparing initial hypotheses.

Planning Investigation Examination
At the planning stage, the initial hypothesis testing, identification of evidence, determining the place or source of evidence, analyzing the relationship of evidence with related parties, and preparing an investigation program.

Implementation
At the implementation stage: a collection of evidence, physical testing, confirmation, observation, analysis and testing of documents, interviews, refinement of hypotheses and review of working papers.

Reporting
The contents of the report of the investigation audit include elements against the law, facts and processes of the incident, the impact of financial losses due to irregularities/acts against the law, causes of unlawful actions, parties involved in irregularities/actions against the law that occur, and forms of cooperation between the parties involved in irregularities/actions against the law.

Follow Up
At this follow-up stage: the process has been submitted from the audit team to the leadership of the organization and formally subsequently submitted to law enforcement. Submitting a report on the results of an investigative audit is expected to have entered the investigation stage. Regarding testimonies in the proceedings in court, an investigative audit team can be appointed by the organization to provide expert information if needed.

Investigative Audit Techniques
Audit techniques are the methods used in auditing the fairness of the presentation of financial statements. Audit techniques commonly used in general audits such as (Olukowade & Balogun, 2016): i. Physical examination. In the usual physical examination is the calculation of cash, valuable paper, inventory, fixed assets, and tangible goods. For this technique, the investigator uses his senses to know or understand something. ii. Confirmation. Asking for confirmation is asking the other party (of the investigated) to confirm the truth or untruth of information. In an investigation, the investigator must consider whether a third party has an interest in the investigation. iii. Checking Documents. Document checks are always carried out in every investigation. With technological advances, documents certainly become more extensive, including information that is processed, stored, and transferred. iv. Analytical Review. Analytical reviews emphasize reasoning, the thought process. With ethical reasoning will lead to an investigator auditor on the description of the fair, proper or proper individual data deduced from the picture obtained globally, thoroughly. Analytical review is based on a comparison between what is faced with what should occur. v. Request an Explanation of Oral or Written from the Audite. Requests for information must be strengthened or collaborated with information from other sources or strengthened in other ways. vi. Recalculate. Recalculating is checking the correctness of the calculation. In investigations, the calculations faced are very complex, based on complex contracts or agreements, there may have been changes and renegotiations many times with different officials. vii. Observe. This technique is also not much different from a physical examination. Investigators also use their senses to make observations.
Only in investigative audits, audit techniques are exploratory, looking for 'arable areas', or probing or deepening. Of the seven audit techniques, in an investigative audit, the emphasis is on analytical review. To get the maximum results of an investigation, a fraud auditor must also master some investigative techniques, including (Olukowade & Balogun, 2016

Forensic Accounting
Forensic accounting arises because of the rapid development of fraud that occurs, to reveal the fraud required knowledge of forensic accounting. The term forensic accounting is a translation of forensic accounting. According to Cannon Webster's Collegiate Dictionary, the meaning of forensics can be interpreted "concerning the court" or "concerning the application of scientific knowledge to legal matters".
The Criminal Procedure Code (KUHAP) article 179 paragraph (1) states: "Everyone who is asked for his opinion as a judicial medical expert or doctor or other experts must provide expert testimony for justice". In practice, other expert groups include accountants or investigative audit executors who provide expert information for justice. However, they are not yet commonly known as forensic accountants.
Initially, in the United States, forensic accounting was used to determine the distribution of inheritance or reveal motives for murder. For the application of accounting to break the law, the term used forensic accounting (not audit). The practice of forensic accounting grew not long after the economic crisis hit Indonesia in 1997. Forensic accounting can have a useful role in enforcing the law in Indonesia, but its role is still not maximal. Currently, the Financial Transaction Reports and Analysis Center (PPATK) is trying to develop forensic accounting which began to develop in Indonesia since the 1997 economic crisis.

a. Private Sector Practice
Fraud if associated with weak corporate governance, can occur both in the public sector and in the private sector. The impact of fraud occurs in the corporate sector, that is the share price of the corporation concerned is lower than the market price. This will affect the assessment of investors when making decisions. It is not uncommon for investors to pay premium shares if the company is indicated to want to improve the weaknesses of corporate governance (Olukowade & Balogun, 2016).
Some terms in accounting treasury, namely fraud auditing, forensic accounting, investigative accounting, litigation support, and valuation analysis ". In everyday use litigation support is the broadest term and covers the other four terms. Bologna and Lindquist did not touch the term valuation analysis (Olukowade & Balogun, 2016).
This analysis is related to accounting or accounting elements. The parties in dispute in business dealings can ask one party to buy all of the other party's shares, or they can agree that the final buyer is the bidder submitting the highest price. The case of corruption, a calculation is needed on how much this country is losing. This is a general description of the scope of forensic accounting in the private or business sector.

b. Practices in the Government Sector
In the public sector (government), the practice of forensic accountants is similar to what was described above, namely in the private sector. The difference is that the stages in the entire set of forensic accounting are divided among various institutions. Some institutions conduct audits of state finances, there are several institutions that are part of the internal government, there are court institutions, there are institutions that support activities against crime in general, and corruption in particular such as (PPATK), and other institutions such as the KPK. There are also non-governmental organizations that function as pressure groups.
Each of these institutions has a mandate and authority regulated in the constitution, laws or other provisions. This mandate and authority will colour the scope of forensic accounting applied. Besides, political conditions and various other conditions will affect the scope of forensic accounting applied, including legal or non-legal approaches. The impact that occurs in the government sector if there is fraud is the disruption of the implementation of state administration. If not supported by vigorous law enforcement, accounting standards and others, the level of corruption and weakness in the administration of the country will increase.

Fraud
Fraud is a deliberate fraud that can cause losses unnoticed by the injured party and provides benefits for the perpetrators of fraud (Reurink, n.d.). In everyday terms, fraud is given a different name, such as theft, seizure, extortion, exploitation, embezzlement, forgery, and others. Fraud generally occurs because there is pressure to misuse or encouragement to take advantage of opportunities that exist, and there is justification (generally accepted) for these actions. Misstatement consists of two types, namely, error and fraud (Richhariya, 2012). Fraud is translated as cheating per Statement of Auditing Standards (PSA) No. 70; likewise, the errors and irregularities are each translated as errors and irregularities per the previous PSA No. 32.

Causes of Fraud
The factors that cause fraud occur first because of the opportunity (opportunity), by knowing the perpetrators can see the opportunity to teach their fraud activities in order to get wealth and profit. Second, the pressure (pressure) where the financial or non-financial situation is the most usual incentive to commit fraud. Third, rationalization occurs because jealousy, resentment, anger, want to get rich quick and believe they are high can be a motivator for someone committing fraud. These factors are better known as the fraud triangle. The cause of fraud explained by Bologna with GONE theory consists of four factors, i.e. (Reurink, n.d.): 1. Greed, associated with the existence of selfish behavior that potentially exists in every person. 2. Opportunity, relating to the state of an organization or community institution in such a way that there is an opportunity for someone to commit fraud against it. 3. Needs, related to the factors needed by individuals to support their lives which, according to him, is reasonable. 4. Exposure (disclosure), relating to actions or consequences that will be faced by perpetrators of fraud if the perpetrators are found committing fraud.

Signs of a Fraud
Fraud can be detected as early as possible if management or internal auditors are aware of these signs of fraud. Some of the signs of fraud are (Reurink, n.d

Elements of Fraud
Elements of fraud or fraud are as follows (Imoniana, 2013): 1. A false agreement of material facts, or in some instances an opinion. 2. The desire to do something wrong or to achieve a goal that is not consistent with regulations or public policy. 3. Disguising a goal through forgery and misrepresentation to carry out a plan. 4. The offender's confidence in the victim's negligence or inaccuracy. 5. Concealment from evil

Fraud Classification
Fraud can be classified into three types, according to the Association of Certified Fraud Examinations (ACFE) viz (Popoola & Che-Ahmad, 2013): a. Financial Statement Fraud committed by management, namely in the form of material misstatements of financial statements that are detrimental to investors and creditors. This fraud can be financial or nonfinancial. b. Asset Misappropriation Asset misuse can be classified as 'cash fraud' and fraudulent inventories and other assets, and fraudulent, fraudulent expenses. c. Corruption Corruption occurs when it meets three criteria which are a condition that a person can be charged with a corruption law, the three conditions are: 1) against the law, 2) enriching oneself or another person or corporation, 3) harming the country's finances or the country's economy.

How to Prevent fraud
The root of the problem of fraud is "fraud by need, by greed, and by opportunity". The purpose of the phrase is if we want to prevent fraud, eliminate or suppress the cause as little as possible. An internal auditor can also do several things to prevent fraud, among others (Olukowade & Balogun, 2016)are: Fraud in a digital environment is fraud related to computers or information technology (IT). The United States Department of Justice defines computer fraud as any illegal act that requires knowledge of computer technology to initiate fraud, investigate, or implement it. Computers have a significant influence on the environment of modern society and have resulted in many changes in a short period. Each company is also not immune from the use of information technology. The advantages of computers in the form of speed and accuracy in completing work to reduce the amount of labour, costs and minimize the possibility of making mistakes, resulting in people increasingly dependent on computers. Negative impacts can arise if an error occurs caused by computer equipment that will result in substantial losses for the user (users) or interested parties (Justenhoven et al., n.d.).
In today's world, information is the main asset and knowledge is power. Fraud in a digital environment is not a myth; several large and high cases occur due to the IT environment. The development of crime by using computer technology is increasingly diverse. To minimize fraud cases in a digital environment, the key lies in management to develop systems and procedures to prevent or increase the possibility of detection.

Fraud Related to Computers
Fraud in accounting is an act that causes reporting errors in financial statements. A crime related to a computer, in extensive terms, means a crime that has been committed or abetted through the use of a computer and a crime in which the computer itself is a victim.
Crimes commonly used with computers include embezzlement, property theft and proprietary information, fraud, impersonation and counterfeiting. A crime related to a computer is a crime related to work/position. That is, this is done by people who have the skills, knowledge and access. Fraud is easier for people in the organization than for outsiders. Therefore, it is essential to look at computer-related crime from various perspectives (Charlesworth & Charlesworth, 2019) : 1. Individual crime and motivation. 2. External environmental factors that increase motivation to commit computer crime.
3. An internal organizational culture that minimizes or maximizes the possibility of fraud.

Reasons for Fraud in a Digital Environment
Several compelling reasons make the automated accounting system tend to open up opportunities for fraud. The principal reason is the possibility of the functioning of the computer without human involvement (impersonal). Fraud in the information technology environment usually occurs because of (Charlesworth & Charlesworth, 2019): 1. Controlling that is too old-fashioned and does not perform its barrier function adequately, at least for the average white-collar criminal. 2. Normal, rigid controls emphasized in a manual system suddenly disappear because of an automation system. 3. Management seems to be ready to face the challenges of securing the computer in its face. 4. Computer suppliers can be understood, more interested in selling innovative application solutions. 5. Employees who usually are overly enthusiastic about safeguarding physical assets are generally apathetic to EDP safeguards. 6. Auditors cannot follow the challenges of the presence of a modern system. 7. Access to the system has increased dramatically since the late 1970s. 8. System users are no longer intimidated or scared by automation. 9. Outsiders have a better chance of penetrating the network that results in fraudulent "encryption" dates sent through most networks and increases the use of "dial-up" connections.

Types of fraud in digital environments
Many cases of computer fraud that can not be revealed in the company environment because fraud perpetrators have used various methods to commit computer fraud. The categorization of computer fraud through the use of a data processing model can be detailed as follows (Charlesworth & Charlesworth, 2019) : i. The simplest and most common way to carry out fraud is to change computer input. ii. Computer fraud can be done through the use of a system (processor) by unauthorized persons, including theft of time and computer services as well as the use of computers for purposes outside of the initial job description. iii. Computer fraud can be achieved by disrupting software that processes company data or computer technology.
This includes changing software, making illegal copies or using it without authorization. iv. Computer fraud can be done by changing or destroying company data files or making copies, using or searching data without authorization. v. Computer fraud can be carried out by stealing or misusing system output.