summaries: In this paper, a graph view-consistent learning framework (GVCLN) is proposed. Specifically, two view learners are used to give predictions for the input. Then, a consistency loss is employed to force the two viewers giving the same predictions. Moreover, a co-training scheme is proposed to alleviate the label sparsity problem.

summaries: This paper proposes a view-consistent framework to address the issues of expensive labels. In particular, this work first uses graph neural networks and graph attention networks to construct two different latent features of the same data. Then, it uses the same classification neural networks to produce the node classification outcomes. Finally, it uses the classification outcome to construct a so-called "view loss". In addition, it uses an incremental strategy to gradually included pseudo labels until some termination conditions are satisfied.

summaries: This paper adopts a multi-view learning approach for graph representation learning where some labels are assumed to be available. It uses graph convolution network (GCN) and graph attention network (GAT) to create two different views of the same graph and then define a loss function to force the output due to the two views to be consistent. The low label rate scenario is considered and pseudo labels are created to define an additional loss function to better enforce consistency. Three datasets are used for performance evaluation.

abstractText: Recent years, methods based on neural networks have made great achievements in solving large and complex graph problems. However, high efficiency of these methods depends on large training and validation sets, while the acquisition of ground-truth labels is expensive and time-consuming. In this paper, a graph view-consistent learning network (GVCLN) is specially designed for the semisupervised learning when the number of the labeled samples is very small. We fully exploit the neighborhood aggregation capability of GVCLN and use dual views to obtain different representations. Although the two views have different viewing angles, their observation objects are the same, so their observation representations need to be consistent. For view-consistent representations between two views, two loss functions are designed besides a supervised loss. The supervised loss uses the known labeled set, while a view-consistent loss is applied to the two views to obtain the consistent representation and a pseudo-label loss is designed by using the common high-confidence predictions. GVCLN with these loss functions can obtain the view-consistent representations of the original feature. We also find that preprocessing the node features with specific filter before training is good for subsequent classification tasks. Related experiments have been done on the three citation network datasets of Cora, Citeseer, and PubMed. On several node classification tasks, GVCLN achieves state-of-the-art performance.

authors:
James Atwood
Don Towsley

title: Diffusion-convolutional neural networks

venue: In Advances in neural information processing systems,

authors:
Avrim Blum
Tom Mitchell

title: Combining labeled and unlabeled data with co-training

venue: In Proceedings of the eleventh annual conference on Computational learning theory,

authors:
Joan Bruna
Wojciech Zaremba
Arthur Szlam
Yann LeCun

title: Spectral networks and locally connected networks on graphs

venue: arXiv preprint arXiv:1312.6203,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey E. Hinton

title: A simple framework for contrastive learning of visual

venue: representations. ArXiv,

authors:
Kyunghyun Cho
Bart Van Merriënboer
Dzmitry Bahdanau
Yoshua Bengio

title: On the properties of neural machine translation: Encoder-decoder approaches

venue: arXiv preprint arXiv:1409.1259,

authors:
Michaël Defferrard
Xavier Bresson
Pierre Vandergheynst

title: Convolutional neural networks on graphs with fast localized spectral filtering

authors:
David K Duvenaud
Dougal Maclaurin
Jorge Iparraguirre
Rafael Bombarell
Timothy Hirzel
Alán Aspuru-Guzik
Ryan P Adams

title: Convolutional networks on graphs for learning molecular fingerprints

authors:
Bo Han
Quanming Yao
Xingrui Yu
Gang Niu
Miao Xu
Weihua Hu
Ivor Tsang
Masashi Sugiyama

title: Co-teaching: Robust training of deep neural networks with extremely noisy labels

authors:
Michael Kampffmeyer
Arnt-Borre Salberg
Robert Jenssen

title: Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition workshops,

authors:
Thomas N Kipf
Max Welling

title: Semi-supervised classification with graph convolutional networks

venue: arXiv preprint arXiv:1609.02907,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing

authors:
Qimai Li
Zhichao Han
Xiao-Ming Wu

title: Deeper insights into graph convolutional networks for semi-supervised learning

venue: arXiv preprint arXiv:1801.07606,

authors:
Qimai Li
X. Wu
Hongmei Liu
Xiaotong Zhang
Zhichao Guan

title: Label efficient semisupervised learning via graph filtering

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Sitao Luan
Mingde Zhao
Xiao-Wen Chang
Doina Precup

title: Break the ceiling: Stronger multiscale deep graph convolutional networks

authors:
Waseem Rawat
Zenghui Wang

title: Deep convolutional neural networks for image classification: A comprehensive review

venue: Neural computation,

authors:
Ke Sun
Zhouchen Lin
Hantao Guo
Zhanxing Zhu

title: Virtual adversarial training on graph convolutional networks in node classification

venue: In Chinese Conference on Pattern Recognition and Computer Vision (PRCV),

authors:
Ke Sun
Zhanxing Zhu
Zhouchen Lin

title: Multi-stage self-supervised learning for graph convolutional networks

venue: arXiv preprint arXiv:1902.11038,

authors:
JM Vandenberg
BT Matthias

title: Clustering hypothesis of some high-temperature superconductors

venue: Science, pp. 194–196,

authors:
Xiao-Ming Wu
Zhenguo Li
Anthony M So
John Wright
Shih-Fu Chang

title: Learning with partially absorbing random walks

authors:
Muhan Zhang
Yixin Chen

title: Link prediction based on graph neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Muhan Zhang
Zhicheng Cui
Marion Neumann
Yixin Chen

title: An end-to-end deep learning architecture for graph classification

venue: In Thirty-Second AAAI Conference on Artificial Intelligence,

authors:
Si Zhang
Hanghang Tong
Jiejun Xu
Ross Maciejewski

title: Graph convolutional networks: Algorithms, applications and open challenges

venue: In International Conference on Computational Social Networks,

authors:
Yanqiao Zhu
Yichen Xu
Feng Yu
Qiang Liu
Shu Wu
Liang Wang

title: Deep graph contrastive representation learning

venue: arXiv preprint arXiv:2006.04131,

heading: 1 INTRODUCTION

text:
Convolutional neural networks (CNNs) (Krizhevsky et al., 2012) performed outstandingly in solving problems such as image classification (Rawat & Wang, 2017), semantic segmentation (Kampffmeyer et al., 2016) and machine translation (Cho et al., 2014) etc. This is because CNNs can effectively reuse the convolution kernel and use the given input to train optimal parameters. The original data mentioned in above problems all have a grid-like data structure, that is, Euclidean spatial data. In reality, there are also lots of non-Euclidean spatial data, such as social networks, telecommunication networks, biological networks, and brain connection structures, etc. These data are usually represented in the form of graphs, where every node in the graph represents a single individual. Graph problems can be roughly divided into there direction: link prediction (Zhang & Chen, 2018), graph classification (Zhang et al., 2018a) and node classification (Kipf & Welling, 2016). In this paper, we focus on semi-supervised node classification when the label rate is very low.
Many new methods have been proposed to generalize the convolution operation to process graph structure data on arbitrary graphs for node classification. These methods can be divided into spatial convolution and spectral convolution methods (Zhang et al., 2018b). For spatial methods, they directly define graph convolution by designing certain operations on node’s neighbors. For example, Duvenaud et al. (2015) propose a convolutional neural network that can directly operate on graph data, which can provide an end-to-end feature learning method; Atwood & Towsley (2016) propose a fusion convolutional neural network (DCNNS), which introduces the graph fusion method to incorporate the context information of the nodes in the graph node classification; The Graph Attention Network (GATs) (Veličković et al., 2017) introduces the attention mechanism into the graph data processing to construct the attention layer for semi-supervised learning. The spectral method generally defines the graph convolution operation on spectral representation of graph. For example, Bruna et al. (2013) propose that graph convolution can be defined in the Fourier domain based on the eigenvalue decomposition of the graph Laplacian matrix; Defferrard et al. (2016) propose to use the Chebyshev expansion of the graph Laplacian to approximate the spectral domain filtering, which can
avoid high computational complexity brought by eigenvalue decomposition; Kipf & Welling (2016) propose a simpler Graph Convolutional Networks (GCNs) for semi-supervised learning, which can achieve higher classification accuracy by using a simple two-layer networks. However, large training and validation sets are required in these methods to complish effect classification task, while obtaining true labels is time-consuming, laborious and costly. And they send original graph node features directly into networks for training, however, there are lots of redundant information in the original features of the nodes.
In order to train an efficient model with only a few label nodes and even without validation, we put forward our own method: graph view-consistent learning network (GVCLN), which constructs a node classification network based on the consistency between two views. First, we independently train two-view encoders (can be different) to obtain two representations of every node. The function of the viewers is converting high-dimensional node features into low-dimensional embeddings (Zhu et al., 2020). The clustering hypothesis (Vandenberg & Matthias, 1977) show that examples in the same cluster are more likely to have the same label. According to this hypothesis, the decision boundary should try to pass through the place where the data is relatively sparse, so as to avoid dividing the data points in dense clusters on both sides of the decision boundary. Although the two views have different viewing angles, their observation objects are the same, so their observation results should be consistent. Therefore, the features encoded by the two viewers should make the decision boundary pass through the place where the date is sparse, that is, there should be consistency between the two views. Then, we design three loss functions, namely, supervised loss function, consistency loss function, and pseudo-label loss function. The supervised loss uses the known labeled set, while a view-consistent loss is applied to the two views to obtain the consistent representation and a pseudo-label loss is designed by using the common high-confidence predictions as pseudo label. GVCLN with these loss functions can obtain the view-consistent representation of the original feature. Our contributions are summarized as follows:
• We propose a graph view-consistent learning framework for semi-supervised node classification, which fully demonstrates the theoretical structure of graph view-consistency.
• We design GVCLN to successfully tackle label insufficiency in semi-supervised learning.
• We demonstrate the high efficacy and efficiency of the proposed methods on various semisupervised node classification tasks.

heading: 2 RELATED WORK

heading: 2.1 NOTATIONS

text:
A graph contains two parts: node and edge. Each node represents an individual, which can be a paper or a person, etc. The edge indicates a connection between two nodes. If the edge connecting two nodes in the graph is directional, it is directed graph, otherwise it is undirected graph. A simple and connected undirected graph can be written as G = (V, E), where V is the node set and E is the set of edges. n = |V | represents the number of all nodes in G. Considering that the node itself has a great influence on the graph structure, the graph used in the calculation of the network is generally a self-loop graph, namely G̃ = ( V, Ẽ ) , which attaches a self-loop to each node in G. A denotes the
adjacency matrix andD is the diagonal degree matrix. Therefore, the adjacency matrix and diagonal degree matrix of G̃ are defined as Ã = A + I and D̃, respectively. I indicates the identity matrix. The node feature matrix is X ∈ Rn×d, in which, each node i is associated with a d-dimensional feature vector xi. The normalized graph Laplacian matrix is defined as L = I − D−1/2AD−1/2, which is a symmetric positive semidefinite matrix with eigendecomposition UΛU>, where Λ is a diagonal matrix of the eigenvalues of L, and U ∈ Rn×n is a unitary matrix that consists of the eigenvectors of L. The graph convolution operation between signal x and filter gγ (Λ) = diag (γ) is defined as gγ (L) ∗ x = Ugγ (Λ)U>x, where the parameter γ ∈ Rn corresponds to a vector of spectral filter coefficients.

heading: 2.2 FEATURE AGGREGATION

text:
Graph neural Networks (GNNs) follow a neighborhood feature aggregation scheme. Many GNNs use different aggregation approaches. Here, we introduce two most classic feature aggregation ways.
Graph convolutional network (GCN) (Kipf & Welling, 2016) has outstanding effects in solving the problem of semi-supervised node classification. Kipf & Welling (2016) define the graph convolutional layer for feature aggregation in GCN. Symmetrically normalized operation is used on the adjacency matrix of G̃, i.e., Â = D̃−1/2ÃD̃−1/2. The graph convolutional layer is defined by,
H l+1 = σ ( ÂH lΘ ) , (1)
where H l is the feature of the l-th layer, Θ is the trainable weight matrix of the layer, and σ(·) is a nonlinear activation function, e.g., ReLU(·) = max(0, ·). Veličković et al. (2017) present graph attention network (GAT), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By giving different attention weights αij to different neighbor nodes, each node-i are able to aggregate their neighborhoods features to update their own features. The graph attention layer is defined by,
hl+1i = σ (∑ j∈Ni αijWh l j ) , (2)
where hi represents latent embedding feature at node i,W is a linear transformations weight matrix, and αij are normalized attention coefficients.

heading: 3 MODEL ARCHITECTURE

text: In this section, we first introduce the overall abstract structure of GVCLN, and then use graph convolutional layer and graph attention layer to build a specific GVCLN model to achieve viewconsistent learning for node classification.

heading: 3.1 MODEL ARCHITECTURE OVERVIEW

text: A general understanding of our GVCLN is shown in Fig. 1. In GVCLN, feature X and the graph A are encoded by two different viewers at the same time, namely Viewer 1 and Viewer 2, the two viewers aggregate features with three-head graph layers. Viewer 1 uses the graph convolution layer and Viewer 2 is the graph attention layer. The three-head representations, H(v)1 , H (v) 2 , H (v) 3 , v ∈ {1, 2}, are of concatenation and perform dropout. After dropout, the two-view latent features, Z(1) and Z(2), pass through a same non-linear graph convolutional layer to obtain the prediction, P (1) and P (2). A view-consistency loss function is made for the dual-view output consistently.

heading: 3.2 VIEW-CONSISTENCY LEARNING

text:
We consider applying view-consistent learning to the view 1 and view 2 in Figure 1. Inspired by contrastive learning, Chen et al. (2020) inputs the original sample and the sample after data augmentation (flipping or cropping, etc.) into two same views respectively. Since data augmentation does not change the label information of sample, the output representations of two views are similar. By maximizing the consistency of the two views, better results can be achieved. In this paper, we do not do data augmentation to the data, but used two different viewers. We believe that the same data has the similar output through different viewers and classifiers. In Figure 1, there is a consistency loss function between the view 1 and the view 2 to realize view-consistent learning of two views. The soft cross entropy function is employed to construct the consistency loss function:
`view = − 1
n n∑ i=1 k∑ j=1 p (1) ij ln p (2) ij . (3)
where k is the number of class, P (1) = [p(1)ij ] and P (2) = [p (2) ij ] are the predictions of the two viewers for node-i, respectively.
The concept of co-training was proposed by (Blum & Mitchell, 1998), co-training assumes that each data point has two views, and each view is sufficient for learning an effective model. Co-training learns separately on two views with labeled data samples to obtain two different learners and then uses the unlabeled data samples in the respective views of the two learners to make predictions. The basic learning process of collaborative training is: Each learner selects the data sample with the top high-confidence predictions in unlabeled data and adds it to the labeled data sample set of another learner for training. This process is repeated until a specific stopping condition is met. Co-training needs to meet the following two conditions:
• Views are sufficient and redundant, that is, for each view, if enough labeled data is given, each view can learn a learner with good performance.
• Conditionally independent, that is, the label information of each view is conditionally independent of the label information of another view.
Our view-consistent learning method is similar to co-training, which requires pseudo-labels for training. The selected pseudo-label data cannot be completely clean (the predicted label is exactly the same as the real label), but contains noise (the situation where the predicted label is inconsistent with the real label). Due to the powerful representation capabilities of deep neural networks, even noisy labels can be fitted by deep neural networks. Based on (Han et al., 2018), which can train deep neural networks with extremely noisy labels, we propose our own training strategy. We use labeled samples to train two independent learners, namely view 1 and view 2. When the two-view networks are well trained, according to their predictions, the predicted labels of some nodes with the same prediction representations and common high-confidence prediction are selected as pseudo labels. Then, pseudo label set is added to the original labeled set for training. This is repeated until the stop condition is reached.
We believe that pseudo labels have similar importance as real labels, but the number of pseudo-labels taken each time can be controlled by the loss function value and the accuracy of the verification set, and different numbers of pseudo-labels can be given to data sets with different label rates.
GVCLN selects nodes with high confidence and with same prediction results in the two views as pseudo labels. The pseudo labels are voted with high-confidence predictions of the two viewers taken from unlabeled nodes.
The pseudo-label loss has the same effect as consistency loss, that is: making the two views more and more consistent. Meanwhile, supervised loss has the function of fine-tuning and controlling the direction of network optimization.
We choose cross entropy as a loss function to calculate the loss for both the labeled samples and the pseudo labels:
CEi = − k∑ j=1 yij ln pij , (4)
where yij denotes the ground-truth label, i.e., the i-th node belongs to the j-th class, pij is the network prediction probability of the i-th label.
Thus, the supervised loss function of labeled samples is:
`sup = − 1
s s∑ i=1 k∑ j=1 yij ln pij . (5)
where s denotes the number of labeled nodes,
For pseudo labels, we also use cross entropy to calculate the loss function:
`pseudo = − 1
t t∑ m=1 k∑ i=1 y′ij ln pij . (6)
where y′ij is the pseudo label and t denote the number of nodes of the pseudo labels.

heading: 3.3 ALGORITHM

text:
Then, the total loss of GCVLN is given by:
`total = ` (1) sup + ` (2) sup + β`view + `pseudo, (7)
where β is a trade-off parameter, which is used to weigh the label loss function and the consistency loss function.
We first pre-train the GVCLN with `view and `sup, and second use the total loss function, `total. The specific process of GVCLN can be seen in Algorithm 1.
Algorithm 1 The GCVLN algorithm. 1: Input: X,A, epochmax, epochpre, β. 2: Output: P (1) and P (2). 3: Initialize: Model initialization. 4: X = ÂmX 5: for epoch ∈ [1, epochmax] do 6: if epoch < epochpre then 7: ` = `(1)sup + ` (2) sup + β`view .
8: Do optimization. 9: else 10: `total = ` (1) sup + ` (2) sup + β`view + `pseudo . 11: Do optimization. 12: end if 13: end for
In this paper, we use a renormalization filter (Li et al., 2019) to preprocess features as X ← ÂmX . The filter strength is tuned by the filter parameter m. When the label rate is low, m is increased, and when the label rate is high, m is reduced. By this operation, label efficient can be achieved.

heading: 4 EXPERIMENT

text: In this section, we use GVCLN to carry out related experiments and analyze the results in detail. The datasets include three citation network of Cora, Citeseer and PubMed. We conducted experiments with different label rates and without verification set. The division of the experimental dataset is the same as Luan et al. (2019). Classification accuracy of the test set is used as metric to evaluate the quality of model.

heading: 4.1 DATASETS

text:
We use three citation datasets: Cora, Citeseer and PubMed, which are widely used in graph problems. The specific information of the three data sets can be found in Table 1.
Cora: The Cora dataset contains 2708 nodes, which represent 2708 scientific documents and they can be divided into 7 categories. The citation network has 5,429 connection references, so the resulting graph contains 5,429 edges. The node feature is represented by a word vector containing 1433 independent and unique words, that is, the node feature is represented by a 1433-dimensional vector.
Citeseer: The Citeseer dataset contains 3327 nodes, which can be divided into 6 categories and each node represents one document. The citation network contains 4732 citation relationships. The feature of each document is represented by a word vector containing 3703 independent and unique words, that is, the feature of a node is represented by a 3703-dimensional vector.
PubMed: The PubMed dataset contains 19717 scientific papers, which can be divided into three categories. The citation network contains 44,338 citation relationships. The feature of each paper is represented by a word vector containing 500 independent unique words, and the feature of a node is represented by a 500-dimensional vector.

heading: 4.2 SETTING

text: GVCLN use graph convolutional layer and graph attention layer to build two viewers. The dimension of the hidden layer of Viewer 1 is set to 16, the drop rate is set to 0.2, the first large layer uses three-head graph convolutional layer to concatenate together, the second large layer uses only one graph convolutional layer; The dimension of the hidden layer of Viewer 2 is set to 8, the drop rate is set to 0.6, the first large layer uses three-head graph attention layer to concatenate together, and the second large layer uses only one graph convolutional layer. Other specific parameters are given in Appendix. Since the quality of the training set sampled each time is different, when the sampled training set node is located in the cluster center, the training and testing results are better. When the sampled training set node is located at the classification boundary, it will cause the training to be biased and the effect is poor, so the experimental results of this paper are the average results of ten tests. In the experiment, when the label rate is low, the filtering strength is increased and the number of pseudo labels is increased. As the label rate increases, the filtering strength and the number of pseudo labels are gradually reduced. At the same time, when the number of real label increases, the number of pseudo labels should be appropriately reduced.

heading: 4.3 RESULTS

text:
In our paper, the framework of view-consistent learning is applied to the problem of semi-supervised graph node classification and combined with graph feature filtering and pseudo-label learning, so that the GVCLN performs outstandingly when dealing with low label rate problems and without validation. The division of the experimental dataset is the same as Luan et al. (2019). Classification accuracy of the test set is used as metric to evaluate the quality of model. The test results without validation can be seen in Table 2. Then, we make a specific analysis of the test results of the three datasets.
Our baseline is Stronger Multi-scale Deep Graph Convolutional Networks (Luan et al., 2019), which generalize spectral graph convolution and deep GCN in block Krylov subspace forms and devise two architectures, both with the potential to be scaled deeper but each making use of the multi-scale information differently. Luan et al. (2019) designed three sub-models: Linear Snowball, Snowball and Truncated Krylov, all of which can efficiently classify graph nodes. We used the same experimental
setup as Luan et al. (2019), and conducted training and testing without a validation set and with different label rates. Our method GVCLN can surpass Luan et al. (2019) under different label rates. In the appendix, we give a list of parameter settings for GVCLN and Luan et al. (2019). We also compare against other methods, including label propagation using ParWalks (LP) (Wu et al., 2012), Chebyshev networks (Cheby) (Defferrard et al., 2016), Co-training (Li et al., 2018), Self-training (Li et al., 2018), Multi-stage self-supervised (M3S) training (Sun et al., 2019b), graph convolutional networks (GCN) (Kipf & Welling, 2016), GCN with sparse virtual adversarial training (GCN-SVAT) (Sun et al., 2019a) and GCN with dense virtual adversarial training (GCN-DVAT) (Sun et al., 2019a).
The second, third and fourth column of Table 2 show the accuracy of our proposed method GVCLN and other existing methods under different label rates. It can be seen that GVCLN can reach the highest level on three citation datasets. For Citeseer, when the labeling rate is 0.5%, the GVCLN test accuracy rate is nearly 6.4% higher than the highest level of Truncated Krylov.
Fig. 2 and Fig. 3 show the accuracy curve and loss function curve of the Cora dataset when the label rate is 0.5%. It can be seen that adding a pseudo-label when the epoch of training is 200 is helpful for the accuracy and loss function decline.
Fig. 4 is a t-SNE visualization diagram of the output results when each category contains 20 labeled samples, that is, the label rate is 5.6%. It can be seen from the visualization that the classification effect is still relatively good. The seven categories of the Cora data set finally appear to be clustered.
Figure 5 and Figure 6 are the t-SNE visualization diagrams of the Citeseer data set and PubMed data set, respectively.
The renormalization filter with filter strength m=20 is used in the GVCLN model to preprocess the node features, which plays an important role in achieving label efficiency. In the experiment, we performed t-SNE visualization analysis on the original node features and filtered node features respectively, and the visualization results can be seen in Figure 7 and Figure 8. It can be clearly seen that all nodes in the original node feature visualization result are clustered together, and the node features after filtering are scattered and show certain signs of clustering.

heading: 5 CONCLUSIONS

text:
We propose a new model, GVCLN, to solve the node classification problem in the case of low label rate. GVCLN adopts a dual-view structure to view-consistent learning. For the two viewers, we use the graph convolutional layer and the graph attention layer, respectively, and finally pass through a non-linear graph convolutional layer. Because the graph convolutional layer of viewer 1 is relatively simple, it can quickly learn to obtain node representations, and then increases the representation ability of graph attention layer by supervised the consistency loss function in the learning. In contrast, the graph attention layer of viewer 2 is more complicated, it can prevent the addition of pseudo-labels from making the GVCLN unstable, and prevent the pseudo-label errors amplify step by step. Thus, We propose a view-consistency learning method, and carry out relevant practices on the task of graph node classification. Good results can be obtained on the three citation datasets on all label rate and without validation.
We also found that when the label rate is very low, there may be only a few labeled nodes. If these few labeled nodes are well represented (can support the entire dataset), the classification effect is particularly significant. On the contrary, when they are not enough to support the entire dataset, the classification effect will be worse. The direction of our future work will shift from a semi-supervised low label rate to unsupervised direction.


summaries: This submission explores the question of identifying conserved quantities in Hamiltonian dynamics for physical systems by attempting to learn canonical transformations. The approach closely resembles previous work on "Hamiltonian neural networks" but the loss is augmented with a term enforcing the invariance of the dynamics under the transformation and with a term that ensures the resulting transformed coordinates satisfy the constraints of the algebraic relations that emerge from the Poisson bracket. Together these two additions allow the authors to train a network that performs a change of coordinates which is subsequently optimized to bring it closer to a canonical transformation. Perhaps the main observation is that some of the cyclic coordinates identified by the network have a clear relation to the underlying conserved quantities.

summaries: .** The authors propose using a neural network to learn a canonical transformation of the data coordinates before learning a Hamiltonian. This is a novel contribution in that previous work has shown how to learn Hamiltonians with neural networks but it has not shown how to learn the proper canonical transformation. In the course of learning this canonical transformation, they show how to project out other symmetries (linear and angular momentum) and hence improve upon HNNs while also learning these other symmetries to a good approximation.

summaries: This paper presents the results of a NN trained to learn symmetries in physics, specifically, to learn and preserve quantities that are preserved (e.g., energy, angular momentum). The input is a sequence generated from a Hamiltonian dynamics. Results of experiments on 2 and 3 body problems and a harmonic oscillator are presented. The training networks are small, shallow feedforward networks. There is some customization of the training networks to incorporate "cyclic" coordinates. Results indicated empirical conservation up to small error of physically conserved quantities. The paper is fairly easy to read, with much relevant background provided.

abstractText: This paper continues the quest for designing the optimal physics bias for neural networks predicting the dynamics of systems when the underlying dynamics shall be inferred from the data directly. The description of physical systems is greatly simplified when the underlying symmetries of the system are taken into account. In classical systems described via Hamiltonian dynamics this is achieved by using appropriate coordinates, so-called cyclic coordinates, which reveal conserved quantities directly. Without changing the Hamiltonian, these coordinates can be obtained via canonical transformations. We show that such coordinates can be searched for automatically with appropriate loss functions which naturally arise from Hamiltonian dynamics. As a proof of principle, we test our method on standard classical physics systems using synthetic and experimental data where our network identifies the conserved quantities in an unsupervised way and find improved performance on predicting the dynamics of the system compared to networks biasing just to the Hamiltonian. Effectively, these new coordinates guarantee that motion takes place on symmetry orbits in phase space, i.e. appropriate lower dimensional sub-spaces of phase space. By fitting analytic formulae we recover that our networks are utilising conserved quantities such as (angular) momentum.

authors:
Peter Battaglia
Razvan Pascanu
Matthew Lai
Danilo Jimenez Rezende

title: Interaction networks for learning about objects, relations and physics

venue: In Advances in neural information processing systems,

authors:
Roberto Bondesan
Austen Lamacraft

title: Learning symmetries of classical integrable systems

venue: arXiv preprint arXiv:1906.04645,

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

venue: In International conference on machine learning,

authors:
Miles Cranmer
Sam Greydanus
Stephan Hoyer
Peter Battaglia
David Spergel
Shirley Ho

title: Lagrangian neural networks

venue: In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations,

authors:
Miles D Cranmer
Rui Xu
Peter Battaglia
Shirley Ho

title: Learning symbolic physics with graph networks

venue: arXiv preprint arXiv:1909.05862,

authors:
Radek Grzeszczuk
Demetri Terzopoulos
Geoffrey Hinton

title: Neuroanimator: Fast neural network emulation and control of physics-based models

venue: In Proceedings of the 25th annual conference on Computer graphics and interactive techniques,

authors:
Jayesh K. Gupta
Kunal Menda
Zachary Manchester
Mykel J. Kochenderfer

title: A general framework for structured learning of mechanical systems, 2019

authors:
L.D. Landau
E.M. Lifshitz

title: Mechanics: Volume 1

venue: Number v. 1. Elsevier Science,

authors:
Shuo-Hui Li
Chen-Xiao Dong
Linfeng Zhang
Lei Wang

title: Neural canonical transformation with symplectic flows

venue: Physical Review X,

authors:
Michael Lutter
Christian Ritter
Jan Peters

title: Deep lagrangian networks: Using physics as model prior for deep learning

authors:
F. Pedregosa
G. Varoquaux
A. Gramfort
V. Michel
B. Thirion
O. Grisel
M. Blondel
P. Prettenhofer
R. Weiss
V. Dubourg
J. Vanderplas
A. Passos
D. Cournapeau
M. Brucher
M. Perrot
E. Duchesnay

title: Scikit-learn: Machine learning in Python

venue: Journal of Machine Learning Research,

authors:
H. Rein
S.F. Liu

title: REBOUND: an open-source multi-purpose N-body code for collisional dynamics

authors:
Subham Sahoo
Christoph Lampert
Georg Martius

title: Learning equations for extrapolation and control

venue: Proceedings of the 35th International Conference on Machine Learning,

authors:
Alvaro Sanchez-Gonzalez
Victor Bapst
Kyle Cranmer
Peter Battaglia

title: Hamiltonian graph networks with ode integrators

venue: arXiv preprint arXiv:1909.12790,

authors:
Michael D. Schmidt
Hod Lipson

title: Distilling free-form natural laws from experimental data

authors:
Peter Toth
Danilo Jimenez Rezende
Andrew Jaegle
Sébastien Racanière
Aleksandar Botev
Irina Higgins

title: Hamiltonian generative networks

venue: arXiv preprint arXiv:1909.13789,

authors:
Nicholas Watters
Daniel Zoran
Theophane Weber
Peter Battaglia
Razvan Pascanu
Andrea Tacchetti

title: Visual interaction networks: Learning a physics simulator from video

authors:
Sebastian J. Wetzel
Roger G. Melko
Joseph Scott
Maysum Panju
Vijay Ganesh

title: Discovering symmetry invariants and conserved quantities by interpreting siamese neural networks, 2020

heading: 1 Introduction

text:
Building in a bias to neural networks has been a key mechanism to achieve extra-ordinary performance in tasks such as classification. A standard example is to utilise translation invariance in convolutional neural networks Krizhevsky et al. (2012) and by now building in equivariance to other symmetries such as rotational symmetries has proven to be very successful (e.g. Cohen & Welling (2016)).
Possible motions are constrained due to symmetries of the system. In technical terms, motion takes place on a subspace of phase space. Energy conversation – related to invariance under time translation – has been utilised in the context of Hamiltonian Neural Networks (HNNs) Greydanus et al. (2019) where the energy functional, i.e. the Hamiltonian is inferred from data. This approach has seen large improvements in predicting the dynamics over baseline neural networks which simply try to predict the change of phase-space coordinates in time. Here we extend this approach by learning and incorporating additional constraints due to further symmetries of the system.
Coarsely speaking, finding symmetries corresponds to finding good coordinates. In classical mechanics this is achieved by performing canonical transformations and identifying cyclic coordinates which reveal conserved quantities. The aim of this paper is to demonstrate that multiple conserved quantities can indeed be automatically found in this way which has not been demonstrated beforehand.
Similar in spirit to learning the Hamiltonian, we formulate loss functions which enforce a representation in terms of cyclic coordinates and use them as the input for our Hamiltonian, differing from previous flow-based approaches searching for these coordinates (Bondesan & Lamacraft, 2019; Li et al., 2020). We experimentally find as a proof of principle that this mechanism identifies the underlying conserved quantities such as angular momentum, momentum, the splitting into decoupled subsystems, and can find the number of conserved quantities.
0.6 0.8 1.0 1.2 1.4 m1
0.6
0.8
1.0
1.2
1.4
m 2 40.0
900.0500.0 200.0
80.0 80 .0 5.0 30 .0 5.0 30.0
Figure 1: Effect of additional loss components: Loss-contours for the HNN-loss (6) (shown in gray) and the Poisson loss (7) (red) arising from the angular momentum (‖{L,H}‖2) in the 2-body Hamiltonian (10) with respect to two model parameters m1 and m2. The data model corresponds to m1 = m2 = g = 1 (indicated with a star) and we evaluate the loss over our training set. The analytic constraint m1 = m2 which arises from evaluating the Poisson bracket {L,H} ∼ (m1 − m2) is clearly visible and provides additional constraints on the model parameter space.
We demonstrate significant improvement in the predictions of the underlying Hamiltonian and subsequently the dynamics of the system. From our trained networks, we can find analytic expressions for the conserved quantities and determine the number of conserved quantities.

heading: 2 Theory

text:
We briefly describe the standard techniques in Hamiltonian mechanics which our network utilises.1 We consider a classical system with N particles in d spatial dimensions. Such a system can be described by the variables (q,p), where q = (q1, ..., qN ·d) are typically the positions for each dimension of the objects and p = (p1, ..., pN ·d) are the corresponding momenta. This is the input to our network and we are interested in predicting the timeevolution of this system, i.e. (q,p) at later time steps.
This pair (q,p) is an element of phase space in which every point corresponds to a state the system can take. The time evolution is governed by the Hamiltonian H(q,p)2 and the associated Hamiltonian equations:
dq dt = ∂H dp = {q,H} , dp dt = −∂H dq = {p,H} , (1)
where {•, •} are the Poisson bracket. They are defined as
{f, g} := N ·d∑ i=1 ∂f ∂qi ∂g ∂pi − ∂f ∂pi ∂g ∂qi . (2)
The Poisson bracket does not only arise for the time evolution of the canonical coordinates (q,p) but also for any function of these coordinates g(q,p) which does not explicitly depend on time:
dg(q,p)
dt = N ·d∑ i=1 ∂g ∂qi dqi dt − ∂g ∂pi dpi dt = {g,H} , (3)
where we have used the Hamiltonian equations (1) in the last step. From this expression we see that the Poisson bracket of a conserved quantity with the Hamiltonian H(q,p) vanishes and that the Hamiltonian itself is a conserved quantity.
The physics of this system is invariant under diffeomorphic coordinate transformations on the canonical coordinates and we will use coordinate transformations to reveal the constants of motions and hence the symmetries of the system. We use a particular type of diffeomorphic transformations T , namely canonical transformations which are transformations that leave the structure of the Hamiltonian equations (1) and in particular the Poisson bracket unchanged:
T : (q,p) 7→ (Q(q,p),P(q,p)) , {f, g}p,q = {f, g}P,Q, H(p,q) = H̃(P(p,q),Q(p,q)) . (4)
1We refer the reader for more details to standard textbooks such as Landau & Lifshitz (1982). 2We focus on time-independent Hamiltonians for simplicity.
• • Learning Symmetries
Training the neural network
14
{Pi, Pj} = {Qi, Qj} = 0 {Qi, Pj} = δij ·Pi = − ∂ℋ∂Qi ·Qi = ∂ℋ∂Pi ·Pi = 0
L = ∥ ∂ℋ∂p − ·qtarget∥2
+∥ ∂ℋ∂q + ·ptarget∥2 + . . .
We are interested in finding transformations T such that at least one coordinate satisfies
0 = Ṗi = − ∂H ∂Qi = {Pi,H} . (5)
Such a coordinate Pi is conserved in the system and the Hamiltonian does not depend on the associated Qi, i.e. it depends on fewer degrees of freedom and the motion in phase space is restricted to a lower dimensional manifold. Put differently, the cyclic coordinates provide via constraints of the type (5) additional restrictions on the allowed Hamiltonian function which we learn with our symmetry control neural networks (cf. Figure 1 shows explicit constraint from angular momentum conservation in a 2-body example in addition to constraints arising from satisying the Hamiltonian equations of motion).
Symmetry Control Neural Networks: In a first step we search for such cyclic coordinates with a network Tψ and use them as input for our Hamiltonian Hφ. The structure of our two trainable networks is shown in Figure 2. In order to find cyclic coordinates and to identify a Hamiltonian, our loss function contains several components:
1. The first loss ensures that our Hamiltonian satisfies Hamiltonian equations (1), which we can ensure as follows:
LHNN = N ·d∑ i=1 ∥∥∥∥∂Hφ(P,Q)∂pi − dqidt ∥∥∥∥ 2 + ∥∥∥∥∂Hφ(P,Q)∂qi + dpidt ∥∥∥∥ 2 . (6)
The time derivatives are provided by the data and the derivatives of the Hamiltonian with respect to the input variables can be obtained using auto-differentiation. This is the same loss as introduced in Greydanus et al. (2019).
2. To ensure that our transformation Tψ are of the type we are interested in (cf. Eq. (4)), i.e. our new coordinates fullfil the Poisson algebra, we enforce the following loss:
LPoisson = N ·d∑ i,j=1 ‖{Qi, Pj} − δij‖2 + N ·d∑ i,j>i ‖{Pi, Pj}‖2 + ‖{Qi, Qj}‖2 , (7)
where in some practical applications we only enforce this loss on n cyclic coordinate pairs. The first part of this loss ensures that a vanishing solution is not allowed.
3. Hamilton’s equations have still to be satisfied with respect to the new coordinates. For the cyclic coordinates we have enforced by our architecture that Hφ is independent of Qi. However, to ensure that Pi is actually conserved, we require the following additional loss:
L(n)HQP = n∑ i=1 ∥∥∥∥dPidt ∥∥∥∥ 2 + ∥∥∥∥dQidt − ∂Hφ(P,Q)∂Pi ∥∥∥∥ 2
+ β N ·d∑ i=n+1 ∥∥∥∥dPidt + ∂Hφ(P,Q)∂Qi ∥∥∥∥ 2 + ∥∥∥∥dQidt − ∂Hφ(P,Q)∂Pi ∥∥∥∥ 2 , (8)
where n denotes the number of cyclic variables we are imposing and β denotes a hyperparameter for our experiments. The time derivatives can be calculated using either expressions in (3).
Our total loss is a weighted sum of these three components:
L = LHNN + α1LPoisson + α2L(n)HQP , (9)
where the weights αi are tuned.
For integrating the solutions in time from our respective symmetry control network we use a fourth order Runge-Kutta integrator as in Greydanus et al. (2019) which unlike symplectic integrators allows for a comparison with neural network approaches directly predicting the dynamics of a system.3

heading: 3 Experiments

text:
Our experiments are designed with the following goals in mind:
1. SCNN-base: We want to compare the performance of symmetry control neural networks with HNNs and baseline neural networks which directly predict (q̇, ṗ). In these experiments we generally use the maximal number of conserved quantities. To evaluate the performance, we use the accuracy of the prediction and whether physically conserved quantities are actually conserved.
2. SCNN-constraint: We explore whether imposing domain knowledge about symmetries improves the performance. This is motivated by the fact that we often know about the existence of certain conserved quantities.
3. Analytic formulae: We evaluate whether our networks actually use conserved quantities in close proximity to known physically conserved quantities. To do this we fit analytical formulae on the conserved quantities and check how accurately we can recover known formulae.
Throughout, we run a hyperparameter analysis on how the different loss components influence our results to provide a first numerical analysis of these constraints. We test our method on synthetic examples and freely available experimental data, several of which have already been used in Greydanus et al. (2019).

heading: 3.1 Physical Systems and Objective of Experiments

text:
We consider the two-body problem in two dimensions which is governed by the following Hamiltonian
H = p 2 x1
2m1 + p2y1 2m1 + p2x2 2m2 + p2y2 2m2 − g ‖q1 − q2‖2 . (10)
To simplify the problem we set m1 = m2 = g = 1. We generate datasets with near circular orbits where the whole system has a small velocity in an arbitrary direction to avoid accidental symmetries (see Appendix B for more details). Besides the Hamiltonian itself, conserved quantities are the total momentum in x and y direction, and the angular momentum:
Px = px1 + px2 , Py = py1 + py2 ,
L = (px1 − px2) (qy1 − qy2)− (py1 − py2) (qx1 − qx2) . (11)
The second system is the 3-body system which is governed by the following Hamiltonian:
H = 3∑ i=1 p2xi 2mi + p2yi 2mi − 1 2 3∑ i=1,j 6=i
g
‖qi − qj‖2 , (12)
3We find that the main numerical inaccuracy in the prediction arises from inaccuracies of the Hamiltonian Hφ rather than the choice of this integrator when comparing it with standard symplectic integrators (Rein & Liu, 2012). Here in this paper, our focus is on predicting the Hamiltonian most accurately rather than making most accurate predictions with a Hamiltonian.
where we take a univeral mass and coupling mi = g = 1. As in the 2-body example, we consider systems which have a small velocity in an arbitrary direction (cf. Appendix B). Again, the overall momentum in both directions and the angular momentum are conserved quantities:
Px = px1 + px2 + px3 , Py = py1 + py2 + py3 ,
L = (2qx1 − qx2 − qx3) py1 − (2qy1 − qy2 − qy3) px1 + even permutations . (13)
The third system which we are interested in is the n-dimensional coupled oscillator which is described by the following Hamiltonian
H = n∑ i=1 p2i 2m + n∑ i,j=1 qiAijqj , (14)
where we use m = 1/2 and the coupling matrix Aij is a symmetric n × n−matrix with positive eigenvalues. As this matrix is not diagonal, the potential energy of each oscillator in these coordinates is influenced by the position of the other oscillators. By diagonalization the system can be described by decoupled harmonic oscillators. The conserved quantities are the energies of these subsystems:
Ei = p̃ 2 i + λiq̃ 2 i , (15)
where p̃ and q̃ are the canonical coordinates associated to the respective eigenvalue λi of the coupling matrix Aij . Note that these systems include spring-like interactions.
To include a model with few conserved quantities and to show that the SCNN can be used to determine the number of conserved quantities, we consider the double pendulum, a chaotic system, where only energy is conserved. We test the performance of SCNNs on a charged massive particle in a background magnetic field where the kinetic and potential energy are not decoupled in the Hamiltonian. To estimate the feasibility of our approach with experimental data where energy is not conserved, we test SCNNs on data from two experiments presented in (Schmidt & Lipson (2009)): two-masses-three-springs is a coupled two-dimensional harmonic oscillator, and we combine data from the physical pendulum to a real spherical pendulum. Details about these systems can be found in Appendix A and B.

heading: 3.2 Results

heading: 3.2.1 2-Body Problem

text:
We have trained our networks with data from 800 trajectories with 50 datapoints each and checked the performance on 200 different trajectories with 50 datapoints (cf. Appendix B for more details). We tested a large set of different hyperparameters (cf. Appendix C), where the large bulk of the models converges and reveals the conserved quantities. The implementation of our experiments is submitted alongside with this paper. We compare the following networks which all converge to a reasonable accuracy and our large training set ensures that overfitting does not pose a problem here:
• The baseline architecture has two hidden dense layers with 200 hidden units and tanh activation. The output is directly (q̇, ṗ).
• HNN: We use the original architecture with two hidden dense layers, both networks using tanh activations and 200 hidden units.
• SCNN-base: we use a transformation network architecture with two hidden dense layers (tanh activation) and a Hamiltonian neural network with two hidden dense layers as above. We enforce four cyclic coordinates (the maximum possible number). As described below, we vary the respective loss-weights and find that a loss with α1,2 = 0.001 works best.
• SCNN-constraint: we impose domain knowledge about conserved momenta (Px, Py) and angular momentum (L) respectively. The Hamiltonian is calculated with our
standard two hidden dense layers from the input (q,p). Beyond the HNN-loss, we only impose the loss components ensuring {H,Pi} = 0 and {H,L} = 0 respectively (cf. Figure 1 for a visualization). Improve on loss description, if we want to keep it.
For all networks we analyse the time evolution predicted from our trained network by determining the trajectory, the deviation from the ground truth and the evolution of the conserved quantities. A sample for the predicted trajectories is shown for each network in Figure 3. To quantify the accuracy of the respective networks we show the mean-squared deviation from the ground truth trajectories which we obtain over 50 different trajectories on our trained networks. We see a modest improvement with our SCNN-base architecture and an order of magnitude improvement when we include domain knowledge about conserved quantities for SCNN-constraint networks. Finally the bottom line of plots in Figure 3 shows the deviation of conserved quantities predicted from our networks in comparison to the underlying conserved quantity where we find an improvement compared to the HNN network.
To analyse the conserved quantities, we fit polynomials in (q,p) to the output of the cyclic coordinates. Picking the solution with the lowest degree and good accuracy4 we recover known conserved quantities of this system in the SCNN-base-model:
Pc1 =− 4.21 px1 − 4.21 px2 − 1.26 py1 − 1.29 py2 (0.03) , Pc2 =− 0.93 px1 − 0.92 px2 − 3.23 py1 − 3.22 py2 (0.03) , L =− 1.07 qx1py1 + 0.88 qx1py2 + 0.93 qx2py1 − 1.03 qx2py2
+ 1.01 qy1px1 − 0.89 qy1px2 − 0.92 qy2px1 + 0.99 qy2px2 (0.10) . (16) where the remaining terms are smaller than 0.1 and the number in brackets denotes the respective mean-squared error of the fit. The first two are clear linear combinations of the conserved momenta in Equation (11) and the third one is corresponding to angular momentum. To find these analytic fits we have simply used a polynomial ansatz and checked that there are no significant changes when allowing for higher powers in the analytic ansatz (cf. Appendix E).

heading: 3.2.2 Three-body problem

text:
Apart from adapting the input and output dimensions appropriately, we keep the baseline and HNN architecture unchanged to the two-body problem. For our symmetry control networks we report results for a SCNN-base architectures with six cyclic coordinates and SCNN-constraint architectures where we use our domain knowledge and enforce as in the two-body examples the conservation of momenta and angular momentum.
We train our networks starting again at near circular orbits (cf. Appendix B). However, after an initial circular motion for around 10 time steps the motion changes drastically when two particles come close to each other. In our experiments, we are not able to capture these dynamics and we do not aim to predict the dynamics after this initial drastic deviation. We observe in our experiments that our symmetry control networks are capable to predict this initial drastic change correctly. This is in contrast to the HNN and baseline networks whose predictions for the trajectories are worse than for the symmetry control networks and do not capture this change. We show an example of the trajectories in Figure 3, alongside with the mean deviation of the trajectory, and the conservation of angular momentum, energy and the prediction of kinetic and potential energy. Instead of predicting the deviation from circular orbits, we find that the HNN networks predict more or less circular orbits, which explains a relatively good conservation of angular momentum while predicting trajectories which are worse with respect to the true trajectory.

heading: 3.2.3 Coupled harmonic oscillator

text:
Again we keep the HNN architecture and the architectures in comparison to our previous experiments unchanged apart from adapting the input and output dimensions. As these experiments do not pose a large challenge for either HNNs and our symmetry control neural
4We check that there is no significant change in the fit accuracy when including higher order polynomials.
Figure 4: Comparing the test loss for HNN (blue), SCNN with one (red) and two (green) cyclic coordinates over the training time. We can see that too many conserved quantities lead to a dramatic drop in performance, while choosing the correct number of conserved quantities results in a similar performance than the HNN. We averaged over 100 time steps to get a smooth curve.
networks, we would like to focus on fitting analytic expressions for the conserved quantities and the Hamiltonian. For instance, in the case of n = 2, we find that the conserved quantities show explicitly the decoupling of the two sub-systems in our symmetry control networks. We find for the conserved quantities the underlying energies of the subsystem, e.g.:
P1 = 0.67 q 2 1 − 3.6 q1q2 + 8.23 q22 + 0.46 p21 − 1.78 p1p2 + 4.2 p22 (0.29), P2 = −4.53 q21 − 1.74 q1q2 − 0.9 q22 − 4.5 p21 − 1.88 p1p2 − 0.57 p22 (0.26), (17)
where the largest of the neglected terms arises with a factor smaller than 0.05 and the MSE of the fit is shown in the brackets. The entire Hamiltonian is also fitted very accurately as
H = 1.02 q21 − 0.41 q1q2 + 1.89 q22 + 0.97 p21 + 0.99 p22 (0.01), (18)
where the largest term which is neglected has a factor smaller than 0.05 and the MSE error is shown in the brackets. For these experiments we have used a SCNN-base with two cyclic coordinates. Details regarding the data we have used for training and testing can be found in Appendix B.

heading: 3.2.4 Double Pendulum

text: The double pendulum is a good example to see what happens when looking at more conserved quantities than there exist. We tested the SCNN with one and two conserved quantities while setting the values of αi = 1 to increase the effect of our symmetry ensuring loss components (β = 0 for comparability). Comparing with the HNN, we find that the the test loss (LHNN ) of the SCNN is very similar to the loss of the HNN for one conserved quantity, while for two conserved quantities the convergence is much slower and after 50000 steps the difference is still a factor three over the complete test set (see Figure 4). We are not able to find an analytical formular of the conserved quantity but this comes as no surprise due to the complicated structure of the Hamiltonian (cf. Appendix A).

heading: 3.2.5 Further Experiments and Summary of Performance

text: We have analysed the effect of varying the loss weights on our physical systems and find no large effects when varying the weights in the range 10−3 < α < 1 (cf. Appendix D for detailed results). A summary of the performance of our SCNNs in comparison to HNNs can be found in Table 1. To estimate the performance on the trajectories and the overall energy conservation we have integrated the mean square deviation from the true trajectory and the energy of the system respectively. Whereas the performance on the test loss is comparable to the HNN and baseline (a notable exception being the real harmonic oscillator and the double pendulum), the trajectories and energy are predicted much more accurately. An exception is the harmonic oscillator which already is very well captured with a HNN.

heading: 4 Related Work

text:
There are several related directions where symmetries have been learned and utilised with neural networks. Also the use of physics bias not just via the Hamiltonian framework is playing a large role in related works. We briefly comment on several of them:
Test loss Trajectory Energy Task SCNN (HNN) SCNN (HNN) SCNN (HNN) Two-body 5.25± 0.15 82e2± 11e2 14.79± 2.32 (×106) (2.13± 0.04) (114e3± 9e3) (19.12± 1.92)
Three-body 6.85± 1.96 14.0e2± 3.4e2 77.1e2± 55.6e2 (×105) (32.52± 2.08) (60.0e2± 8.8e2) (13.5± 2.4) HO (×106) 42.23± 1.19 1.15e3± 0.96e2 32.39± 9.24 (8.49± 0.23) (2.29e2± 1.39e2) (8.75± 2.88) Double Pend. 24.46± 2.00 13.2e2± 5.0e2 18.6± 5.6 (47.01± 3.97) (18.0e2± 6.3e2) (229.3± 169.4)
Real Sph. Pend. 6.84± 0.78 — — (6.61± 0.75) — — Real HO(×100) 1.85± 0.33 — — (6.10± 1.83) — —
Table 1: Summary of quantitative results across tasks. All quantities are multiplied with 103 if not otherwise stated.
• The baseline approach to predict dynamics has been studied in many other approaches previously (e.g. Grzeszczuk et al. (1998) for early work on this topic).
• An alternative to describing dynamics via Hamiltonian is via Lagrangian. The equivalent approach to HNNs in terms of learning Lagrangians is discussed in (Cranmer et al., 2020; Gupta et al., 2019; Lutter et al., 2019).
• Canonical transformations to cyclic coordinates have been utilised in the context of flow-networks. Bondesan & Lamacraft (2019) aim at finding the canonical transformation to cyclic coordinates but they do not study the improved predictions using those coordinates and their loss uses finite difference as opposed to an auto-differentiation loss we utilise here. Li et al. (2020) requires all coordinates to be cyclic (i.e. the system is integrable) which is not the case in our approach.
• Predicting the coordinates (q,p) from images has been pursued in (Greydanus et al., 2019; Toth et al., 2019). Here this would add an additional network before our input with the target output (q,p) as was explicitly demonstrated to work in (Toth et al., 2019).
• Very accurate prediction of dynamics can be performed using interaction networks (Battaglia et al., 2016; Watters et al., 2017). Biasing the local interactions via the Hamiltonian in graph networks has been discussed in Sanchez-Gonzalez et al. (2019). It would be very interesting to apply our approach in this framework to analyse the performance at larger systems.
• Obtaining analytic formulae for the network output on physical systems has been pursued in (Cranmer et al., 2019; Sahoo et al., 2018; Wetzel et al., 2020).

heading: 5 Discussion

text:
One great challenge in physics is the search for conserved quantities and underlying symmetries. Our architecture provides an approach on how to find these symmetries using neural networks automatically without prior knowledge of them. We find interesting performance on simple toy systems where the symmetries are well known. We are able to identify these symmetries and by using them we can improve on performance with Hamiltonian neural networks.
It will be extremely interesting to extend on this proof of concept analysis by applying it to state of the art predictive networks such as in (Sanchez-Gonzalez et al., 2019) which as of now only utilise the Hamiltonian and not all of the symmetries.
A second line of investigation is in the realm of physics. An example for an area of future application is in integrable systems which for classical systems is satisfied if all coordinates are cyclic. Determining whether a system is integrable is generally speaking unknown and our method provides an automated route to search for the required structures.
We hope to report on progress in these directions at future conferences.

heading: A Additional systems

text: This appendix presents details about some of the physical systems we have tested our SCNNs on.

heading: A.1 Spherical Pendulum

text:
The spherical pendulum is governed by the following Hamiltonian:
H = p2x + p 2 y
2m −m g
√ l2 − q2x − q2y , (19)
where we take a univeral mass m = 12 coupling g = 1 2 and pendulum length l = 1. Our data generation is described in Appendix B. In this system, the angular momentum in z-direction is conserved:
L = qxpy − qypx . (20) Additionally, one can find the different Hamiltonians for the subsystems due to the fact that the spherical pendulum can be seen as a two-dimensional harmonic oscillator with degenerate eigenvalues. We use the same architecture as for the harmonic oscillator, and focus on fitting conserved quantities. We enforce two cyclic coordinates (the maximal possible number) and scan over α1 = α2 ∈ {100, 10−1, 10−2, 10−3}, and are able to extract the conserved quantities (using α = 10−3):
P1 = 1.19 q 2 x + 0.4 qxqy + 0.72 q 2 y + 1.77 p 2 x + 0.51 pxpy + 1.09 p 2 y − 0.83 qxpy + 0.84 qypx , P2 = −0.65 q2x + 0.58 qxqy − 1.38 q2y − 0.96 p2x + 0.77 pxpy − 1.92 p2y − 1.18 qxpy + 1.18 qypx , (21)
with MSE of 0.003 and 0.002 and the omitted terms are below 0.05. One can see here the degenerated energies. The fitted Hamiltonian is:
H = 0.68 q21 + 0.71 q 2 2 + 1.02 p 2 1 + 1.02 p 2 2 (0.03). (22)
The true Hamiltonian (19) and the found one differ only by a factor of 0.005, and, therefore, linear regression finds a harmonic oscillator like structure.

heading: A.2 Charged Particle in a Magnetic Field

text:
The charged particle in a magnetic field, where we choose a circular B-field plus an electric field, is governed by the following Hamiltonian:
H = 1 2m
( (px + q qy B) 2 + (py − q qx B)2 ) + k ( q2x + q 2 y ) , (23)
where we take a univeral mass m = 12 , electric field k = 1, charge q = 1 and B-field B = 1. Here, the angular momentum in z-direction is conserved
L = qxpy − qypx . (24)
We use the same neural network architecture as in the previous example, and enforce cyclic coordinate conditions on the maximum number of coordinates, but in this case we used α1 = α2 = 10
−2 due to the better performance when fitting the analytic quantities. Using linear regression, we find the following analytic expressions for the conserved quantities and for Hamiltonian:
P1 = 3.59 q 2 x + 3.58 q 2 y + 1.80 p 2 x + 1.79 p 2 y − 2.74 qxpy + 2.75 qypx (0.43), P2 = −2.33 q2x − 2.32 q2y − 1.17 p2x − 1.17 p2y + 2.97 qxpy − 2.96 qypx (0.32), H = 2.00 q2x + 2.00 q2y + 1.00 p2x + 1.00 p2y − 2.01 qxpy + 2.01 qypx (0.01), (25)
where we omitted terms smaller than 0.05 and the numbers in the bracket indicate the MSE of the fit.

heading: A.3 Real harmonic oscillator

text: As in the previous cases, we use the SCNN architecture to predict the motion of the particle. We are limited to the given data points from the datasets, and therefore only have one given trajectory. Therefore, the SCNN is not able to find the subsystems of the two dimensional harmonic oscillator, but is able to act on the test set with a good accuracy and performs much better than the HNN (see Table 1).

heading: B Data

heading: B.1 Two-body problem

text:
As described in the main text we set both masses and the coupling to unity. We sample 1000 trajectories with 50 data points each over a time span of 20 which corresponds of at least two third of a full circle. We uniformly sample the starting point for the first body q1 from the box [0.5, 1.5]2. The starting position for the second body is at q2 = −q1. We choose the velocity of both bodies in such a way that the bodies describe near circular orbits and we give the entire system an overall momentum pcom = ( 1, 2)
p1 = (1 + ) pcirc + pcom , p2 = −p1 + 2 pcom , (26)
where and 1,2 are sampled from a normal distribution with σ = 0.05 and 0.1 respectively.
We take 80 percent of the data for training and use the rest for testing.

heading: B.2 Three-body problem

text:
The setup is relatively similar to the two-body problem as we consider three body near circular motions and give the overall system an overall momentum. We initialise the three particles on the corners of an equilateral triangle. The velocity of the first particle is given as follows
p1 = (1 + ) pcirc + pcom , (27)
where the remaining velocities are chosen accordingly as in the two-body problem and where the center of mass motion and the deviation from the circular motion are exactly generated as in the two-body example. We use 5000 trajectories with 20 data points for training and testing of our networks. The time span is 5 which corresponds to motion which is mostly circular. When using longer time spans, we find that convergence of the networks becomes problematic.
B.3 n-dimensional harmonic oscillator
The underlying Hamiltonian (14) is generated as follows. The coupling matrix is generated as the product of an orthogonal matrix S and a diagonal matrix D as Aij = SD 2ST .
The entries of the diagonal matrix D are random entries sampled uniformly in the interval [1 − √ 10] which are then rounded to one digit to ensure that the values are sufficiently rational, i.e. that the motion is periodic. For instance, if we choose an entry in D2 equal to 2 the motion is not periodic whereas if we choose 1.96 the motion is periodic. The matrix S is obtained by generating random normal distributed entries (mean 0 and standard deviation 0) and we then orthonormalize the vectors.
The initial conditions (q(t = 0),p(t = 0)) are a 2n-dimensional vector which we generate as follows. We generate a vector with random entries (uniform sample) in the interval (0− 2). We then normalize the vector to unit length and then re-scale it uniformly to a vector of length in the interval (0.2, 2.0).
For our dataset we utilise 100 initial conditions and generate 500 points on the trajectory. These points are taken from a time span of 10 which corresponds to a time where multiple periodic motions are included. We split it into equal size training and test set.

heading: B.4 Spherical Pendulum

text:
As described above we fix all parameters, and sample the starting conditions (q (t = 0) ,p (t = 0)) from a uniform distribution from the interval (−1, 1). We ensure, that the energy is well-defined and negative.
For our dataset we utilise 100 initial conditions and generate 500 points on the trajectory. These points are take from a time span of 10 which corresponds to a time where multiple periodic motions are included. We use a 80:20 split for the training and test set.
B.5 Charged particle in a magnetic field
As described above we fix all parameters, and sample the starting conditions (q (t = 0) ,p (t = 0)) from a uniform distribution from the interval (−1, 1), We then normalize the vector to unit length and then re-scale it uniformly to a vector of length in the interval (0.1, 1.0). For our dataset we utilise 100 initial conditions and generate 500 points on the trajectory. These points are take from a time span of 10 which corresponds to a time where multiple periodic motions are included. We use a 80:20 split for the training and test set.

heading: B.6 Double pendulum

text: As described in the main text we fix all parameters, and sample the starting conditions (q (t = 0) ,p (t = 0) = 0) from a uniform distribution from the interval (0, 2π). Note, that θ1 and θ2 are angles and therefore, we have to ensure that they always lie in the interval (0, 2π). For our dataset we utilise 100 intial conditions and generate 500 points on the trajectory. These points are take from a time span of 10 which corresponds to a time where multiple periodic motions are included. We use a 80:20 split for the training and test set.

heading: B.7 Real harmonic oscillator

text:
In this real world example we use data from the paper Schmidt & Lipson (2009), namely the example which is called
” two-masses-three-springs“. To measure the data points, they
use a system with two fixed boundary points, connected by a system with three springs and two masses. As this corresponds to two real harmonic oscillators we refer to this system as real harmonic oscillator. The dataset consists of one trajectory from a real world system which is loosing energy. To make it comparable to the paper of Greydanus et al. (2019), we used the same split (taking the first 80 percent as training set, and the remaining part as test set). Therefore we have a different distribution in the training set and the test set. Because the data consists of only a single trajectory of data points we are technically not able to find a conserved quantity. Therefore, we just test the performance and check how well the network generalizes to the test set.

heading: B.8 Real Spherical Pendulum

text: In this real world example we use data from the paper Schmidt & Lipson (2009), namely the example of the real pendulum. Due to the fact that we want to have more than one degree of freedom (otherwise the only conserved quantity is the energy), we use always a pair of two data points from the one-dimensional pendulum to get a two dimensional movement which is physically equivalent to a spherical pendulum. We use the first 80 percent of the data points as training set, and the remaining ones as test set. Because this is a real-world system it looses some energy, and therefore, we have a slightly different distribution on the training set and the test set, but this difference is no limitation for the search of conserved quantities.

heading: C Hyperparameters

text:
We have performed some hyper-parameter searches for our symmetry control networks on which we give an overview here.
By varying the hidden layer size between 50-300, we find that there is a minimum size of layer size 100 to find convergence of our networks. We have varied the number of hidden layers up to 5 hidden layers. Two hidden layers are already sufficient (tested up to 5 hidden layers), hence we restricted ourselves on them. Coarsely speaking, we find that the precise architecture is less relevant in our current experiments.
More relevant are the pre-factors in the loss. Depending on the choice, we can either force our networks for better performance on the predictions or obtaining the conserved quantities. For this trade-off, we have optimized the experiments in this paper on the particle trajectories.
We find that pytorch’s standard orthogonal initialization provides the best results out of the standard initializations. We have not observed a large random seed dependence.

heading: D Loss Prefactors

text: In our analysis of the loss factors we consider SCNNs with maximum number of conserved quantities. This leaves us with two loss hyperparameters α1 and α2 which we analyze here. We scanned them over the values {100, 10−1, 10−2, 10−3} for the 2-body problem. The results are in Table 2 below, where you can see the average error of the trajectories as used in the main text. As a comparison, the error of the HNN is significantly higher with 1.15± 0.09× 10−1. The second measure we compare are the conserved quantities. Therefore, we look at the conservation of energy and angular momentum which is shown in Table 3 and 4. As a comparison, the error of the energy for HNN is 1.91 ± 0.19 × 10−5 and 1.23 ± 0.10 × 10−3 for the angular momentum.

heading: E Power for Multinomial-Expansion

text:
In this appendix we elaborate how we decide which power for the multinomial expansion. For illustration purposes, we use the two-body example here.
We use the test set as the coordinates and fit a multinomial function using LinearRegression and PolynomialFeatures implemented in sklearn (Pedregosa et al., 2011). When looking at the MSE-plot for the different degrees (cf. Figure 5), one can directly find a cut-off to determine the correct degree. Coordinate 1 and Coordinate 2 have degree one (both give us momentum conservation), while the fourth one can be fitted by a multinomial of order two (It is equivalent to the angular momentum). On the other hand, one can directly see, that Coordinate 3 has no multinomial expression, the error is small, but does not converge to a specific value.

title: Symmetry Control Neural Networks