summaries: This work concerns the metric learning between sequences using RNNs. The paper notices the similarity between a dynamical system and an RNN. Then it demonstrates that learning a pair of siamese RNNs is similar to learning synchronization between two subsystems of a dynamical system. Finally, the paper proposes to introduce coupling between the two RNNs in order to improve synchronization.

summaries: Drawing inspiration from dynamic systems, the paper proposes a novel architecture that couple sequences. Such a system has easiness to bring two instances arbitrarily close and authors have shown the superiority of the approach over an action recognition dataset; but the results seem to far from state of the art on the dataset (see questions section). The authors also recognize that currently such systems need to calculate each pairs (can't be cached due to coupling) at inference time, which is slow.

summaries: I liked the formulation and motivation of the paper, explaining the sequence metric learning problem  and drawing parallel between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese style recurrent neural network. The authors propose modification the siamese recurrent network setting called classical Gated Recurrent Unit architecture (CGRU). The premise being two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. The authors describe how this model is able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way with the coupling demonstrating performance of the siamese Gated Recurrent Unit (SGRU) architecture on UCI activity recognition dataset (mobile data).

abstractText: Sequence metric learning is becoming a widely adopted approach for various applications dealing with sequential multi-variate data such as activity recognition or natural language processing and is most of the time tackled with sequence alignment approaches or representation learning. In this paper, we propose to study this subject from the point of view of dynamical system theory by drawing the analogy between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese recurrent neural network. Indeed, a siamese recurrent network comprises two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. We therefore propose a new neural network model that implements this coupling with a new gate integrated into the classical Gated Recurrent Unit architecture. This model is thus able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way. Our experiments show that introducing such a coupling improves the performance of the siamese Gated Recurrent Unit architecture on an activity recognition dataset.

authors:
Abubakar Abid
James Zou

title: Autowarp: learning a warping distance from unlabeled time series using sequence autoencoders

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

authors:
Davide Anguita
Alessandro Ghio
Luca Oneto
Xavier Parra
Jorge Luis Reyes-Ortiz

title: A public domain dataset for human activity recognition using smartphones

venue: ESANN,

authors:
Yoshua Bengio
Patrice Simard
Paolo Frasconi

title: Learning long-term dependencies with gradient descent is difficult

venue: IEEE transactions on neural networks,

authors:
Luca Bertinetto
Jack Valmadre
Joao F Henriques
Andrea Vedaldi
Philip HS Torr

title: Fullyconvolutional siamese networks for object tracking

venue: In European conference on computer vision,

authors:
Stefano Boccaletti
Jürgen Kurths
Grigory Osipov
DL Valladares
CS Zhou

title: The synchronization of chaotic systems

venue: Physics reports,

authors:
Jane Bromley
Isabelle Guyon
Yann LeCun
Eduard Säckinger
Roopak Shah

title: Signature verification using a "siamese" time delay neural network

venue: In Advances in Neural Information Processing Systems,

authors:
Reggie Brown
Ljupčo Kocarev

title: A unifying definition of synchronization for dynamical systems

venue: Chaos: An Interdisciplinary Journal of Nonlinear Science,

authors:
Reggie Brown
Nikolai F Rulkov

title: Designing a coupling that guarantees synchronization between identical chaotic systems

venue: Physical review letters,

authors:
Xingyu Cai
Tingyang Xu
Jinfeng Yi
Junzhou Huang
Sanguthevar Rajasekaran

title: Dtwnet: a dynamic time warping network

authors:
Bo Chang
Minmin Chen
Eldad Haber
Ed H Chi

title: Antisymmetricrnn: A dynamical system view on recurrent neural networks

authors:
Kyunghyun Cho
Bart Van Merriënboer
Caglar Gulcehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using rnn encoder-decoder for statistical machine translation

venue: arXiv preprint arXiv:1406.1078,

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural networks on sequence modeling

venue: arXiv preprint arXiv:1412.3555,

authors:
A De Angeli
R Genesio
A Tesi

title: Dead-beat chaos synchronization in discrete-time systems

venue: IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications,

authors:
Hirokazu Fujisaka
Tomoji Yamada

title: Stability theory of synchronized motion in coupled-oscillator systems

venue: Progress of theoretical physics,

authors:
Ken-ichi Funahashi
Yuichi Nakamura

title: Approximation of dynamical systems by continuous time recurrent neural networks

venue: Neural networks,

title: Convergent activation dynamics in continuous time networks

venue: Neural Networks,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Isao Ishikawa
Keisuke Fujii
Masahiro Ikeda
Yuka Hashimoto
Yoshinobu Kawahara

title: Metric on nonlinear dynamical systems with perron-frobenius operators

authors:
Thomas Kreuz
Florian Mormann
Ralph G Andrzejak
Alexander Kraskov
Klaus Lehnertz
Peter Grassberger

title: Measuring synchronization in coupled model systems: A comparison of different approaches

venue: Physica D: Nonlinear Phenomena,

authors:
Thomas Laurent
James von Brecht

title: A recurrent neural network without chaos

venue: arXiv preprint arXiv:1612.06212,

title: A metric for arma processes

venue: IEEE transactions on Signal Processing,

authors:
Jonas Mueller
Aditya Thyagarajan

title: Siamese recurrent architectures for learning sentence similarity

venue: In AAAI,

authors:
Hyun Oh Song
Yu Xiang
Stefanie Jegelka
Silvio Savarese

title: Deep metric learning via lifted structured feature embedding

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Razvan Pascanu
Tomas Mikolov
Yoshua Bengio

title: On the difficulty of training recurrent neural networks

venue: In International Conference on Machine Learning,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An imperative style, high-performance deep learning library

authors:
Louis M Pecora
Thomas L Carroll

title: Synchronization in chaotic systems

authors:
Michaël Perrot
Amaury Habrard

title: Regressive virtual metric learning

authors:
Hiroaki Sakoe
Seibi Chiba

title: Dynamic programming algorithm optimization for spoken word recognition

venue: IEEE transactions on acoustics, speech, and signal processing,

authors:
Steven J Schiff
Paul So
Taeun Chang
Robert E Burke
Tim Sauer

title: Detecting dynamical interdependence and generalized synchrony through mutual prediction in a neural ensemble

venue: Physical Review E,

authors:
Florian Schroff
Dmitry Kalenichenko
James Philbin

title: Facenet: A unified embedding for face recognition and clustering

title: Improved deep metric learning with multi-class n-pair loss objective

title: Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology

authors:
Bing Su
Gang Hua

title: Order-preserving wasserstein distance for sequence matching

authors:
Bing Su
Ying Wu

title: Learning distance for sequences by learning a ground metric

authors:
Ilya Sutskever
Oriol Vinyals
Quoc V Le

title: Sequence to sequence learning with neural networks. In Advances in neural information processing

authors:
Rahul Rama Varior
Mrinal Haloi
Gang Wang

title: Gated siamese convolutional neural network architecture for human re-identification

title: Attention is all you need

authors:
Kilian Q Weinberger
Lawrence K Saul

title: Distance metric learning for large margin nearest neighbor classification

venue: Journal of Machine Learning Research,

authors:
Xiaopeng Xi
Eamonn Keogh
Christian Shelton
Li Wei
Chotirat Ann Ratanamahatana

title: Fast time series classification using numerosity reduction

venue: In Proceedings of the 23rd international conference on Machine learning,

authors:
Eric P Xing
Michael I Jordan
Stuart J Russell
Andrew Y Ng

title: Distance metric learning with application to clustering with side-information

authors:
Xun Yang
Peicheng Zhou
Meng Wang

title: Person reidentification via structural deep metric learning

venue: IEEE Transactions on Neural Networks and Learning Systems,

heading: 1 INTRODUCTION

text:
Metric learning aims at learning an essential component for numerous machine learning algorithms used for classification or clustering: a similarity. It has the benefit to be usable in weakly supervised settings where only equivalence constraints between samples are known (Xing et al. (2003)), which allows for a large number of applications on various data types: from person re-identification (Yang et al. (2018)), object tracking (Bertinetto et al. (2016)) and gesture recognition (Berlemont et al. (2018)) to sentence similarity computation (Mueller & Thyagarajan (2016)). Among those applications, less attention has been given to design specific sequence metric learning algorithms, specifically with neural networks despite the simplicity of the siamese architecture (Bromley et al. (1994)).
One easy way to adapt existing approaches to sequential data is to learn representations through Sequence-to-Sequence models (Sutskever et al. (2014)) or Transformers (Vaswani et al. (2017)). However, these models would be difficult to learn in a weakly supervised way for providing a similarity metric and further lose temporal dependency information inside the sequence and alignment information between sequences. On the contrary, Dynamic Time Warping (DTW) (Sakoe & Chiba (1978)) is a classical approach to measure distance between sequences and relies on aligning sequences. Its integration inside learning algorithms has been rendered difficult by its non-differentiability and its theoretical quadratic time complexity which badly suits the equivalence constraint framework and some associated more complex losses (Oh Song et al., 2016; Sohn, 2016; Yang et al., 2018). Recent works mitigate these drawbacks notably with virtual metric learning (Perrot & Habrard, 2015; Su & Wu, 2019) and soft versions of DTW (Cai et al., 2019; Abid & Zou, 2018).
Therefore, we aim at designing a neural network architecture specifically adapted to sequence metric learning. Recurrent neural networks (RNN) have a temporal dynamic behavior which allows to study them as dynamical systems. We propose in this paper a new framework for sequence metric
learning based on dynamical system synchronization theory. We propose to replace the concept of metric in a vector space by the concept of synchronization of trajectories in a state space. Instead of computing distances on input representations, we propose to measure how two dynamical systems, and precisely two RNNs, respond to input pairs in term of synchronization. The notion of coupling is crucial when trying to synchronize dynamical systems. We introduce a coupled version of the Gated Recurrent Unit (GRU) (Cho et al. (2014)) to implement coupling inside a siamese architecture. Our experimental evaluation shows that this modification provides an improvement over a classical Siamese GRU implementation.
The paper is organized as follows: Section 2 outlines the state-of-the-art approaches in sequence metric learning, Section 3 describes our framework and our new siamese architecture, Section 4 shows our experimental results to assess the performances of our approach compared to the state of the art, and Section 5 presents our conclusions and perspectives.

heading: 2 RELATED WORK

text:
Recurrent neural networks and dynamical system theory. A main property of RNN is to exhibit a dynamic behavior which enables them to learn temporal sequence correlations. This behavior can therefore be studied using dynamical system theory: an important result being that RNN can approximate any finite-time trajectory of a dynamical system (Funahashi & Nakamura (1993)). Other early works analyzed the RNN convergence stability (Hirsch (1989)) and helped to understand the problem of long term dependencies (Bengio et al. (1994)). Laurent & von Brecht (2016) studied the dynamics of Long-Short Term Memory (LSTM) Neural Networks and GRU and observed that it is chaotic in the absence of input data. They designed a Chaos-Free RNN architecture having a more predictable behavior. In another recent publication, Chang et al. (2019) studied RNN trainability and established a connection with discretized Ordinary Differential Equations stability. They identified a criterion to guarantee that the system can preserve long-term dependencies and proposed a new version of RNN based on those observations. Both papers demonstrate that dynamical system theory is a fertile soil to study and conceive new RNN models. Finally, we would like to mention works on the definition of metrics to compare non-linear dynamical systems (Martin (2000); Ishikawa et al. (2018)) although our objective is not exactly the same, as we propose to use dynamical system synchronization theory to improve metric learning on any type of sequential data, whereas these methods have been conceived to work more specifically with structural data.
Sequence metric learning. DTW is a classical approach for measuring distances between sequences (Sakoe & Chiba (1978)). Numerous improvements have been brought to the original formulation notably to improve the k-nearest neighbor performance (Xi et al. (2006)). Abid & Zou (2018) proposed to learn the DTW parameters that allow to reproduce the Euclidean distances between sequence representations learned with a Sequence-to-Sequence model (Sutskever et al. (2014)). In contrast, Su & Hua (2017) proposed an alternative to DTW, the Order-Preserving Wasserstein (OPW) distance, by viewing the problem of metric learning between sequences as an optimal transport problem regularized to preserve the temporal relationships between the samples, and they solved it with the matrix scaling algorithm. Later (Su & Wu (2019)), the authors reformulated the DTW and OPW distances as parameterized meta-metrics of a single ground metric and proposed an optimization process to learn the metric and the latent alignment with virtual metric learning (Perrot & Habrard (2015)), which reduces the number of constraints. Not only this approach speeds up training but it also outperforms several other metric learning approaches, notably approaches conceived for points generalized to sequences. In comparison, we propose a pure RNN approach similar to Mueller & Thyagarajan (2016) who presented a siamese neural network approach to learn sentence similarities as a l1-norm. In their method, the LSTM network combines the embeddings of the words of the sentence to learn a distance between representations of sentences. Finally, Varior et al. (2016) proposed a siamese convolutional architecture for person re-identification from video data with gates linking parallel layers allowing to accentuate common patterns between both representations. This leads to representations that are more suited to distinguish some pairs of similar or dissimilar images.
In this paper, we introduce an alternative to DTW: a pure neural network approach to sequence metric learning based on the siamese RNN architecture. We propose to enhance the classical Siamese RNN by studying this model from dynamical system point of view, as it has been already done for standard RNN. The disadvantage of DTW-based approaches compared to ours is that it can be
computationally inefficient and non-differentiable, which prevents their possible combination with other gradient-based models.

heading: 3 SYNCHRONIZING GRU SIAMESE NETWORKS

text: In this section, we first draw a parallel between the concept of synchronization for dynamical systems and the task of sequence metric learning with siamese RNN. We then justify from a theoretical point of view the introduction of coupling inside the siamese architecture. We finally introduce our main contribution in Section 3.3: a modified Siamese GRU model implementing this coupling.

heading: 3.1 SYNCHRONIZATION OF CHAOS AND METRIC LEARNING

text:
The concept of synchronization is generally well-understood for time-periodic dynamical systems: this phenomenon is called phase synchronization. However, it is less-known that it can also occur for chaotic dynamical systems (Pecora & Carroll (1990)), that is, systems for which resulting trajectories exponentially diverge for infinitesimally close initial conditions. This practically means that the behavior of such systems can become rapidly unpredictable solely due to small variations of the initial conditions. Common examples of such systems are the double pendulum or the n-body problem. To formalize the concept of synchronization for chaotic systems, Brown & Kocarev (2000) proposed a general definition of it:
Definition 1 Let Z be a dynamical system composed of two subsystems X and Y such that:
X : dx dt = f1(x, y; t) Y : dy dt = f2(y, x; t),
(1)
where x ∈ Rd1 and y ∈ Rd2 with d1, d2 ∈ N. Let φ(z0) be a trajectory of Z with initial conditions z0 = [x0, y0] ∈ Rd1 × Rd2 . Finally let g : X (resp. Y ) ×R → Rk with k ∈ N, be a measurable property of the subsystems. They are synchronized on the trajectory φ(z0) with respect to the property g if there is a time independent function h : Rk × Rk → Rk such that:
||h(g(x), g(y))|| = 0, (2)
where || · || is a norm.
From this definition, it is possible to derive several ways to measure synchronization between two trajectories. Brown & Kocarev (2000) report several slightly different formulations of the synchronization error with the following being the most used. According to them, for identical systems:
h(g(x), g(y)) = lim t→+∞
(g(x)− g(y)), (3)
where h and g are the same as in Definition 1.
We rewrite this synchronization error for discrete systems in a continuous form by replacing the limit by a comparison of the last element of each trajectory X and Y of length T :
h(g(X), g(Y )) = XT − YT , (4)
with g being here a function returning the coordinates of the points. We define d as a distance on discrete dynamical system trajectories derived from the synchronization error by replacing the difference with the Euclidean norm (in definition 1, any norm can be used) to get only positive values:
d(X,Y ) = ||XT − YT ||2. (5) However, RNNs are dynamical systems, and the output sequences are trajectories. Thus, learning a Euclidean distance with a siamese RNN is equivalent to trying to synchronize the output sequences of the two sub-networks of the siamese network for similar pairs.
While being intuitive and suitable for metric learning, the metric of Equation 5 measures synchronization only at one point in time, which forces the system to achieve synchronization at this precise point.
This is called dead-beat synchronization, synchronization in a finite number of steps (De Angeli et al. (1995)). Even if at first sight, this seems not really different from computing distance on input sequence representations, synchronization could actually be assessed at several samples of the sequence and even continuously. We will now specify what type of synchronization siamese RNNs are able to achieve and under which conditions.

heading: 3.2 COMPLETE SYNCHRONIZATION OF COUPLED IDENTICAL SYSTEMS

text:
A special case of Definition 1 is when f1 and f2 are the same function, the dynamical systems share the same parameters, and they are said to be identical. Analogously, the two (or more in case of triplet inputs) sub-networks of a siamese network also share the same parameters; only input sequences differ (Bromley et al. (1994)). To simplify, we will first study the case where the dynamics of the RNNs are solely driven by its initial condition (the initial hidden state) and where no input sequence is given (i.e. a sequence of 0s). We obtain what is called the dynamical system induced by the RNN. In this case, only the initial conditions differ and the sub-networks are identical dynamical systems. According to experiments conducted by Laurent & von Brecht (2016), dynamical systems induced by RNN exhibit a chaotic behavior. Under what conditions identical systems can synchronize? Consider now the following two identical systems:
X : dx dt = f(x; t) + C(y − x)T Y : dy dt = f(y; t) + C(x− y)T , (6)
where x, y ∈ Rn and C is a coupling matrix in Rn×n. This type of coupling is called diffusive because it will dissipate the dynamics of each sub-system (Boccaletti et al. (2002)). X and Y are bidirectionally coupled systems1. Fujisaka & Yamada (1983) showed that the system described by Equation 6 can achieve complete synchronization if C is a multiple of the Identity matrix and a constant c which verifies the following condition:
c > 1
2 λL, (7)
where λL is the largest Liapunov exponent of the system. The Liapunov exponents quantify the sensibility of a system to initial conditions: if it has one positive Liapunov exponent, predictability of its behavior becomes impossible beyond a certain time horizon; it is therefore chaotic (Strogatz (2018)).
But the trajectories of RNNs are most of the time also influenced by external inputs: the input sequence. In this case, the dynamics of the RNNs are mostly driven by these external inputs (Laurent & von Brecht (2016)) and RNN starting from different initial conditions but given identical input sequence will see their trajectories synchronize, i.e. the hidden states become the same after a few steps. Coupling is in this case not necessary to achieve synchronization: regarding metric learning, siamese LSTM actually works without coupling (Mueller & Thyagarajan (2016)). However, coupling could allow to enforce lower distances with sequences that have similar dynamics but are composed of quite different data, i.e. so-called hard positive samples, and even to force the synchronization regardless of the input pair. Hard positive samples are samples that are factually belonging to one class, by the label, but lie close to the decision boundary or even beyond in terms of distance (conversely for hard negative samples). They are particularly studied by the metric learning community to improve the convergence and performances of metric learning models Schroff et al. (2015).
We showed in this section the motivation and aim of implementing coupling into the siamese RNN architecture. Indeed, the induced dynamics of GRU and LSTM are chaotic and, in this case, coupling allows their complete synchronization as the sub-networks of a siamese network share the same weights and represent thus identical dynamical systems. When given input sequences, the dynamics of GRU and LSTM are mostly driven by these external inputs. In this case, while not being critical to achieve synchronization (and therefore low distances between similar elements), coupling could facilitate bringing similar inputs closer, particularly for hard positive pairs.
1Coupling can also be directional, only one system influences the other: they are called in this case driveresponse systems.

heading: 3.3 COUPLED GRU

text:
We present a new neural network model that directly implements coupling within a siamese RNN architecture. From a machine learning perspective, this coupling needs to be trainable such that the network learns to achieve synchronization for similar inputs and stay desynchronized for different ones. We propose to apply the coupling by means of two new gates inside the GRU architecture which we call in the following CGRU (Coupled Gated Recurrent Unit). We chose to use GRU and not LSTM (Hochreiter & Schmidhuber (1997)) because the operation of a GRU is simpler (i.e. defined by fewer equations) while showing comparable performance in general (Chung et al. (2014)). The following equations describe the modifications brought to the architecture. Update, Reset and New gates are not modified. Let us notate h′t−1 and h̃ ′ t the states coming from the second sub-network (see Figure 1):
Hidden state: ht = (1− zt)h̄t + ztht−1 (8) Update: zt = σ(Wizxt + biz +Whzht−1 + bhz) (9)
Coupled New State: h̄t = (1− ct)h̃t + cth̃′t (10) Coupling: ct = σ(Whc(ht−1 + h′t−1) + bhc) (11)
New State: h̃t = tanh(Wih̃xt + bih̃ + rt(Whh̃ht−1 + bhh̃)) (12) Reset: rt = σ(Wirxt + bir +Whrht−1 + bhr). (13)
The Coupling gate ct (see Equation 11) serves the same purpose as zt and rt, controlling the information flow and is thus computed in a similar fashion, but only from the hidden states. This forces the model to apply the coupling on the new content to be added at time t solely based on the previous inputs. Then, in Equation 10, h̃t and h̃′t are combined similarly as h̃t and ht−1 are combined in the original GRU architecture. This prevents h̄t and subsequently ht from exploding and saturate the gates. Finally, in Equation 8, h̄t replaces h̃t: the New state has been replaced by a coupled version of both New states of the siamese GRU. Several possibilities exist to implement this coupling. The idea behind this proposal is to alter as little as possible the GRU architecture and to stay close to the original purpose of each equation. Indeed, the addition of the coupling already greatly modifies the information flow inside the GRU and the gradient flow during training (see the supplementary material for the complete differentiation of CGRU), and RNNs are known to be difficult to train. Therefore, by staying relatively close to the original model, a rigorous comparison is simpler, and the impact of the actual coupling can be studied more reliably. In fact, if ct is a vector of norm equal to zero, each sub-network is exactly a GRU. This suggests to initialize the coupling weights with very
small values and to accentuate the decay. In this way, an increase of the norm of Whc during training would signify that coupling is useful. Another interesting configuration of the coupling weights is when they are all equal to 0.5: in this configuration, the Coupled New States are the same, and the distance between the outputs will become null. That means, theoretically, this approach can make close any pair of input sequences, especially hard-positive samples.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP

text:
We experiment the CGRU architecture on the dataset UCI HAR2 (Anguita et al. (2013)), a dataset of 6 activities3 containing 9 features: total acceleration, body acceleration and angular velocity on 3 axes. The sequences have a length of 128. We chose this dataset because it has been extensively used by the activity recognition community and provides at the same time real data and a simple and well defined benchmark to study the behavior of CGRU and Siamese GRU (SGRU). Moreover, several activities should look very similar (e.g. three variants of walking or standing and sitting), and it should make the dataset harder to process for metric learning algorithms. Finally, walking or running exhibits dynamic components which could be differently processed by CGRU and SGRU. No further preprocessing has been applied. The features are globally centered and the standard deviations oscillate between 0.1 and 0.4. We kept the train-test split proposed by the authors of the dataset: there are 21 users in the training set and we therefore performed a 7-fold validation, leaving each time 3 different users out. Finally, the training set comprises 7352 sequences and the testing set 2947.
We compared SGRU and CGRU on learning the metric describes in Equation 5 using the same hyperparameters for both models. We used the same architecture for both networks, a one-layer network with 20 neurons and an initial learning rate of 0.001 which we decreased by a factor of 0.5 after 10 epochs if training loss does not improve. The training is stopped based on the accuracy on the validation set (early stopping). We chose to use structural loss (Yang et al. (2018)) to train the model since it is a recent loss working on distances and not embeddings. It combines a local term similarly to the n-pair loss (Sohn (2016)) but emphasizes the weights on the hard positive samples, and a global term to improve the generalization. We used the same hyperparameters for the loss as in the original paper. The full equations of the loss are available in the second section of the supplementary material. Regarding the batch size, each batch contains 6 elements of each class (total 36) to form a valid batch for structural loss. We applied a general weight decay with a factor of 10−4. A stronger weight decay was applied on the coupling by adding 1% of the coupling weight norm to the loss, which seems to slightly improve the performances. The gradient was clipped according to (Pascanu et al. (2013)) to a norm of 6. The coupling weights were initialized with a normal distribution having a mean of 0 and standard deviation of 0.1. The purpose of this initialization is to make the model start its training close to the behavior of an SGRU, with a very weak coupling and to let it increase during training. Our implementation was done in Python and CUDA with Pytorch (Paszke et al. (2019)).

heading: 4.2 RESULTS

heading: 4.2.1 STUDY OF THE COUPLING WEIGHT NORM

text:
We first analyze the evolution of the coupling weight norm during training. The coupling is initialized very low which theoretically makes it behave nearly as an SGRU at the beginning of training. We can observe in Figure 2a that the norm increases quickly during the first 20 epochs and more than doubles. Red points indicate the iterations where validation accuracy increased. This correlation thus indicates that the overall generalization is improving with the increase of coupling strength. In Figure 2b, we present the evolution of the loss on the train dataset in terms of the coupling weights on which we compute a linear regression of the first part. We can observe an almost linear relationship between the increase of the norm and the decrease of the loss suggesting again that the coupling is helpful to the model. Those figures also show that the convergence during training is rather smooth.
2https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+ using+smartphones
3walking, walking upstairs, walking downstairs, sitting, standing, laying

heading: 4.2.2 CLASSIFICATION PERFORMANCE ON UCI HAR

text: We now present classification results on UCI HAR using 3 metrics : accuracy, F1 score Macro averaged and Mean Average Precision (MAP), similarly to Su & Wu (2019). The first two are computed by classifying the test samples using 1-nearest neighbor from training samples. The MAP is computed by querying the training set with a test sequence to retrieve all training samples of the same class. The value is averaged for all test sequences. This metric shows the ability of the algorithm to bring close every sequence of each class and not just few references to be used as nearest neighbors. The validation results are presented in Table 1a. We observe an improvement of CGRU over SGRU of about 8% points for accuracy and F1-score, and an improvement of 19% points for the MAP. On the test set, we compared our approach with Regressive Virtual Sequence Metric Learning (RVSML) (Su & Wu (2019)), with OPW and DTW distances. We chose to compare with this approach because it recently outperformed several other metric learning approaches (e.g. Large Margin Nearest Neighbors (Weinberger & Saul (2009)), Regressive Virtual Metric Learning (Perrot & Habrard (2015)) etc.) although not all were specifically adapted for sequences, it is not based on neural networks and uses alignment-based distances. We keep the hyperparameter values recommended by the authors. The test results are presented in Table 1b. Here again we observe that CGRU outperformed SGRU with a notable improvement of 10% points of the accuracy and F1-score. Both neural network approaches clearly outperformed RVSML, especially the OPW variant. This can be, among other factors, attributed to a weak capacity to distinguish similar activities such as standing and sitting. On the other hand, standing was recognized perfectly by both RVSML variants. We also remark that they reach MAP values of the same order as in the original paper (around 0.4 ∼ 0.45) despite the fact that the data are of a completely different nature (signal instead of images). This could suggest that these approaches reached some kind of saturation whereas CGRU and SGRU are able to achieve much higher values.

heading: 4.2.3 PERFORMANCES ON HARD POSITIVE SAMPLES

text: We made the hypothesis that CGRU could perform better on hard positive samples due to coupling theoretically being able to bring close any input pair of sequences: with enough coupling, both networks can output the same sequence whatever of the input pair. We propose to verify this by observing the evolution of the average distance between the positive samples and between the negative samples when white noise is gradually added to the feature sequences of the testing set. The standard deviation of the noise goes from 0 to 2. We also note that both models were trained up to comparable validation accuracy. The results are presented on Figure 3. We observe that the curves for both models evolve similarly with positive distances gradually increasing with the noise. The negative and positive curves join the moment too much noise is added and the sequences become indistinguishable. CGRU produces slightly higher distance average than SGRU but is able to maintain higher margins more longer: up to 1.75 units of standard deviation compared to about 1.25 for SGRU. This shows that, as theoretically possible with the coupling, CGRU better discriminate the hard samples. The third section of the supplementary material extends this argumentation by analyzing average distance to nearest neighbors.

heading: 5 CONCLUSION AND PERSPECTIVES

text:
We presented a new framework for sequence metric learning based on dynamical system synchronization theory. We drew a parallel between synchronized trajectories and output sequences of siamese recurrent neural networks produced from similar input pairs. After characterizing Siamese GRU as identical chaotic systems, we showed the contribution of introducing coupling inside the siamese architecture to achieve synchronization more easily and to increase the capacity of the network to bring closer the embedding of some similar input pairs, especially hard positive samples. This coupling was implemented through a new gate inside the Siamese GRU architecture which allows the network to mix the new content of both sides of the siamese network. Our experiments showed that the Siamese GRU architecture benefits from the coupling, can be smoothly trained with it and fits well with recent complex metric learning losses such as structural loss. CGRU proved to be outperforming SGRU with the same architecture on an activity recognition benchmark and was able to maintain a higher margin for hard samples.
The study of sequence metric learning with synchronization opens several perspectives especially to design new forms of metrics by taking inspiration from the literature of synchronization criteria (Schiff et al. (1996); Kreuz et al. (2007)) though non-differentiability, notably when those metrics use mutual neighbors which are computed using explicitly the time steps, has to be overcome in many cases thanks to an attention mechanism, for example. The coupling itself could be tuned with theoretical contributions (Brown & Rulkov (1997)). One drawback of the proposed architecture is that each pair has to be passed through the network instead of just computing once each representation and then the distance for each pair. This could be balanced by the use of virtual metric learning during training. Finally the coupling allows to bring any pair of inputs close to one another if sufficiently
strong and could be use as an indicator in weakly supervised settings to invert the equivalence constraint of some pairs dynamically if the network is forcing the synchronization too much.


summaries: This work analyzes the effect of co-distillation for distributed training under moderate batch sizes. Using distillation-like techniques to improve synchronous SGD training is an interesting direction. And the paper carefully analyzed this setting while using the same amount of compute, which is not done by prior work to my knowledge. In addition, the writing is good and easy to follow.

summaries: The paper studies the concept of codistillation in data parallel distributed training. In this setting, the standard minibatch SGD algorithm requires exchange of models in every update of every node. Recent work in distributed training has studied "local SGD", where models are exchanged at frequent (usually periodic) intervals after a bunch of local updates. This paper studies an alternative, called "codistillation". The idea is that at a given node, say node $i$, the local model updates are regularized by the most recent models at nodes $\{j, j \neq i\}$ through an appropriately modified loss term. Specifically, the loss term bias the model at node $i$ towards having similar classification outcomes on the training data as the (most recent) local estimate of the model at nodes $\{j, j \neq i\}.

summaries: This paper aims to have a closer look at the role of codistillation for distributed training. Authors provided an answer with their empirical observations. That is, codistillation acts as a regularizer, since the distance between the learned model and the initialization is smaller than sync SGD without codistillation. Then, the authors claim that the codistillation may over-regularize and study how to modify the training configurations to avoid it. There are further discussions on the overfitting and robustness to hyper-parameters in sec 4 and sec 5.

abstractText: Codistillation has been proposed as a mechanism to share knowledge among concurrently trained models by encouraging them to represent the same function through an auxiliary loss. This contrasts with the more commonly used fully-synchronous data-parallel stochastic gradient descent methods, where different model replicas average their gradients (or parameters) at every iteration and thus maintain identical parameters. We investigate codistillation in a distributed training setup, complementing previous work which focused on extremely large batch sizes. Surprisingly, we find that even at moderate batch sizes, models trained with codistillation can perform as well as models trained with synchronous data-parallel methods, despite using a much weaker synchronization mechanism. These findings hold across a range of batch sizes and learning rate schedules, as well as different kinds of models and datasets. Obtaining this level of accuracy, however, requires properly accounting for the regularization effect of codistillation, which we highlight through several empirical observations. Overall, this work contributes to a better understanding of codistillation and how to best take advantage of it in a distributed computing environment.

authors:
Dan Alistarh
Demjan Grubic
Jerry Li
Ryota Tomioka
Milan Vojnovic

title: Qsgd: Communication-efficient sgd via gradient quantization and encoding

venue: In Advances in Neural Information Processing Systems,

authors:
Rohan Anil
Gabriel Pereyra
Alexandre Passos
Robert Ormandi
George E. Dahl
Geoffrey E. Hinton

title: Large scale distributed neural network training through online distillation

venue: In International Conference on Learning Representations (ICLR),

authors:
Mahmoud Assran
Nicolas Loizou
Nicolas Ballas
Mike Rabbat

title: Stochastic gradient push for distributed deep learning

venue: Proceedings of Machine Learning Research,

authors:
Jeremy Bernstein
Yu-Xiang Wang
Kamyar Azizzadenesheli
Anima Anandkumar

title: signsgd: Compressed optimisation for non-convex problems

venue: arXiv preprint arXiv:1802.04434,

authors:
Tom B Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell

title: Language models are few-shot learners

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Priya Goyal
Piotr Dollár
Ross Girshick
Pieter Noordhuis
Lukasz Wesolowski
Aapo Kyrola
Andrew Tulloch
Yangqing Jia
Kaiming He

title: Accurate, large minibatch sgd: Training imagenet in 1 hour

venue: arXiv preprint arXiv:1706.02677,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Tong He
Zhi Zhang
Hang Zhang
Zhongyue Zhang
Junyuan Xie
Mu Li

title: Bag of tricks for image classification with convolutional neural networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Yanping Huang
Youlong Cheng
Ankur Bapna
Orhan Firat
Dehao Chen
Mia Chen
HyoukJoong Lee
Jiquan Ngiam
Quoc V Le
Yonghui Wu

title: Gpipe: Efficient training of giant neural networks using pipeline parallelism

venue: In Advances in neural information processing systems,

authors:
Xianyan Jia
Shutao Song
Wei He
Yangzihao Wang
Haidong Rong
Feihu Zhou
Liqiang Xie
Zhenyu Guo
Yuanzhou Yang
Liwei Yu
Tiegang Chen
Guangxiao Hu
Shaohuai Shi
Xiaowen Chu

title: Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes

venue: arXiv preprint arXiv:1807.11205,

authors:
Tyler B. Johnson
Pulkit Agrawal
Haijie Gu
Carlos Guestrin

title: AdaScale SGD: A user-friendly algorithm for distributed training

authors:
Jared Kaplan
Sam McCandlish
Tom Henighan
Tom B Brown
Benjamin Chess
Rewon Child
Scott Gray
Alec Radford
Jeffrey Wu
Dario Amodei

title: Scaling laws for neural language models

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: In International Conference on Learning Representations,

authors:
Anastasia Koloskova
Tao Lin
Sebastian U Stich
Martin Jaggi

title: Decentralized deep learning with arbitrary communication compression

authors:
Dmitry Lepikhin
HyoukJoong Lee
Yuanzhong Xu
Dehao Chen
Orhan Firat
Yanping Huang
Maxim Krikun
Noam Shazeer
Zhifeng Chen

title: Gshard: Scaling giant models with conditional computation and automatic sharding

authors:
Mu Li
David G. Andersen
Alexander J Smola
Kai Yu

title: Communication efficient distributed machine learning with the parameter server

authors:
Xiangru Lian
Ce Zhang
Huan Zhang
Cho-Jui Hsieh
Wei Zhang
Ji Liu

title: Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent

authors:
Myle Ott
Sergey Edunov
David Grangier
Michael Auli

title: Scaling neural machine translation

venue: arXiv preprint arXiv:1806.00187,

authors:
Myle Ott
Sergey Edunov
Alexei Baevski
Angela Fan
Sam Gross
Nathan Ng
David Grangier
Michael Auli

title: fairseq: A fast, extensible toolkit for sequence modeling

venue: In Proceedings of NAACL-HLT 2019: Demonstrations,

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein

title: Imagenet large scale visual recognition challenge

venue: International Journal of Computer Vision,

authors:
Mohammad Shoeybi
Mostofa Patwary
Raul Puri
Patrick LeGresley
Jared Casper
Bryan Catanzaro

title: Megatron-LM: Training multi-billion parameter language models using GPU model parallelism

title: Local SGD converges fast and communicates little

venue: arXiv preprint arXiv:1805.09767,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Jianyu Wang
Gauri Joshi

title: Cooperative SGD: A unified framework for the design and analysis of communication-efficient SGD algorithms

venue: arXiv preprint arXiv:1808.07576,

authors:
Jianyu Wang
Vinayak Tantia
Nicolas Ballas
Michael Rabbat

title: SlowMo: Improving communication-efficient distributed SGD with slow momentum

authors:
Hao Yu
Sen Yang
Shenghuo Zhu

title: Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Ying Zhang
Tao Xiang
Timothy M. Hospedales
Huchuan Lu

title: Deep mutual learning

venue: In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),

heading: 1 INTRODUCTION

text:
Several recent improvements in the performance of machine learning models can be attributed to scaling the training of neural network models (He et al., 2016; Goyal et al., 2017; Vaswani et al., 2017; Devlin et al., 2018; Shoeybi et al., 2019; Huang et al., 2019; Kaplan et al., 2020; Lepikhin et al., 2020; Brown et al., 2020). Most approaches to scaling up training leverage some form of data parallelism (using multiple workers to compute gradients on different training samples in parallel), and the most common approach to data-parallel training is synchronous first-order optimization.
In synchronous data-parallel training, several replicas of a neural network model are created, each on a different worker. The workers process different mini-batches locally at each step using an optimizer such as Stochastic Gradient Descent (SGD) or Adam (Kingma & Ba, 2015), and the replicas synchronize (i.e., average either gradients or parameters) at every step by communicating either with a centralized parameter server (Li et al., 2014) or using all reduce (Goyal et al., 2017). More computing resources can be used in parallel by increasing the number of workers, effectively increasing the batch size used to compute a stochastic gradient. Increasing the batch size reduces the gradient’s variance and ideally makes it possible to increase the learning rate in proportion to the number of workers. By doing so, the number of steps required to reach a given model quality is also reduced in proportion to the number of workers, and a near-linear speedup is achieved (Goyal et al., 2017). However, it has been observed that the linear learning rate scaling strategy leads to performance degradation for very large batch sizes (Goyal et al., 2017), and even with more principled learning rate scaling mechanisms, synchronous SGD with larger batches eventually yields diminishing returns (Johnson et al., 2020).
Synchronous data-parallel methods ensure that all models are precisely synchronized at every step during training. This incurs substantial communication overhead, which increases with the number of replicas, and can quickly become a bottleneck limiting the processing units’ utilization (e.g., GPU or TPU), especially when devices communicate over commodity interconnects such as Ethernet. A number of approaches have been proposed to reduce communication overhead, including using mixed-precision (Jia et al., 2018) or other forms of compression (Alistarh et al., 2017; Bernstein
et al., 2018), reducing the frequency of synchronization to not occur after every optimizer step (Stich, 2018; Yu et al., 2019), using gossip-based methods for approximate distributed averaging (Lian et al., 2017; Assran et al., 2019; Wang et al., 2020), or using some combination thereof (Wang & Joshi, 2018; Koloskova et al., 2020).
Codistillation is an elegant alternative approach to distributed training with reduced communication overhead (Anil et al., 2018). Rather than synchronizing models to have the same weights, codistillation seeks to share information by having the models represent the same function (i.e., input-output mapping). Codistillation accomplishes this by incorporating a distillation-like loss that penalizes the predictions made by one model on a batch of training samples for deviating from the predictions made by other models on the same batch. In practice, a worker updating one model can compute the predictions made by another model by reading checkpoints of the other model and performing an additional forward pass. Previous work has demonstrated that codistillation is quite tolerant to asynchronous execution using stale checkpoints, e.g., using another model’s checkpoint from up to 50 updates ago without observing a significant drop in accuracy (Anil et al., 2018).
Anil et al. (2018) focuses on the very large batch setting. For example, when training a ResNet-50 on ImageNet, codistilling two models and using batch size 16k per model achieves substantially better performance than training with synchronous SGD and batch size 32k, although the final accuracy is still significantly lower than that achieved by synchronous SGD with a smaller batch size (e.g., 8k or smaller). This performance boost is attributed to an ensembling-like effect introduced by the codistillation loss.
In this paper we study codistillation at moderate batch sizes, i.e., before the performance of synchronous SGD begins to degrade. We demonstrate that it is possible to use codistillation in this regime without losing accuracy. For example, when training a ResNet-50 on ImageNet, we show that codistilling two models, each model using a batch size of 256, achieves comparable performance to training a single model using synchronous SGD with batch size 512. Furthermore, this holds across a range of batch sizes.
Achieving this performance parity involves modifying the way that explicit regularization is used in conjunction with codistillation. This modification stems from new insights into codistillation. Specifically, we demonstrate that codistillation has a regularizing effect. Thus, while increasing the batch size in synchronous SGD helps training by reducing the gradient variance, we conjecture that codistillation helps in a complementary manner via this regularization. Because it has a regularizing effect, care needs to be taken when using codistillation in conjunction with other forms of regularization, such as L2 regularization (weight decay), to avoid over-regularizing. We also evaluate the sensitivity of codistillation to different hyper-parameters like the frequency of reading new checkpoints and learning rate schedule.
Overall, our findings complement previous work on codistillation (Anil et al., 2018; Zhang et al., 2018). We summarize below our main contributions:
1. To the best of our knowledge, we demonstrate for the first time that models trained with codistillation can perform as well as models trained with traditional parallel SGD methods even when trained with the same number of workers and same number of updates, despite using a much weaker synchronization mechanism (Section 3). Previous work at the intersection of codistillation and distributed training used extremely large batch sizes and more workers than the parallel SGD counterparts.
2. Complementing the existing work on codistillation, we show that codistillation acts as a regularizer (Section 4). Our work demonstrates that explicitly accounting for its regularization effect is a key ingredient to using codistillation without losing accuracy (compared to parallel SGD methods).

heading: 2 CODISTILLATION: BACKGROUND AND RELATED WORK

text:
Codistillation is proposed as a mechanism for sharing information between multiple models being trained concurrently (Anil et al., 2018; Zhang et al., 2018). In typical multi-phase distillation, first, a teacher model is trained using standard supervised learning, and then a student model is trained to predict the outputs of the teacher model without any updating of the teacher. In contrast, when two
Algorithm 1: Codistillation Input: Loss function L(y, ŷ) and codistillation loss function D(y, y′) Input: Model architecture fθ(x) and initial model parameters {θ1i : i = 1, . . . , n} Input: Number of iterations K, learning rates {ηk}Kk=1, and penalty coefficients {αk}Kk=1
1 for k = 1, . . . ,K do 2 for i = 1, . . . , n do 3 x, y = get next minibatch()
4 θk+1i = θ k i − ηk∇θi ( L(y, fθki (x)) + α k 1 n−1 ∑ j 6=iD ( fθki (x), fθkj (x) )) 5 end 6 end
or more models codistill, there is only one phase, and in addition to minimizing the usual supervised loss on the training data, an additional loss term is used to share information between models by encouraging each model to make similar predictions to the other(s).
Codistillation, as described in Zhang et al. (2018), is shown in Algorithm 1. Here, n ≥ 2 models are trained concurrently. The ith model is updated on line 4 by taking a gradient step to minimize the combination of a standard supervised loss function L (e.g., cross-entropy or MSE) and a distillationlike loss D which penalizes differences between the predictions made by model i and those made by model j, averaged over all other models j 6= i. Zhang et al. (2018) and Anil et al. (2018) both report using Kullback-Liebler (KL) divergence for D in their experiments, although Anil et al. (2018) mentions that other options are possible (e.g., mean squared error between the logits of different models). Zhang et al. (2018) does not explicitly include a penalty parameter αk, and instead (implicitly) takes αk = 1 for all k.
Anil et al. (2018) focuses on codistillation as a mechanism for scaling distributed training, and demonstrates that using codistillation allows for better scaling with larger batch sizes than conventional synchronous parallel SGD. When different models reside on different GPUs, implementing line 4 requires that the devices communicate. Anil et al. (2018) proposes to have workers exchange parameter checkpoints, and these checkpoints are updated periodically rather than after every iteration. This leads to some delay, i.e., actually using θk ′
j with k ′ < k in line 4. However, it is argued
that such delay is tolerable for codistillation because large changes in parameters θ do not necessarily correspond to large changes to the function fθ. Experimentally, it is reported that exchanging checkpoints every 50 updates does not noticeably degrade performance in their setup. Anil et al. (2018) also reports using a burn-in period since, “in the beginning of training, the distillation term in the loss is not very useful or may even be counterproductive.” During the burn-in period of the first K̂ K steps, the models train independently; i.e., αk = 0 for k = 1, . . . , K̂. Below we find that codistillation can tolerate large delays (e.g., updating checkpoints after hundreds or thousands of updates) without any burn-in period and without any significant loss in accuracy.
One common application of distillation is to train a single model to mimic an ensemble’s predictions to avoid the computational overhead of deploying an ensemble. Motivated by this, previous works (Anil et al., 2018; Zhang et al., 2018) explore the connection between codistillation and ensemble methods, showing that two models trained using codistillation achieve performance close to that of ensembling two independently-trained models. Anil et al. (2018) also conducts an experiment that compares codistilling two models with simply using label smoothing as a regularizer when training one model, and reports that codistillation performs significantly better.

heading: 3 CODISTILLATION, SYNCHRONOUS SGD, AND REGULARIZATION

text:
The aim of this section is to demonstrate that codistillation can be used as a mechanism for distributed training without losing accuracy, even when used in conjunction with batch sizes that are not particularly large. To achieve this, we also show that regularization needs to be handled carefully. Typical training regimes use some form of regularization to avoid overfitting. For example, the common practice is to use L2 regularization (weight decay) when training convolutional networks (Goyal et al., 2017) and to use label smoothing when training neural machine translation
models (Ott et al., 2018). Directly incorporating codistillation on top of existing training pipelines can lead to underfitting. We begin this section by illustrating this point.

heading: 3.1 CODISTILLATION “OUT OF THE BOX” CAN OVER-REGULARIZE

text:
We train a ResNet50 (He et al., 2016) model to perform image classification on the ImageNet dataset (Russakovsky et al., 2015). To begin, we compare training a single synchronous SGD model using batch size 512 with codistilling two models using batch size 256 per model. In both cases we perform training using 16 GPUs and batch size 32 per GPU. Similar to Anil et al. (2018), we follow the training procedure proposed in Goyal et al. (2017) for the learning rate warmup, learning rate schedule, L2 regularization, and other hyperparameters. Specifically, synchronous SGD training (all reduce) is run for 90 epochs. The learning rate starts at 0.1, regardless of batch size, and is warmed up over the first five epochs to 0.1 × b/256, where b is the effective batch size across all workers (e.g., 512 when using synchronous SGD and 16 GPUs). The learning rate is then decreased by a factor of 0.1 at the beginning of epochs 30, 60, and 80.
When training with codistillation, workers read the checkpoints of the other model once every 2500 updates. At this frequency of checkpoint reading, we did not observe any benefit to using a burn-in period. We experimented with reading checkpoints more frequently and also did not observe any improvement in performance. When training with two models and batch size 256, we use the same total number of GPUs (16) as an all reduce model with batch size 512, as well as the same number of gradient updates per GPU (with each update processing a mini-batch of size 32). Hence, when training using codistillation we only train the two models for 45 epochs (so that the number of updates per codistilled model is the same as the number of updates performed by all reduce using the same total number of GPUs) and scale all milestones in the learning rate schedule by half, i.e.., warming up over the first 2.5 epochs, and decreasing at epochs 15, 30, and 40.
In Fig. 1(a) and 1(b), we observe that the model trained with codistillation achieves higher training loss and lower top-1 accuracy (around 2% lower) compared to the model trained with synchronous SGD. The codistillation performance is calculated using one of the two models (model 1) being codistilled. This suggests that although the codistillation loss may provide a useful training signal, without other modifications it is not sufficient to achieve the same accuracy as synchronous SGD using the same number of GPUs.
Since both training loss and validation accuracy are affected, we hypothesize that the problem stems from over-regularization. To further investigate this we plot the change in model weights, relative to their initialization, over the course of training in Fig. 1(c). We observe that parameters do not move as far in parameter space when training with codistillation as they do when using synchronous SGD, providing further evidence that codistillation acts as a regularizer.

heading: 3.2 ACCOUNTING FOR CODISTILLATION’S REGULARIZATION EFFECT BRIDGES THE GAP

text:
Motivated by the observations above, we conduct an experiment using less explicit regularization when training with codistillation. Goyal et al. (2017) recommends using a constant L2 regularization of 10−4 throughout training. We propose to start with the same initial value (10−4), reduce it to 10−5 after the first learning rate decay, and reduce it further to 0 after the second learning rate decay. When training with codistillation, we observe that the model’s training loss saturates slower and we shift the learning rate schedule by a few epochs to account for this (from 15, 30, 40 to 18, 38, 44).
In Fig. 2(a), we compare the top-1 validation accuracy on ImageNet for the ResNet50 model trained with codistillation and all reduce, with the above-mentioned modifications to the codistillation setup. Accounting for codistillation’s regularization effect helps narrow the performance gap, with all reduce now performing only marginally better (76.1% for all reduce vs 75.9% for codistillation). We verify that with a constant L2 regularization of 10−4 throughout training, codistillation performs much worse than all reduce, implying that the improved performance is not only from the modified learning rate schedule; see Fig. 12 in the Appendix. To validate that the improvement is not specific to the ResNet50 architecture, we repeat the experiment using the larger ResNeXt101, ResNet152 (Fig. 9 in Appendix) and ResNext152 (Fig. 10 in Appendix) architectures. In Fig. 2(b), we observe that codistillation performs marginally better than all reduce. We believe that the larger ResNeXt101 may benefit more from codistillation due to having more capacity than the ResNet50. The training loss curves for both models are shown in Fig. 8 in the Appendix.
In synchronous SGD (all reduce), more workers can be added to increase the effective batch size (summed across all workers). This reduces the gradient’s variance and the model can be trained with a larger learning rate and fewer steps, while maintaining a similar level of accuracy (Goyal et al., 2017). In Fig. 3 and Table A.2, we demonstrate a similar effect with codistillation. As one doubles the batch size per worker (and hence doubles the effective batch size), the learning rate can also be doubled and the model reaches similar performance in half the number of steps.
The experiments above use the step-wise learning rate schedule described in Goyal et al. (2017). We want to ascertain that our findings are not dependent on this specific learning rate schedule. Hence we train the ResNet50 and ResNeXt101 models with the cosine learning rate schedule (He et al., 2019). In Fig. 4, we observe that the final validation performance for the two approaches is very close, confirming that codistillation works consistently across different learning rate schedules. The corresponding training loss plots are shown in Fig. 11 in the Appendix.
So far, all our experiments have focused on the ImageNet dataset and the ResNet family of models. Next, we evaluate the codistillation mechanism for neural machine translation (NMT). Specifically, we train the “big” transformer model (Vaswani et al., 2017) (6 blocks in the encoder and decoder networks) on the WMT’16 En-De translation dataset, following the setup described in Ott et al. (2018). We note that for codistillation, reducing the explicit regularization is again important for achieving performance comparable to all reduce (see details in Appendix, Section A.1).
This observation is in line with our previous observations on convolutional models. In Fig. 5, we observe that the model trained with codistillation reaches a similar validation loss as the model trained with all reduce. This confirms that codistillation with adjusted regularization schedule also extends to NMT. The corresponding training plot is shown in Fig. 14 in the Appendix.
There are several important aspects of the above results that are worth emphasizing. (i) Unlike prior work on codistillation, we train both setups with the same number of workers for the same number of steps. We believe this constraint to be particularly important to better understand the impact of codistillation on many practical distributed training workflows. (ii) Our objective is not to show that codistillation can outperform all reduce, and our initial
expectation was actually that replacing all reduce by codistillation would degrade the performance when using the same number of updates (as suggested in Fig. 1), since codistillation is a much weaker form of synchronization. With these experiments, we want to quantify this performance loss and understand how we can minimize it. Interestingly, we find that the performance gap can be narrowed down significantly, with codistillation even outperforming all reduce in some settings (Fig. 4(b)). (iii) We believe that previous works did not investigate the regularization effect of codistillation because they focused on the ensembling effect obtained by increasing the number of workers. Being aware of the regularization effect and explicitly accounting for it is important when using codistillation in practice, as seen in Fig. 2.

heading: 4 CODISTILLATION HELPS REDUCE OVERFITTING

text:
Section 3 illustrated that applying codistillation in addition to other forms of regularization can result in over-regularizing, and progressively reducing the explicit regularization helps improve training and generalization. This section further explores the regularizing effect of codistillation by examining settings where we expect the model to overfit.
We simulate a scenario where the ResNet50 model is likely to overfit in a controlled way by training the model using only a fraction of the full ImageNet training set. When training on (1/k)th of the training set, we multiply the number of epochs by k so that the total number of model updates performed is the same as when training with the full training set. The learning rate and weight decay schedules are also modified accordingly. As we train on less data, we expect the model to overfit, i.e., to obtain lower training loss and lower validation accuracy.
In Figure 6, we indeed observe that overfitting occurs when using less training data. However, the overfitting is less severe when training using codistillation, providing further support for the hypothesis that codistillation acts as a regularizer.
An interesting side-effect of this observation is that codistillation could be an interesting alternative to all reduce for training over-parameterized models. We further verify this hypothesis by training a “big” Transformer model for a small NMT dataset (IWSLT 2014 German-English translation dataset that contains 153K training sentences, 7K development sentences, and 7K test sentences). The model trained with codistillation achieves a validation NLL of 2.31 whereas the model trained with all reduce reaches a validation NLL of 2.37.

heading: 5 ROBUSTNESS TO UPDATE FREQUENCY AND LEARNING RATE

text:
To further understand the codistillation setup, we conduct additional ablations examining the effect of some hyper-parameter choices.
In distributed training, model weights are exchanged regularly between the training workers. In synchronous SGD, model parameters are communicated every gradient update, via all reduce, which introduces the large communication overhead. In codistillation, the frequency of reading model checkpoints should be treated as a hyper-parameter and should be tuned for the specific use cases. Zhang et al. (2018) always uses the most recent version of the model for computing the codistillation loss, equivalent to reading a new checkpoint at every iteration. Anil et al. (2018) reports exchanging checkpoints every 50 updates.
In the ImageNet results presented in Sections 3 and 4, we exchange the model checkpoints once every 2500 updates. In Fig. 7(a), we evaluate how the validation performance for the ResNet models varies with the frequency of exchanging the checkpoints). The curve corresponding to 1× is the baseline setup, with checkpoints exchanged every 2500 updates. Label 2× denotes the case where checkpoints are exchanged twice as frequently (every 1250 updates), and other curves similarly show 0.5× and 4×. The corresponding training loss curves are shown in Fig. 15(a) in the Appendix. We observe that the setup is quite robust to the frequency of exchange. Although there are some small improvements by using a higher frequency, the results do not degrade too much with a lower frequency. Similar trends hold for the Transformer model (for language translation task), as shown in Fig. 16(a) in the Appendix.
In Fig. 7(b), we vary the base learning rate when training the codistillation model on the ImageNet dataset, using the setup proposed in Goyal et al. (2017) and report the accuracy on the validation dataset. We observe that increasing the learning rate from 0.1 to 0.2 has almost no impact on the validation accuracy. Beyond that, higher learning rates hurt the validation accuracy more. The corresponding plot with the training loss is shown in Fig. 15(b) in the Appendix.

heading: 6 CONCLUSION

text: In this work, we demonstrate for the first time (to the best of our knowledge), that even for moderate batch sizes, models trained with codistillation can perform as well as models trained with traditional parallel SGD methods (while using comparable computational resources). Complementing existing works, we show that codistillation acts as a regularizer, and that accounting for this property is essential when using codistillation. There are several exciting and potentially impactful directions for extending our understanding of codistillation. One important direction is to codistill more than two models. The loosely synchronized nature of codistillation potentially allows using different topologies for determining which models codistill together. Zhang et al. (2018) does consider one version of the setup, where all models codistill with each other, but they do so in a non-distributed setting. Another possible future work will be to consider codistillation between models that are different, e.g., in terms of architecture, capacity, or hyper-parameters, and evaluate if that improves performance over the standard knowledge distillation based approaches.

heading: A APPENDIX

text:
A.1 IMPLEMENTATION DETAILS
The code for all experiments is implemented using PyTorch 1.5 (Paszke et al., 2017). All experiments are run for 3 seeds and plots report the average over all seeds. The codistillation loss is the mean squared error between the logits of the two models (without centering them, as we found in preliminary experiments that centering yielded similar results).
The experiments involving the ImageNet dataset are based on the setup proposed in Goyal et al. (2017). We consider two schedules to control the value of the codistillation loss coefficient αk – the constant schedule, αk = 1 for all k, and the step-wise schedule where we increase αk every time we decrease the learning rate. We did not find any significant difference in the performance of the two cases and report the performance for the case where αk increases by 0.5 every time we decrease the learning rate.
The experiments for translation tasks use the FairSeq library (Ott et al., 2019). These experiments are based on the setup proposed in Ott et al. (2018). We consider two schedules to control the value of αk - the constant schedule and the exponential schedule where we increase αk by a factor γ = 1.1 every epoch. We find in Fig. 17 that the exponential schedule works better than the constant alpha schedule (2.12 vs 2.15 validation perplexity), and this is the schedule we use in all other experiments on this task. When training the translation models with codistillation, we explicitly reduce the amount of regularization (to account for regularization added by codistillation) by removing the label smoothing loss.
A.2 ADDITIONAL RESULTS ON IMAGENET
In Fig. 8, we plot the training loss for the ResNet50 (left) and ResNeXt101 (right) models for the ImageNet dataset with some variations to the setup proposed in Goyal et al. (2017), as in Section 3.2. Specifically, Goyal et al. (2017) recommends using a constant L2 weight decay set to 10−4 throughout training. Keeping the initial value of this weight decay to 10−4, we reduce it to 10−5 after the first learning rate decay and further to 0 after the second learning rate decay. We also modify the learning rate schedule from Goyal et al. (2017), which is based on how the training loss changes (and saturates) during training. Due to the regularization effect of codistillation, we observe that the model’s training loss saturates slower and we shift the schedule by a few epochs to account for this (from 15, 30, 40 to 18, 38, 44). We note that for both models, the all reduce setup reaches
a lower training loss. Despite this, the performance on the validation dataset (in terms of top-1 validation accuracy) is very similar for the two setups (for both models) as shown in Fig. 2.
In Fig. 11, we show the training performance for ResNet50 and ResNeXt101 models when using the cosine learning rate schedule (He et al., 2019). Again, while the all reduce setup reaches a lower training loss, the performance on the validation dataset (in terms of top-1 validation accuracy) is very similar for the two setups (for both models) as shown in Fig. 4.
In Fig. 12 we verify that the good performance observed in Fig. 2 is not due only to the shifted learning rate schedule. We observe that, indeed, with this shifted schedule, if we keep the L2 weight decay to a constant value of 10−4 throughout training, then codistillation is significantly worse than all reduce both in terms of training loss and validation accuracy.
In synchronous SGD (all reduce), more workers can be added to increase the effective batch size (summed across all workers). The increased batch size reduces the gradient’s variance and the
model can be trained with a larger learning rate and fewer steps, while maintaining a similar level of accuracy (Goyal et al., 2017). In Fig. 3 and Table A.2, we demonstrate a similar effect with codistillation. As one doubles the batch size per worker (and hence doubles the effective batch size), the learning rate can also be doubled and the model reaches similar performance in half the number of steps.
Batch Size Training Loss Validation Accuracy 2× 128 1.12 75.61 2× 256 1.08 75.92 2× 512 1.12 75.51 2× 1024 1.13 75.45 2× 2048 1.14 75.26
A.3 EFFECT OF VARYING α
In Fig 13, we evaluate codistillation setup by varying the value of αk. Note that the value is not changed during training and hence αk is denoted by α.
A.4 ADDITIONAL RESULTS ON WMT’16 EN-DE
In Fig. 14 we plot the training loss of the all reduce and codistillation setups on the WMT’16 En-De dataset. We observe that codistillation has a higher training loss, but almost matches the validation loss of all reduce as shown in Fig. 5.
A.5 ADDITIONAL RESULTS ON CODISTILLATION’S ROBUSTNESS
To further understand the codistillation setup, we conduct additional ablations examining the effect of some hyper-parameter choices.
In Fig. 15(a), we compare the training performance of ResNet50 models trained on the ImageNet dataset, as we vary the frequency of exchanging the checkpoints. The curve corresponding to 1× is the baseline setup, with checkpoints exchanged every 2500 updates. Label 2× denotes the case where checkpoints are exchanged twice as frequently (every 1250 updates), and other curves similarly show 0.5× and 4×. The training loss is robust to changes to the frequency, reaching very similar values across the whole range. The corresponding top-1 validation accuracy is shown in Fig. 7(a).
In Fig. 15(b), we vary the base learning rate when training the codistillation model on the ImageNet dataset, using the setup proposed in Goyal et al. (2017) and report the loss on the training dataset.
We observe that increasing the learning rate from 0.1 to 0.2 has almost no impact on the training loss. Beyond that, higher learning rates affect the training loss more. The corresponding plot with the top-1 validation accuracy is shown in Fig. 7(b) and exhibits a similar trend.
In Fig. 16, we vary the frequency of exchanging the model checkpoints for the Transformer model. We note that more frequent updates lead to a higher validation loss initially, but as training continues, this setup eventually outperforms (slightly) the one with less frequent updates. Overall, the model’s performance does not decline drastically even when we exchange checkpoints 2.5 times less often.
In Fig. 17, both the initial value of α (codistillation loss coefficient) and γ (scaling factor for α) are varied when training the Transformer model. Since we are using a geometric progression for increasing the value of α, changing γ alone can increase α to very large values, thus making training unstable. Hence, for each γ ∈ {1, 1.1, 1.5} we vary the initial α to achieve optimal results. The case of γ = 1 corresponds to keeping α constant. This case slightly under-performs as compared to the cases where we vary α.