summaries: This paper proposes a self-supervised encoder-discriminator based framework for embedding the multivariate time series into a compact fixed dimensional representation. The approach dubbed Temporal Neighborhood Coding (TNC) leverages the concept of a neighborhood in time (with stationary properties), and learns time series representations by ensuring the distribution of neighboring signals is distinguishable from the distribution of non-neighboring signals, in the encoding space. Empirical evidence is provided that such embedding of time series results in clusters of higher quality, as well that use of such obtained representations for supervised tasks outperforms few competitor (unsupervised) approaches.

summaries: The authors propose a novel unsupervised encoding scheme for time series. Utilizing a statistical test for non-stationarity, the authors derive a Temporal Neighborhood Coding (TNC) scheme and combine it with ideas from Positive-Unlabeled (PU) learning to learn informative hidden representations of time series windows. The representations are evaluated in terms of how well they can be clustered and how much they influence classification performance on three data sets. The supreme performance was demonstrated when comparing to the state of the art methods and a $k$NN (for classification) baseline. Furthermore, the authors illustrate how the learnt representations remain interpretable as long as the encoding network is reasonably small.

summaries: The paper proposes an unsupervised representation (embedding) learning method for time-series. While unsupervised representation learning has been extensively studied and shown good performance in fields like NLP and vision, it is relatively new to the time-series community. This paper, in contrast to recent work (CPC and Triplet-Loss), has the following differences:

abstractText: Time series are often complex and rich in information but sparsely labeled and therefore challenging to model. In this paper, we propose a self-supervised framework for learning generalizable representations for non-stationary time series. Our approach, called Temporal Neighborhood Coding (TNC), takes advantage of the local smoothness of a signal’s generative process to define neighborhoods in time with stationary properties. Using a debiased contrastive objective, our framework learns time series representations by ensuring that in the encoding space, the distribution of signals from within a neighborhood is distinguishable from the distribution of non-neighboring signals. Our motivation stems from the medical field, where the ability to model the dynamic nature of time series data is especially valuable for identifying, tracking, and predicting the underlying patients’ latent states in settings where labeling data is practically impossible. We compare our method to recently developed unsupervised representation learning approaches and demonstrate superior performance on clustering and classification tasks for multiple datasets.

authors:
Ahmed Alaa
Mihaela van der Schaar

title: Attentive state-space modeling of disease progression

authors:
Shahin Amiriparian
Michael Freitag
Nicholas Cummins
Björn Schuller

title: Sequence to sequence autoencoders for unsupervised representation learning from audio

venue: In Proc. of the DCASE

authors:
Shaojie Bai
J Zico Kolter
Vladlen Koltun

title: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling

venue: arXiv preprint arXiv:1803.01271,

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Edward Choi
Mohammad Taha Bahadori
Elizabeth Searles
Catherine Coffey
Michael Thompson
James Bost
Javier Tejedor-Sojo
Jimeng Sun

title: Multi-layer representation learning for medical concepts

venue: In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,

authors:
Edward Choi
Andy Schuetz
Walter F Stewart
Jimeng Sun

title: Medical concept representation learning from electronic health records and its application on heart failure prediction

venue: arXiv preprint arXiv:1602.03686,

authors:
Youngduck Choi
Chill Yi-I Chiu
David Sontag

title: Learning low-dimensional representations of medical concepts

venue: AMIA Summits on Translational Science Proceedings,

authors:
Ching-Yao Chuang
Joshua Robinson
Lin Yen-Chen
Antonio Torralba
Stefanie Jegelka

title: Debiased contrastive learning

venue: arXiv preprint arXiv:2007.00224,

title: Unsupervised learning of disentangled representations from video

venue: In Advances in Neural Information Processing Systems,

authors:
Amir Dezfouli
Hassan Ashtiani
Omar Ghattas
Richard Nock
Peter Dayan
Cheng Soon Ong

title: Disentangled behavioural representations

authors:
Jeff Donahue
Karen Simonyan

title: Large scale adversarial representation learning

authors:
Marthinus C Du Plessis
Gang Niu
Masashi Sugiyama

title: Analysis of learning from positive and unlabeled data

authors:
Marthinus Christoffel Du Plessis
Masashi Sugiyama

title: Class prior estimation from positive and unlabeled data

venue: IEICE TRANSACTIONS on Information and Systems,

authors:
Charles Elkan
Keith Noto

title: Learning classifiers from only positive and unlabeled data

venue: In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp

authors:
Hassan Ismail Fawaz
Germain Forestier
Jonathan Weber
Lhassane Idoumghar
Pierre-Alain Muller

title: Deep learning for time series classification: a review

venue: Data Mining and Knowledge Discovery,

authors:
Jean-Yves Franceschi
Aymeric Dieuleveut
Martin Jaggi

title: Unsupervised scalable representation learning for multivariate time series

authors:
Michael U Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics

venue: Journal of Machine Learning Research,

authors:
Aapo Hyvarinen
Hiroshi Morioka

title: Unsupervised feature extraction by time-contrastive learning and nonlinear ICA

authors:
Ryuichi Kiryo
Gang Niu
Marthinus C Du Plessis
Masashi Sugiyama

title: Positive-unlabeled learning with non-negative risk estimator

authors:
Martin Längkvist
Lars Karlsson
Amy Loutfi

title: A review of unsupervised feature learning and deep learning for time-series modeling

venue: Pattern Recognition Letters,

authors:
Thomas A Lasko
Joshua C Denny
Mia A Levy

title: Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data

venue: PloS one,

authors:
Wee Sun Lee
Bing Liu

title: Learning with positive and unlabeled examples using weighted logistic regression

venue: In International Conference on Machine Learning,

authors:
Qi Lei
Jinfeng Yi
Roman Vaculin
Lingfei Wu
Inderjit S Dhillon

title: Similarity preserving representation learning for time series clustering

venue: In IJCAI,

authors:
Xiaoli Li
Bing Liu

title: Learning to classify texts using positive and unlabeled data

authors:
Sindy Löwe
Peter O’Connor
Bastiaan Veeling

title: Putting an end to end-to-end: Gradient-isolated learning of representations

authors:
Xinrui Lyu
Matthias Hueser
Stephanie L Hyland
George Zerveas
Gunnar Rätsch

title: Improving clinical predictions through unsupervised time series representation learning

venue: arXiv preprint arXiv:1812.00490,

authors:
Qianli Ma
Jiawei Zheng
Sen Li
Gary W Cottrell

title: Learning representations for time series clustering

authors:
Pankaj Malhotra
Vishnu TV
Lovekesh Vig
Puneet Agarwal
Gautam Shroff

title: Timenet: Pre-trained deep recurrent neural network for time series classification

venue: arXiv preprint arXiv:1706.08838,

authors:
Tomas Mikolov
Kai Chen
Greg Corrado
Jeffrey Dean

title: Efficient estimation of word representations in vector space

venue: arXiv preprint arXiv:1301.3781,

title: A new method for detecting atrial fibrillation using RR intervals

venue: Computers in Cardiology,

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

title: A comparison between the silhouette index and the davies-bouldin index in labelling ids clusters

venue: In Proceedings of the 11th Nordic Workshop of Secure IT Systems,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

venue: arXiv preprint arXiv:1511.06434,

authors:
Alec Radford
Rafal Jozefowicz
Ilya Sutskever

title: Learning to generate reviews and discovering sentiment

venue: arXiv preprint arXiv:1704.01444,

authors:
Nikunj Saunshi
Orestis Plevrakis
Sanjeev Arora
Mikhail Khodak
Hrishikesh Khandeparkar

title: A theoretical analysis of contrastive unsupervised representation learning

authors:
Peter Schulam
Fredrick Wigley
Suchi Saria

title: Clustering longitudinal clinical marker trajectories from electronic health data: Applications to phenotyping and endotype discovery

venue: In Twenty-Ninth AAAI Conference on Artificial Intelligence,

authors:
Harini Suresh
Jen J Gong
John V Guttag

title: Learning tasks for multitask learning: Heterogenous patient populations in the icu

venue: In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,

authors:
Sana Tonekaboni
Shalmali Joshi
Kieran Campbell
David K Duvenaud
Anna Goldenberg

title: What went wrong and when? instance-wise feature importance for time-series black-box models

venue: Advances in Neural Information Processing Systems,

authors:
Xiang Wang
David Sontag
Fei Wang

title: Unsupervised learning of disease progression models

venue: In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp

authors:
Xiaolong Wang
Abhinav Gupta

title: Unsupervised learning of visual representations using videos

venue: In Proceedings of the IEEE International Conference on Computer Vision, pp

authors:
Qiang Yang
Xindong Wu

title: 10 challenging problems in data mining research

venue: International Journal of Information Technology & Decision Making,

authors:
Tom Young
Devamanyu Hazarika
Soujanya Poria
Erik Cambria

title: Recent trends in deep learning based natural language processing

venue: IEEE Computational Intelligence Magazine,

heading: 1 INTRODUCTION

text:
Real-world time-series data is high dimensional, complex, and has unique properties that bring about many challenges for data modeling (Yang & Wu, 2006). In addition, these signals are often sparsely labeled, making it even more challenging for supervised learning tasks. Unsupervised representation learning can extract informative low-dimensional representations from raw time series by leveraging the data’s inherent structure, without the need for explicit supervision. These representations are more generalizable and robust, as they are less specialized for solving a single supervised task. Unsupervised representation learning is well studied in domains such as vision (Donahue & Simonyan, 2019; Denton et al., 2017; Radford et al., 2015) and natural language processing (Radford et al., 2017; Young et al., 2018; Mikolov et al., 2013), but has been underexplored in the literature for time series settings. Frameworks designed for time series need to be efficient and scalable because signals encountered in practice can be long, high dimensional, and high frequency. Moreover, it should account for and be able to model dynamic changes that occur within samples, i.e., non-stationarity of signals.
The ability to model the dynamic nature of time series data is especially valuable in medicine. Health care data is often organized as a time series, with multiple data types, collected from various sources at different sampling frequencies, and riddled with artifacts and missing values. Throughout their stay at the hospital or within the disease progression period, patients transition gradually between distinct
∗http://www.cs.toronto.edu/~stonekaboni/
clinical states, with periods of relative stability, improvement, or unexpected deterioration, requiring escalation of care that alters the patient’s trajectory. A particular challenge in medical time-series data is the lack of well-defined or available labels that are needed for identifying the underlying clinical state of an individual or for training models aimed at extracting low-dimensional representations of these states. For instance, in the context of critical-care, a patient’s stay in the critical care unit (CCU) is captured continuously via streaming physiological signals by the bedside monitor. Obtaining labels for the patient’s state for extended periods of these signals is practically impossible as the underlying physiological state can be unknown even to the clinicians. This further motivates the use of unsupervised representation learning in these contexts. Learning rich representations can be crucial in facilitating the tracking of disease progression, predicting the future trajectories of the patients, and tailoring treatments to these underlying states.
In this paper, we propose a self-supervised framework for learning representations for complex multivariate non-stationary time series. This approach, called Temporal Neighborhood Coding (TNC), is designed for temporal settings where the latent distribution of the signals changes over time, and it aims to capture the progression of the underlying temporal dynamics. TNC is efficient, easily scalable to high dimensions, and can be used in different time series settings. We assess the quality of the learned representations on multiple datasets and show that the representations are general and transferable to many downstream tasks such as classification and clustering. We further demonstrate that our method outperforms existing approaches for unsupervised representation learning, and it even performs closely to supervised techniques in classification tasks. The contributions of this work are three-fold:
1. We present a novel neighborhood-based unsupervised learning framework for non-stationary multivariate time series data.
2. We introduce the concept of a temporal neighborhood with stationary properties as the distribution of similar windows in time. The neighborhood boundaries are determined automatically using the properties of the signal and statistical testing.
3. We incorporate concepts from Positive Unlabeled Learning, specifically, sample weight adjustment, to account for potential bias introduced in sampling negative examples for the contrastive loss.

heading: 2 METHOD

text:
We introduce a framework for learning representations that encode the underlying state of a multivariate, non-stationary time series. Our self-supervised approach, TNC, takes advantage of the local smoothness of the generative process of signals to learn generalizable representations for windows of time series. This is done by ensuring that in the representation space, the distribution of signals proximal in time is distinguishable from the distribution of signals far away, i.e., proximity in time is identifiable in the encoding space. We represent our multivariate time series signals as X ∈ RD×T , where D is the number of features and T is the number of measurements over time. X[t− δ2 ,t+ δ2 ] represents a window of time series of length δ, centered around time t, that includes measurements of all features taken in the interval [t− δ2 , t+ δ 2 ]. Throughout the paper, we refer to this window as Wt for notational simplicity. Our goal is to learn the underlying representation of Wt, and by sliding this window over time, we can obtain the trajectory of the underlying states of the signal.
We define the temporal neighborhood (Nt) of a window Wt as the set of all windows with centroids t∗, sampled from a normal distribution t∗ ∼ N (t, η · δ). Where N is a Gaussian centered at t, δ is the size of the window, and η is the parameter that defines the range of the neighborhood. Relying on the local smoothness of a signal’s generative process, the neighborhood distribution is characterized as a Gaussian to model the gradual transition in temporal data, and intuitively, it approximates the distribution of samples that are similar to Wt. The η parameter determines the neighborhood range and depends on the signal characteristics and how gradual the time series’s statistical properties change over time. This can be set by domain experts based on prior knowledge of the signal behavior, or for more robust estimation, it can be determined by analyzing the stationarity properties of the signal for every Wt. Since the neighborhood represents similar samples, the range should identify the approximate time span within which the signal remains stationary, and the generative process does not change. For this purpose, we use the Augmented Dickey-Fuller (ADF) statistical test to
determine this region for every window. Proper estimation of the neighborhood range is an integral part of the TNC framework. If η is too small, many samples from within a neighborhood will overlap, and therefore the encoder would only learn to encode the overlapping information. On the other hand, if η is too big, the neighborhood would span over multiple underlying states, and therefore the encoder would fail to distinguish the variation among these states. Using the ADF test, we can automatically adjust the neighborhood for every window based on the signal behavior. More details on this test and how it is used to estimate η is described in section 2.
Now, assuming windows within a neighborhood possess similar properties, signals outside of this neighborhood, denoted as N̄t, are considered non-neighboring windows. Samples from N̄t are likely to be different from Wt, and can be considered as negative samples in a context of a contrastive learning framework. However, this assumption can suffer from the problem of sampling bias, common in most contrastive learning approaches (Chuang et al., 2020; Saunshi et al., 2019). This bias occurs because randomly drawing negative examples from the data distribution may result in negative samples that are actually similar to the reference. This can significantly impact the learning framework’s performance, but little work has been done on addressing this issue (Chuang et al., 2020). In our context, this can happen when there are windows from N̄t that are far away from Wt, but have the same underlying state. To alleviate this bias in the TNC framework, we consider samples from N̄t as unlabeled samples, as opposed to negative, and use ideas from Positive-Unlabeled (PU) learning to accurately measure the loss function. In reality, even though samples within a neighborhood are all similar, we cannot make the assumption that samples outside this region are necessarily different. For instance, in the presence of long term seasonalities, signals can exhibit similar properties at distant times. In a healthcare context, this can be like a stable patient that undergoes a critical condition, but returns back to a stable state afterwards.
In PU learning, a classifier is learned using labeled data drawn from the positive class (P ) and unlabeled data (U ) that is a mixture of positive and negative samples with a positive class prior π (Du Plessis et al., 2014; Kiryo et al., 2017; Du Plessis & Sugiyama, 2014). Existing PU learning methods fall under two categories based on how they handle the unlabeled data: 1) methods that identify negative samples from the unlabeled cohort (Li & Liu, 2003); 2) methods that treat the unlabeled data as negative samples with smaller weights (Lee & Liu, 2003; Elkan & Noto, 2008). In the second category, unlabeled samples should be properly weighted in the loss term in order to train an unbiased classifier. Elkan & Noto (2008) introduces a simple and efficient way of approximating the expectation of a loss function by assigning individual weights w to training examples from the unlabeled cohort. This means each sample from the neighborhood is treated as a positive example with unit weight, while each sample from N̄ is treated as a combination of a positive example with weight w and a negative example with complementary weight 1− w. In the original paper (Elkan & Noto, 2008), the weight is defined as the probability for a sample from the unlabeled set to be a positive sample, i.e. w = p(y = 1|x) for x ∈ U . In the TNC framework, this weight represents the probability of having samples similar to Wt in N̄ . By incorporating weight adjustment into the TNC loss (Equation 1), we account for possible positive samples that occur in the non-neighboring distribution. w can be approximated using prior knowledge of the underlying state distribution or tuned as a hyperparameter. Appendix A.6 explains how the weight parameter is selected for our different experiment setups and also demonstrates the impact of weight adjustment on performance for downstream tasks.
After defining the neighborhood distribution, we train an objective function that encourages a distinction between the representation of samples of the same neighborhood from the outside samples. An ideal encoder preserves the neighborhood properties in the encoding space. Therefore representations Zl = Enc(Wl) of samples from a neighborhood Wl ∈ Nt, can be distinguished from representation Zk = Enc(Wk) of samples from outside the neighborhood Wk ∈ N̄t. TNC is composed of two main components:
1. An Encoder Enc(Wt) that maps Wt ∈ RD×δ to a representation Zt ∈ RM , in a lower dimensional space (M D × δ), where D × δ is the total number of measurements in Wt.
2. A Discriminator D(Zt, Z) that approximates the probability of Z being the representation of a window in Nt. More specifically, it receives two samples from the encoding space and predicts the probability of those samples belonging to the same temporal neighborhood.
Published as a conference paper at ICLR 2021
Enc Enc
Representation
D at a
Enc Enc
Representation
D at a
Weighting
(a) Neighborhood samples
Enc Enc
Representation
D at a
Enc Enc
Representation
D at a
Weighting
(b) Non-neighboring samples
Figure 1: Overview of the TNC framework components. For each sample window Wt (indicated with the dashed black box), we first define the neighborhood distribution. The encoder learns the distribution of windows sampled from Nt and N̄t, in the representation space. Then samples from these distributions are fed into the discriminator alongside Zt, to predict the probability of the windows being in the same neighborhood.
TNC is a general framework; therefore, it is agnostic to the nature of the time series and the architecture of the encoder. The encoder can be any parametric model that is well-suited to the signal properties (Oord et al., 2016; Bai et al., 2018; Fawaz et al., 2019). For the DiscriminatorD(Zt, Z) we use a simple multi-headed binary classifier that outputs 1 if Z and Zt are representations of neighbors in time, and 0 otherwise. In the experiment section, we describe the architectural details of the models used for our experiments in more depth.
Figure 1 provides a summary overview of the TNC framework. We formalize the objective function of our unsupervised learning framework in Equation 1. In essence, we would like the probability likelihood estimation of the Discriminator to be accurate, i.e., close to 1 for the representation of neighboring samples and close to 0 for windows far apart. Samples from the non-neighboring region (N̄ ) are weight-adjusted using the w parameters to account for positive samples in this distribution.
L = −EWt∼X [EWl∼Nt [logD(Zt, Zl)︸ ︷︷ ︸ D(Enc(Wt), Enc(Wl)) ]+EWk∼N̄t [(1−wt)×log (1−D(Zt, Zk))︸ ︷︷ ︸ D(Enc(Wt), Enc(Wk)) +wt×logD(Zt, Zk)]]
(1)
We train the encoder and the discriminator hand in hand by optimizing for this objective. Note that the Discriminator is only part of training and will not be used during inference. Similar to the encoder, it can be approximated using any parametric model. However, the more complex the Discriminator, the harder it becomes to interpret the latent space’s decision boundaries since it allows similarities to be mapped on complex nonlinear relationships.
Defining the neighborhood parameter using the ADF test: As mentioned earlier, the neighborhood range can be specified using the characteristics of the data. In non-stationary time series, the generative process of the signals changes over time. We define the temporal neighborhood around every window as the region where the signal is relatively stationary. Since a signal may remain in an underlying state for an unknown amount of time, each window’s neighborhood range may vary in size and must be adjusted to signal behavior. To that end, we use the Augmented Dickey-Fuller (ADF) statistical test to derive the neighborhood range η. The ADF test belongs to a category of tests called "Unit Root Test", and is a method for testing the stationarity of a time series. For every Wt, we want to find the neighborhood range around that window that indicates a stationary region. To determine this, we start from η = 1 and gradually increase the neighborhood size η, measuring the p-value from the test at every step. Once p-value is above a threshold (in our setting 0.01), it means that it fails to reject the null hypothesis and suggests that within this neighborhood region, the signal is no longer stationary. This way, we find the widest neighborhood within which the signal remains relatively stationary. Note that the window size δ is constant throughout the experiment, and during ADF testing, we only adjust the neighborhood’s width.

heading: 3 EXPERIMENTS

text:
We evaluate our framework’s usability on multiple time series datasets with dynamic latent states that change over time. We compare classification performance and clusterability against two state-ofthe-art approaches for unsupervised representation learning for time series: 1. Contrastive Predictive Coding (CPC) (Oord et al., 2018) that uses predictive coding principles to train the encoder on a probabilistic contrastive loss. 2. Triplet-Loss (T-Loss), introduced in (Franceschi et al., 2019), which employs time-based negative sampling and a triplet loss to learn representations for time series windows. The triplet loss objective ensures similar time series have similar representations by minimizing the pairwise distance between positive samples (subseries) while maximizing it for negative ones. (See Appendix A.2 for more details on each baseline.)
For a fair comparison and to ensure that the difference in performance is not due to the differences in the models’ architecture, the same encoder network is used across all compared baselines. Our objective is to compare the performance of the learning frameworks, agnostic to the encoder’s choice. Therefore, we selected simple architectures to evaluate how each framework can use a simple encoder’s limited capacity to learn meaningful representations. We assess the generalizability of the representations by 1) evaluating clusterability in the encoding space and 2) using the representations for a downstream classification task. In addition to the baselines mentioned above, we also compare clusterability performance with unsupervised K-means and classification with a K-Nearest Neighbor classifier, using Dynamic Time Warping (DTW) to measure time series distance. All models are implemented using Pytorch 1.3.1 and trained on a machine with Quadro 400 GPU 1. Below we describe the datasets for our experiments in more detail.

heading: 3.1 SIMULATED DATA

text: The simulated dataset is designed to replicate very long, non-stationary, and high-frequency time series for which the underlying dynamics change over time. Our generated time series consists of 2000 measurements for 3 features, generated from 4 different underlying states. We use a Hidden Markov Model (HMM) to generate the random latent states over time, and in each state, the time series is generated from a different generative process, including Gaussian Processes (GPs) with different kernel functions and Nonlinear Auto-regressive Moving Average models with different sets of parameters (α and β). Besides, for it to further resemble realistic (e.g., clinical) time series, two features are always correlated. More details about this dataset are provided in the Appendix A.1. For this experimental setup, we use a two-directional, single-layer recurrent neural network encoder. We have selected this simple architecture because it handles time series with variable lengths, and it easily extends to higher-dimensional inputs. The encoder model encodes multi-dimensional signal windows of δ = 50 into 10 dimensional representation vectors. The window size is selected such that it is long enough to contain information about the underlying state but not too long to span over multiple underlying states. A more detailed discussion on the window size choice is presented in Appendix A.4.

heading: 3.2 CLINICAL WAVEFORM DATA

text:
For a real-world clinical experiment, we use the MIT-BIH Atrial Fibrillation dataset (Moody, 1983). This dataset includes 25 long-term Electrocardiogram (ECG) recordings (10 hours in duration) of human subjects with atrial fibrillation. It consists of two ECG signals; each sampled at 250 Hz. The signals are annotated over time for the following different rhythm types: 1) Atrial fibrillation, 2) Atrial flutter, 3) AV junctional rhythm, and 4) all other rhythms. Our goal in this experiment is to identify the underlying type of arrhythmia for each sample without any information about the labels. This dataset is particularly interesting and makes this experiment challenging due to the following special properties:
• The underlying heart rhythm changes over time in each sample. This is an opportunity to evaluate how different representation learning frameworks can handle alternating classes in non-stationary settings;
1Code implementation can be found at https://github.com/sanatonek/TNC_ representation_learning
• The dataset is highly imbalanced, with atrial flutter and AV junctional rhythm being present in fewer than 0.1% of the measurements. Data imbalance poses many challenges for downstream classification, further motivating the use of unsupervised representation learning;
• The dataset has samples from a small number of individuals, but over an extended period (around 5 million data points). This realistic scenario, common in healthcare data, shows that our framework is still powerful in settings with a limited number of samples.
The simple RNN encoder architecture used for other experiment setups cannot model the highfrequency ECG measurements. Therefore, inspired by state-of-the-art architectures for ECG classification problems, the encoder Enc used in this experiment is a 2-channel, 1-dimensional strided convolutional neural network that runs directly on the ECG waveforms. We use six convolutional layers with a total down-sampling factor of 16. The window size is 2500 samples, meaning that each convolutional filter covers at least half a second of ECG recording, and the representations are summarized in a 64-dimensional vector.

heading: 3.3 HUMAN ACTIVITY RECOGNITION (HAR) DATA

text: Human Activity Recognition (HAR) is the problem of predicting the type of activity using temporal data from accelerometer and gyroscope measurements. We use the HAR dataset from the UCI Machine Learning Repository 2 that includes data collected from 30 individuals using a smartwatch. Each person performs six activities: 1) walking, 2) walking upstairs, 3) walking downstairs, 4) sitting, 5) standing, and 6) laying. The time-series measurements are pre-processed to extract 561 features. For our purpose, we concatenate the activity samples from every individual over time using the subject identifier to build the full-time series for each subject, which includes continuous activity change. Similar to the simulated data setting, we use a single-layer RNN encoder. The selected window size is 4, representing about 15 seconds of recording, and the representations are encoded in a 10-dimensional vector space.

heading: 4 RESULTS

text: In this section we present the results for clusterability of the latent representations and downstream classification performance for all datasets and across all baselines. Clusterability indicates how well each method recovers appropriate states, and classification assesses how informative our representations are for downstream tasks.

heading: 4.1 EVALUATION: CLUSTERABILITY

text:
Many real-world time series data have underlying multi-category structure, naturally leading to representations with clustering properties. Encoding such general priors is a property of a good representation (Bengio et al., 2013). In this section, we assess the distribution of the representations in the encoding space. If information of the latent state is properly learned and encoded by the framework, the representations of signals from the same underlying state should cluster together. Figures 2a, 2b, and 2c show an example of this distribution for simulated data across compared approaches. Each plot is a 2-dimensional t-SNE visualization of the representations where each data point in the scatter plot is an encoding Z ∈ R10 that represents a window of size δ = 50 of a simulated time series. We can see that without any information about the hidden states, representations learned using TNC cluster windows from the same hidden state better than the alternative approaches. The results show that CPC and Triplet Loss have difficulty separating time series that are generated from non-linear auto-regressive moving average (NARMA) models with variable regression parameters.
To compare the representation clusters’ consistency for each baseline, we use two very common cluster validity indices, namely, the Silhouette score and the Davies-Bouldin index. We use K-means clustering in the representation space to measure these clusterability scores. The Silhouette score measures the similarity of each sample to its own cluster, compared to other clusters. The values can range from −1 to +1, and a greater score implies a better cohesion. The Davies-Bouldin Index
2https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+ using+smartphones
measures intra-cluster similarity and inter-cluster differences. This is a positive index score, where smaller values indicate low within-cluster scatter and large separation between clusters. Therefore, a lower score represents better clusterability (more details on the cluster validity scores and how they are calculated can be found in Appendix A.5).
Table 1 summarizes the scores for all baselines and across all datasets, demonstrating that TNC is superior in learning representations that can distinguish the latent dynamics of time series. CPC performs closely to Triplet loss on waveform data but performs poorly on the simulated dataset, where signals are highly non-stationary, and transitions are less predictable. However, for the HAR dataset, CPC clusters the states very well because most activities are recorded in a specific order, empowering predictive coding. Triplet loss performs reasonably well in the simulated setting; however, it fails to distinguish states 0 and 2, where signals come from autoregressive models with different parameters and have a relatively similar generative process. Performing K-means on the original time series generally does not generate coherent clusters, as demonstrated by the scores. However, the performance is slightly better in time series like the ECG waveforms, where the signals are formed by consistent shapelets, and therefore the DTW measures similarity more accurately.

heading: 4.2 EVALUATION: CLASSIFICATION

text:
We further evaluate the quality of the encodings using a classification task. We train a linear classifier to evaluate how well the representations can be used to classify hidden states. The performance of all baselines is compared to a supervised classifier, composed of an encoder and a classifier with identical architectures to that of the unsupervised models, and a K-nearest neighbor classifier that uses DTW metric. The performance is reported as the prediction accuracy and the area under the precision-recall curve (AUPRC) score since AUPRC is a more accurate reflection of model performance for imbalance classification settings like the waveform dataset.
Table 2 demonstrates the classification performance for all datasets. The performance of the classifiers that use TNC representations are closer to the end-to-end supervised model in comparison to CPC and Triplet Loss. This provides further evidence that our encodings capture informative parts of the time series and are generalizable to be used for downstream tasks. In datasets like the HAR, where
an inherent ordering usually exists in the time series, CPC performs reasonably. However, in datasets with increased non-stationarity, the performance drops. Triplet Loss is also a powerful framework, but since it samples positive examples from overlapping windows of time series, it is vulnerable to map the overlaps into the encoding and, therefore, fail to learn more general representations. TNC, on the other hand, samples similar windows from a wider distribution, defined by the temporal neighborhood, where many of the neighboring signals do not necessarily overlap. The lower performance of the CPC and Triplet Loss methods can also be partly because none of these methods explicitly account for the potential sampling bias that happens when randomly selected negative examples are similar to the reference Wt.

heading: 4.3 EVALUATION: TRAJECTORY

text: This section investigates the trajectories of our learned encodings over time to understand how the state transitions are captured and modeled in the representation space. This is an important property for non-stationary time series where underlying states change over time, and capturing those changes is critical in many application domains such as healthcare. Figure 3 shows a sample from the simulated dataset. The top panel shows the signal measurements over time, and the shaded regions indicate the underlying latent states. The bottom panel illustrates the 10-dimensional representation of a sliding window Wt estimated over time. From the bottom panel of Figure 3, we can see that the encoding pattern changes at state transitions and settle into a different pattern, corresponding to the new state. This change happens at every transition, and we can see the distinct patterns for all 4 underlying states in the representations. This analysis of the trajectory of change could be very informative for the users’ post-analysis; for instance, in clinical applications, it could help clinicians visualize the evolution of the patient state over time and plan treatment based on the state progression

heading: 5 RELATED WORK

text:
While integral for many applications, unsupervised representation learning has been far less studied for time series (Längkvist et al., 2014), compared to other domains such as vision or natural language processing (Denton et al., 2017; Radford et al., 2015; Gutmann & Hyvärinen, 2012; Wang & Gupta, 2015). One of the earliest approaches to unsupervised end-to-end representation learning in time series is the use of auto-encoders (Choi et al., 2016a; Amiriparian et al., 2017; Malhotra et al., 2017) and seq-to-seq models (Lyu et al., 2018), with the objective to train an encoder jointly with a decoder that reconstructs the input signal from its learned representation. Using fully generative models like variational auto-encoders is also useful for imposing properties like disentanglement, which help with the interpretability of the representations (Dezfouli et al., 2019). However, in many cases, like for high-frequency physiological signals, the reconstruction of complex time series can be challenging; therefore, more novel approaches are designed to avoid this step. Contrastive Predictive Coding (Oord et al., 2018; Löwe et al., 2019) learns representations by predicting the future in latent space, eliminating the need to reconstruct the full input. The representations are such that the mutual information between the original signal and the concept vector is maximally preserve using a lower bound approximation and a contrastive loss. Very similarly, in Time Contrastive Learning (Hyvarinen & Morioka, 2016), a contrastive loss is used to predict the segment-ID of multivariate time-series as a way to extract representation. Franceschi et al. (2019) employs time-based negative sampling and a triplet loss to learn scalable representations for multivariate time series. Some other approaches use inherent similarities in temporal data to learn representations without supervision. For instance, in similarity-preserving representation learning (Lei et al., 2019), learned encodings are constrained to preserve the pairwise similarities that exist in the time domain, measured by DTW distance. Another group of approaches combines reconstruction loss with clustering objectives to cluster similar temporal patterns in the encoding space (Ma et al., 2019).
In healthcare, learning representation of rich temporal medical data is extremely important for understanding patients’ underlying health conditions. However, most of the existing approaches for learning representations are designed for specific downstream tasks and require labeling by experts (Choi et al., 2016b;c; Tonekaboni et al., 2020). Examples of similar works to representation learning in the field of clinical ML include computational phenotyping for discovering subgroups of patients with similar underlying disease mechanisms from temporal clinical data (Lasko et al., 2013; Suresh et al., 2018; Schulam et al., 2015), and disease progression modeling, for learning the hidden vector of comorbidities representing a disease over time (Wang et al., 2014; Alaa & van der Schaar, 2019).

heading: 6 CONCLUSION

text: This paper presents a novel unsupervised representation learning framework for complex multivariate time series, called Temporal Neighborhood Coding (TNC). This framework is designed to learn the underlying dynamics of non-stationary signals and to model the progression over time by defining a temporal neighborhood. The problem is motivated by the medical field, where patients transition between distinct clinical states over time, and obtaining labels to define these underlying states is challenging. We evaluate the performance of TNC on multiple datasets and show that our representations are generalizable and can easily be used for diverse tasks such as classification and clustering. We finally note that TNC is flexible to be used with arbitrary encoder architectures; therefore, the framework is applicable to many time series data domains. Moreover, in addition to tasks presented in this paper, general representations can be used for several other downstream tasks, such as anomaly detection, which is challenging in supervised learning settings for time series data in sparsely labeled contexts.

heading: ACKNOWLEDGMENTS

text: Resources used in preparing this research were provided, in part, by the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. This research was undertaken, in part, thanks to funding from the Canadian Institute of Health Research (CIHR) and the Natural Sciences and Engineering Research Council of Canada (NSERC).

heading: A APPENDIX

heading: A.1 SIMULATED DATASET

text:
The simulated time series consists of 3 features generated from different underlying hidden states. Figure A.1 shows a sample from this dataset. Each panel in the figure shows one of the features, and the shaded regions indicate the underlying state of the signal in that period. We use a Hidden Markov Model (HMM) to generate these random latent states over time. The transition probability is set equal to %5 for switching to an alternating state, and %85 for not changing state. In each state, the time series is generated from a different signal distribution. Table 3 describes the generative process of each signal feature in each state. Note that feature 1 and 2 are always correlated, mainly to mimic realistic clinical time series. As an example, physiological measurements like pulse rate and heart rate are always correlated.
In the state 1, the correlated features are generated by a Gaussian Process (GP) with a periodic kernel. Feature 3, which is uncorrelated with the other two features, comes from another GP with a squared exponential kernel. In addition to GPs, we also have multiple Non-Linear Auto-Regressive Moving Average (NARMA) time series models. The linear function of NARMAα and NARMAβ are shown in Equation 2 and 3.
NARMAα : y(k + 1) = 0.3y(k) + 0.05y(k) n−1∑ i=0 y(k − i) + 1.5u(k − (n− 1))u(k) + 0.1 (2)
NARMAβ : y(k + 1) = 0.1y(k) + 0.25y(k) n−1∑ i=0 y(k − i) + 2.5u(k − (n− 1))u(k)− 0.005 (3)
A white Gaussian noise with σ = 0.3 is added to all signals, and overall, the dataset consists of 500 instances of T = 2000 measurements.

heading: A.2 BASELINE IMPLEMENTATION DETAILS

text:
Implementation of all baselines are included in the code base for reproducibility purposes, and hyper-parameters for all baselines are tuned using cross-validation.
Contrastive Predictive Coding (CPC): The CPC baseline first processes the sequential signal windows using an encoder Zt = Enc(Xt), with a similar architecture to the encoders of other baselines. Next, an autoregressive model gar aggregates all the information in Z≤t and summarizes it into a context latent representation ct = gar(Z≤t). In our implementation, we have used a single layer, a one-directional recurrent neural network with GRU cell and hidden size equal to the encoding size as the auto-regressor. Like the original paper, the density ratio is estimated using a linear transformation, and the model is trained for 1 step ahead prediction.
Triplet-Loss (T-Loss): The triplet loss baseline is implemented using the original code made available by the authors on Github3.
KNN and K-means: These two baselines for classification and clustering are implemented using the tslearn library 4, that integrates distance metrics such as DTW. Note that evaluating DTW is computationally expensive, and the tslearn implementation is not optimized. Therefore, for the waveform data with windows of size 2500, we had to down-sampled the signal frequency by a factor of two.

heading: A.3 TNC IMPLEMENTATION EXTRA DETAILS

text: To define the neighborhood range in the TNC framework, as mentioned earlier, we use the AugmentedDickey Fuller (ADF) statistical test to determine this range (η) as the region for which the signals remain stationary. More precisely, we gradually increase the range, from a single window size up to 3 times the window size (the upper limit we set), and repeatedly perform the ADF test. We use the p-value from this statistical test to determine whether the Null hypothesis can be rejected, meaning that the signal is stationary. At the point where the p-value is above our defined threshold (0.01), we can no longer assume that the signal is stationary, and this is where we set the η parameter. Now, once the neighborhood is defined, we make sure the non-neighboring samples are taken from the distribution of windows with at least 4×η away fromWt, ensuring a low likelihood of belonging to the neighborhood. Note that for implementation of ADF, we use the stats model library 5. Unfortunately, this implementation is not optimized and does not support GPU computation, therefore evaluating the neighborhood range using ADF slows down TNC framework training. As a future direction, we are working on an optimized implementation of the ADF score for our framework.

heading: A.4 SELECTING THE WINDOW SIZE

text:
The window size δ is an important factor in the performance of a representation learning framework, not only for TNC but also for similar baselines such as CPC and triplet loss. Overall, the window size should be selected such that it is long enough to contain information about the underlying state and not too long to span over multiple underlying states. In our settings, we have selected the window sizes based on our prior knowledge of the signals. For instance, in the case of an ECG signal, the selected window size is equivalent to 7 seconds of recording, which is small enough such that the ECG remains in a stable state and yet has enough information to determine that underlying state. Our understanding of the time series data can help us select an appropriate window size, but we can also experiment with different δ to learn this parameter. Table 4 shows classification performance results for the simulation setups, under different window sizes. We can clearly see the drop in performance for all baseline methods when the window size is too small or too large.
3https://github.com/White-Link/UnsupervisedScalableRepresentationLearningTimeSeries 4https://tslearn.readthedocs.io/en/stable/index.html 5https://www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.
adfuller.html

heading: A.5 CLUSTERING METRICS

text:
Most cluster validity measures assess certain structural properties of a clustering result. In our evaluation, we have used two measures, namely the Silhouette score and Davies-Bouldin index, to evaluate the representations’ clustering quality. Davies-Bouldin measures intra-cluster similarity (coherence) and inter-cluster differences (separation). Let C = {C1, ..., Ck} be a clustering of a set D of objects. The Davies-Bouldin score is evaluated as follows:
DB = 1
k k∑ i=1 maxj s(C) + s(C) δCiCj
(4)
Where s(C) measures the scatter within a cluster, and δ is a cluster to cluster distance measure. On the other hand, the silhouette score measures how similar an object is to its cluster compared to other clusters. Both measures are commonly used for the evaluation of clustering algorithms. A comparison of 2 metrics has shown that the Silhouette index produces slightly more accurate results in some cases. However, the Davies-Bouldin index is generally much less complex to compute Petrovic (2006).

heading: A.6 SETTING THE WEIGHTS FOR PU LEARNING

text: As mentioned in the Experiment section, the weight parameter in the loss is the probability of sampling a positive window from the non-neighboring region. One way to set this parameter is using prior knowledge of the number and the distribution of underlying states. Another way is to learn it as a hyperparameter. Table 5 shows the TNC loss for different weight parameters. The loss column reports the value measured in Equation 1, and the accuracy shows how well the discriminator identifies the neighboring samples from non-neighboring ones for settings with different weight parameters. To also assess the impact of re-weighting the loss on downstream classification performance, we compared these performance measures for weighted and non-weighted settings. Table 6 demonstrates these results and confirms that weight adjusting the loss for non-neighboring samples improves the quality of learned representations.

heading: A.7 SUPPLEMENTARY FIGURE

heading: A.7.1 CLINICAL WAVEFORM DATA

text: In order to understand what TNC framework encodes from the high dimensional ECG signals, we visualize the trajectory of the representations of an individual sample over time. Figure A.3 demonstrates this example, where the top 2 rows are ECG signals from two recording leads and the bottom row demonstrates the representation vectors. We see that around second 40 the pattern in the representations change as a result of an artifact happening in one of the signals. With the help from our clinical expert, we also tried to interpret different patterns in the encoding space. For instance, between time 80 and 130, where features 0-10 become more activated, the heart rate (HR) has increased. Increase in HR can be seen as increased frequency in the ECG signals and is one of the indicators of arrhythmia that we believe TNC has captured. Figure A.2 shows the distribution of the latent encoding of ECG signals for different baselines, with colors indicating the arrhythmia class.

heading: A.7.2 HAR DATA

text: Figure A.4 and A.5 are similar plots to the ones demonstrated in the previous section, but for the HAR dataset. As shown in Figure A.5, the underlying states of the signal are clearly captured by the TNC framework as different patterns in the latent representations.

heading: A.7.3 SIMULATION DATA

text: In addition to the initial experiment, we also show the trajectory of the encoding for a smaller encoding size (3). In this setting, we have 4 underlying states in the signal, and only 3 dimensions for the encoding.

title: TEMPORAL NEIGHBORHOOD CODING


summaries: This paper presents a method of composing stacked neural network blocks by linearly combining module "template" weights.  The work extends "isometric networks", in which each block in the stack has the same operational structure, by parameterizing each block using a mixture of the weights from a bank of K blocks ("templates").  In multitask learning experiments, the mixtures naturally learn to share common components between tasks, while learning task-specific components where needed.  Further experiments describe behavior of the system applied to transfer learning and domain adaptation scenarios.

summaries: The authors propose a method to design network architectures as a combination of network components. Their idea consists in imposing an architecture that is a sequence of identical network blocks, and learn a series of templates, which provide an instantiation of model weights for one such network block. Model weights are then estimated for a specific task as a linear combination of a set of template weights.

summaries: This paper considers “modular multi-task learning” where parameters in each layer/task are generated as a (layer/task-specific) linear (mixture) combination of a common pool of parameters. Exploring this idea, several observations are made: (1) single task isometric model performance on ImageNet can be improved, (2) Multi-task learning is supported and parameter sharing (selecting same mixture components) emerges in early layers, with specialisation emerging in later layers. With multi-domain learning, the opposite effect is achieved with domain-wise specificity arising in earlier layers, and sharing in later layers. (3) Parameter-efficient transfer learning is supported by fine-tuning the task-specific weights for new tasks. (4) Parameter-efficient domain adaptation is supported by optimising the task-specific weights for new domains.

abstractText: Conditional computation and modular networks have been recently proposed for multitask learning and other problems as a way to decompose problem solving into multiple reusable computational blocks. We propose a novel fully-differentiable approach for learning modular networks. In our method, the modules can be invoked repeatedly and allow knowledge transfer to novel tasks by adjusting the order of computation. This allows soft weight sharing between tasks with only a small increase in the number of parameters. We show that our method leads to interpretable self-organization of modules in case of multi-task learning, transfer learning and domain adaptation while achieving competitive results on those tasks. From practical perspective, our approach allows to: (a) reuse existing modules for learning new task by adjusting the computation order, (b) use it for unsupervised multi-source domain adaptation to illustrate that adaptation to unseen data can be achieved by only manipulating the order of pretrained modules, (c) show how our approach can be used to increase accuracy of existing architectures for image classification tasks such as IMAGENET, without any parameter increase, by reusing the same block multiple times.

authors:
Jacob Andreas
Marcus Rohrbach
Trevor Darrell
Dan Klein

title: Neural module networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Bowen Baker
Otkrist Gupta
Nikhil Naik
Ramesh Raskar

title: Designing neural network architectures using reinforcement learning

venue: arXiv preprint arXiv:1611.02167,

title: On reinforcement learning for deep neural architectures

authors:
Woong-Gi Chang
Tackgeun You
Seonguk Seo
Suha Kwak
Bohyung Han

title: Domain-specific batch normalization for unsupervised domain adaptation

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Dario Floreano
Peter Dürr
Claudio Mattiussi

title: Neuroevolution: from architectures to learning

venue: Evolutionary intelligence,

authors:
Pengsheng Guo
Chen-Yu Lee
Daniel Ulbricht

title: Learning to branch for multi-task learning

venue: arXiv preprint arXiv:2006.01895,

authors:
David Ha
Andrew M. Dai
Quoc V. Le

title: URL https://openreview.net/forum?id= rkpACe1lx

venue: Hypernetworks. In 5th International Conference on Learning Representations,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Sebastian Herzog
Christian Tetzlaff
Florentin Wörgötter

title: Evolving artificial neural networks with feedback

venue: Neural Networks,

authors:
Judy Hoffman
Eric Tzeng
Taesung Park
Jun-Yan Zhu
Phillip Isola
Kate Saenko
Alexei Efros
Trevor Darrell

title: Cycada: Cycle-consistent adversarial domain adaptation

venue: In International conference on machine learning,

authors:
Andrew Howard
Ruoming Pang
Hartwig Adam
Quoc V. Le
Mark Sandler
Bo Chen
Weijun Wang
Liang-Chieh Chen
Mingxing Tan
Grace Chu
Vijay Vasudevan
Yukun Zhu

title: Searching for mobilenetv3

venue: IEEE/CVF International Conference on Computer Vision,

authors:
Ronghang Hu
Jacob Andreas
Marcus Rohrbach
Trevor Darrell
Kate Saenko

title: Learning to reason: End-to-end module networks for visual question answering

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Ronghang Hu
Marcus Rohrbach
Jacob Andreas
Trevor Darrell
Kate Saenko

title: Modeling relationships in referential expressions with compositional modular networks

authors:
Ronghang Hu
Jacob Andreas
Trevor Darrell
Kate Saenko

title: Explainable neural computation via stack neural module networks

venue: In Proceedings of the European conference on computer vision (ECCV),

authors:
Lukasz Kaiser
Ilya Sutskever

title: Neural gpus learn algorithms

venue: In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations,

authors:
Kohitij Kar
Jonas Kubilius
Kailyn Schmidt
Elias B Issa
James J DiCarlo

title: Evidence that recurrent circuits are critical to the ventral streams execution of core object recognition behavior

venue: Nature neuroscience,

authors:
Louis Kirsch
Julius Kunze
David Barber

title: Modular networks: Learning to decompose neural computation

venue: In Advances in Neural Information Processing Systems,

venue: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems

authors:
Ming-Yu Liu
Thomas Breuel
Jan Kautz

title: Unsupervised image-to-image translation networks

venue: In Advances in neural information processing systems,

authors:
Mingsheng Long
Yue Cao
Jianmin Wang
Michael Jordan

title: Learning transferable features with deep adaptation networks

authors:
Mingsheng Long
Han Zhu
Jianmin Wang
Michael I Jordan

title: Deep transfer learning with joint adaptation networks

authors:
Jiaqi Ma
Zhe Zhao
Jilin Chen
Ang Li
Lichan Hong
Ed H Chi

title: Snr: Sub-network routing for flexible parameter sharing in multi-task learning

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Kevis-Kokitsi Maninis
Ilija Radosavovic
Iasonas Kokkinos

title: Attentive single-tasking of multiple tasks

authors:
Krzysztof Maziarz
Efi Kokiopoulou
Andrea Gesmundo
Luciano Sbaiz
Gabor Bartok
Jesse Berent

title: Gumbel-matrix routing for flexible multi-task learning

authors:
Risto Miikkulainen
Jason Liang
Elliot Meyerson
Aditya Rawal
Daniel Fink
Olivier Francon
Bala Raju
Hormoz Shahrzad
Arshak Navruzyan
Nigel Duffy

title: Evolving deep neural networks

venue: In Artificial Intelligence in the Age of Neural Networks and Brain Computing,

authors:
Ishan Misra
Abhinav Shrivastava
Abhinav Gupta
Martial Hebert

title: Cross-stitch networks for multi-task learning

authors:
Pramod Kaushik Mudrakarta
Mark Sandler
Andrey Zhmoginov
Andrew G. Howard

title: K for the price of 1: Parameter-efficient multi-task and transfer learning

venue: In 7th International Conference on Learning Representations,

authors:
Alejandro Newell
Lu Jiang
Chong Wang
Li-Jia Li
Jia Deng

title: Feature partitioning for efficient multi-task architectures

venue: arXiv preprint arXiv:1908.04339,

authors:
Vardaan Pahuja
Jie Fu
Sarath Chandar
Christopher J Pal

title: Structure learning for neural module networks

venue: arXiv preprint arXiv:1905.11532,

authors:
Xingchao Peng
Kate Saenko

title: Synthetic to real adaptation with generative correlation alignment networks

venue: IEEE Winter Conference on Applications of Computer Vision (WACV),

authors:
Xingchao Peng
Qinxun Bai
Xide Xia
Zijun Huang
Kate Saenko
Bo Wang

title: Moment matching for multi-source domain adaptation

authors:
Senthil Purushwalkam
Maximilian Nickel
Abhinav Gupta
Marc’Aurelio Ranzato

title: Task-driven modular networks for zero-shot compositional learning

authors:
Ettore Randazzo
Alexander Mordvintsev
Eyvind Niklasson
Michael Levin
Sam Greydanus

title: Self-classifying mnist digits. Distill, 2020

authors:
Clemens Rosenbaum
Tim Klinger
Matthew Riemer

title: Routing networks: Adaptive selection of non-linear functions for multi-task learning

venue: arXiv preprint arXiv:1711.01239,

authors:
Clemens Rosenbaum
Ignacio Cases
Matthew Riemer
Atticus Geiger
Lauri Karttunen
Joshua D Greene
Dan Jurafsky
Christopher Potts

title: Dispatched routing networks

venue: Technical report, Technical Report Stanford AI Lab, NLP Group Tech Report 2019-1,

authors:
Kuniaki Saito
Yoshitaka Ushiku
Tatsuya Harada
Kate Saenko

title: Strong-weak distribution alignment for adaptive object detection

authors:
Mark Sandler
Jonathan Baccash
Andrey Zhmoginov
Andrew Howard

title: Non-discriminative data or weak model? on the relative importance of data and model resolution

venue: IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops

authors:
Baochen Sun
Jiashi Feng
Kate Saenko

title: Return of frustratingly easy domain adaptation

venue: In Thirtieth AAAI Conference on Artificial Intelligence,

authors:
Ximeng Sun
Rameswar Panda
Rogerio Feris

title: Adashare: Learning what to share for efficient deep multi-task learning

venue: arXiv preprint arXiv:1911.12423,

authors:
Eric Tzeng
Judy Hoffman
Ning Zhang
Kate Saenko
Trevor Darrell

title: Deep domain confusion: Maximizing for domain invariance

venue: arXiv preprint arXiv:1412.3474,

authors:
Eric Tzeng
Judy Hoffman
Kate Saenko
Trevor Darrell

title: Adversarial discriminative domain adaptation

authors:
Dequan Wang
Evan Shelhamer
Shaoteng Liu
Bruno Olshausen
Trevor Darrell

title: Fully test-time adaptation by entropy minimization

venue: arXiv preprint arXiv:2006.10726,

authors:
Daan Wierstra
Faustino J Gomez
Jürgen Schmidhuber

title: Modeling systems with internal state using evolino

venue: In Proceedings of the 7th annual conference on Genetic and evolutionary computation,

authors:
Zuxuan Wu
Tushar Nagarajan
Abhishek Kumar
Steven Rennie
Larry S Davis
Kristen Grauman
Rogerio Feris

title: Blockdrop: Dynamic inference paths in residual networks

authors:
Siming Yan
Xuyang Fang
Bowen Xiao
Harold Rockwell
Yimeng Zhang
Tai Sing Lee

title: Recurrent feedback improves feedforward representations in deep neural networks

venue: URL http://arxiv.org/abs/1912.10489

authors:
Werner Zellinger
Thomas Grubinger
Edwin Lughofer
Thomas Natschläger
Susanne Saminger-Platz

title: Central moment discrepancy (cmd) for domain-invariant representation learning

venue: arXiv preprint arXiv:1702.08811,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Barret Zoph
Quoc V Le

title: Neural architecture search with reinforcement learning

venue: arXiv preprint arXiv:1611.01578,

heading: 1 INTRODUCTION

text:
Most modern interpretations of neural networks treat layers as transformations that convert lowerlevel features, such as pixels, into higher-level abstractions. Adopting this view, many designs that break up monolithic models into interacting modular components localize them to particular processing stages thus disallowing parameter sharing across layers. This restriction, however, can be limiting when we need to continually grow the model, be that for transferring knowledge to a new task or preventing catastrophic forgetting. It also prohibits module reuse within a model even though certain parameter-efficient architectures and complex decision making processes may involve recurrent components (Kaiser & Sutskever, 2016; Randazzo et al., 2020) and feedback loops (Herzog et al., 2020; Yan et al., 2019; Kar et al., 2019).
Here we propose a simple alternative design that represents an entire model as a mixture of modules each of which can contribute to computation at any processing stage. As opposed to many other approaches (Zoph & Le (2016); Bengio (2016); Kirsch et al. (2018a); Rosenbaum et al. (2019) etc.), we use a soft mixture of modules to obtain a more flexible model and produce a differentiable optimization objective that can be optimized end-to-end and does not involve high-variance estimators. Following our approach, the parameters of every block (or layer) of the network is computed as a linear combination of a set of “template” block parameters thus representing the entire model as: (a) a databank of template blocks and (b) vectors of “mixture weights” that are used to generate weights for every layer. This simple design can be utilized for a variety of applications from producing compact networks and training multi-task models capable of re-using individual modules to knowledge transfer and domain adaptation. The experimental results show that: (a) when used for multi-task training, our model organizes its modules where tasks share first few layers while specializing closer to the head, while (b) in domain adaptation problems modules instead specialize on processing the image, while sharing later processing stages. Moreover, our self-organizing model achieves promising results in multi-task learning and model personalization. The rest of the paper is organized as follows: in Section 2 we go over the related literature and discuss the existing approaches to modular
networks and neural architecture search; in Section 3 we describe the architecture of our conversational model and various aspects of learning the best module sequence for each specific task; we introduce the experimental results in a single-task setting, multi-task learning, continual learning and domain adaptation in Section 4 and provide a final discussion in Section 5.

heading: 2 PRIOR WORK

text: The idea of constructing a deep neural network as a composition of reusable computation blocks has been extensively explored in recent literature. First conditioned computation methods used evolution algorithms (Wierstra et al. (2005); Floreano et al. (2008); Miikkulainen et al. (2019)) to determine a suitable model architecture; later on, reinforcement learning was used to optimize the model layout and parameters (Zoph & Le (2016); Bengio (2016); Baker et al. (2016)). Recently, Kirsch et al. (2018a) proposed an end-to-end modular neural network framework that learns to choose the best among several module candidates at each layer according to the input data. Conditional computation routing is even more appealing for the multitask learning application, as it allows to both learn reusable computation blocks and adjust the model to each specific task with minimal network alteration. For instance, Misra et al. (2016) use an additional set of modules for each task and enforce similarity between the corresponding task-specific modules weights. Rosenbaum et al. (2017; 2019) used reinforcement learning to perform task- and data-specific routing; Sun et al. (2019) train both the model weights and the task-specific policy that determines which layers should be shared for a given task; Ma et al. (2019); Maziarz et al. (2019) introduced a multitask model where each layer’s output is computed as a weighted sum of a set or candidate modules outputs, which is similar to our method, with two important differences: a) in the works by Ma et al. (2019); Maziarz et al. (2019), the output is computed as a linear combination of candidates outputs, not module weights, and b) the module candidates in their methods are tied to the specific location in the network. Maninis et al. (2019) added a task-specific residual adapters to specialize the feature extractor to each task, The method proposed by Purushwalkam et al. (2019) performs zero-shot multitask learning by finding a task-specific routing using a gating mechanism. This approach is somewhat similar to ours, but is less efficient in terms of the number of parameters since a) the method in Purushwalkam et al. (2019) uses a modular network on top of a classical ResNet (He et al. (2016)) while our method uses only about 100 additional parameters and b) the modules used in Purushwalkam et al. (2019) are layer-specific and not reusable while there is no such constraint in our architecture. Wu et al. (2018); Newell et al. (2019); Guo et al. (2020) perform task-specific model compression. There also exists a number of studies on the effectiveness of modular networks in the context of visual question answering. One of the early papers Andreas et al. (2016) used a natural language parser to determine the layout of the composite network consisting of predefined modules that solve different kinds of subtasks, such as find, measure, describe etc; further works improved this approach by switching from an external parser to a fixed (Hu et al. (2017b)) or arbitrary query structure (Hu et al. (2017a; 2018); Pahuja et al. (2019)).

heading: 3 COMPOSITIONAL MODELS AND MODULE MIXTURES

heading: 3.1 COMPOSITIONAL MODELS

text:
In conventional convolutional deep neural networks, most layers are different from each other and do not conform to a particular fixed design. Various model blocks may have different resolutions and different numbers of filters, some blocks may have or lack a residual connection and so on. This makes most model activations at different layers incompatible with each other. Recently, it has been demonstrated (Sandler et al., 2019) that high performing convolutional networks can nevertheless be composed of identical blocks. Such networks that were called isometric networks essentially iterate on the same activation space. The architecture of an isometric network includes the following core components: (a) input adapter that maps model input i ∈ I into the activation space Z; (b) model body, a sequence of blocks all sharing the same architecture and mapping the space Z to itself; (c) output adapter mapping the output of the last block into the embedding space E, and finally (d) logits layer for classification models, mapping embedding spaceE into the final predicted probabilities.
While isometric networks achieve near SOTA performance, the fact that they iterate on the same internal activation space Z makes them a perfect candidate for studying modular architectures. Specifically, instead of treating each block as an independently trainable mapping, we may view the entire model body as a composition of modules fk each of which may appear in a sequence more than once. In the following, we will denote a set of trainable modules as F = {f1, f2, . . . , fK} and their composition f = fmL ◦ · · · ◦ fm1 forming a model body1 will frequently be denoted as f(m1,...,mL) with (m1, . . . ,mL) being called a signature of this model.

heading: 3.2 LEARNING COMPOSITIONAL MODELS

text:
Mixture modules Compositional models discussed in Section 3.1 can be trained on a particular task for any predefined choice of the model signature. But, while the study of hand-designed modular architectures is an interesting exploration in itself, one may want to go further and identify a sequence of modules (m1,m2, . . . ,mL) together with the modules F themselves that is the most advantageous choice for solving a particular problem. In some previous explorations of modular architectures, authors used a probabilistic approach where the model body (m1, . . . ,mL) was sampled for every input batch or sample from the optimized probability distribution p(m1,m2, . . . ,mL). For example, in Kirsch et al. (2018b), modules2 were sampled from a distribution defined by per-layer conditional probabilities p(m`|x`−1) with x`−1 being the output of the previous layer (` − 1), i.e., x`−1 = f(m1,...,m`−1)(i). Since discrete nature of such a choice limits the flexibility of the model and since in practice learning such a model may be complicated due to high-variance estimator noise, we instead replace a composition of blocks from F with a composition of mixture modules, each of which is essentially a smooth superposition of blocks from F . More precisely, each mixture module hξ with ξ ∈ RK has the same architecture as blocks from F , but every parameter θ in this mixture module (be that some convolutional kernel, or a batch normalization β or γ parameter) is a linear combination ∑K k=1 ξkθk of the corresponding parameters θk from fk ∈ F . Notice that since each block contains multiple convolutions with different nonlinearities, a mixture module obtained by linearly combining weights from F is not a linear combination of modules from F . It is true however that if ξi = δij for some fixed3 j (which we later refer to as “one-hot” vectors), a mixture module hξ becomes identical to fj ∈ F . Due to the fact that hξ mixes parameters from F , we will
1with L is the total number of layers in the model 2here each mk is actually a set of several chosen modules the output of which is combined 3Here δij is a Kronecker delta
sometimes refer to blocks from F as templates, which are mixed together to finally form an actual unit of computation.
After we choose a particular task and replace the model body with a composition of mixture modules hξL ◦ · · · ◦ hξ1 , we obtain a differentiable objective that can be optimized with respect to a model signature Ξ = (ξ1, . . . , ξL) and parameters of F . In our experiments, we constrained each of ξ` by using a softmax nonlinearity, which allowed us to guarantee that ∑ i(ξ`)i = 1 for every layer `. Of course, even with this choice of the nonlinearity, we are not guaranteed that the final signature Ξ will contain only one-hot vectors ξ` that can be interpreted as blocks from F .
Regularization of mixture weights While in our experiments, mixture weights ξ` frequently approached a set of approximately one-hot vectors4, one can use additional regularizers to ensure that all ξ` are one-hot and the resulting model can be represented as a composition of individual template modules. In Appendix A, we outline two choices that we experimented with: (a) entropy regularization that penalizes maxi(ξ`)i < 1, but does not involve template weights and (b) a regularizer that penalizes Ξ and also effectively attracts individual templates towards the generated mixture weights to which they contribute the most.
Weight generation interpretation Consider a single block of our isometric network and let F be a function family containing single-block tensor transformations corresponding to all possible choices of block weights. All neural networks obtained by composing L such blocks without any weighttying form a function family FL. Now, instead, consider a model composed of mixture-weight blocks with templates from F . In this case, the family of all possible single-block transformations is only controlled by a vector of mixture weights ξ and narrows down to a family further denoted as FF . A composition of mixture-weights blocks is thus FLF . In a sense, choosing templates F , we introduce a small function family that can be parameterized by just a handful of real-valued parameters Ξ. This perspective shows a connection of the proposed approach to other approaches parameterizing neural network weights like, for example, HYPERNETWORKS (Ha et al., 2017).

heading: 4 APPLICATIONS AND EXPERIMENTAL RESULTS

heading: 4.1 ISOMETRIC MODEL DETAILS

text:
Before discussing our experimental results, we first need to briefly describe the architecture of the Isometric network used as a basis of these experiments. As described in Section 3.1, our implementation of the modular Isometric network contains the following four components (see Appendix B.1 for details): (a) input adapter, a nonlinear convolution with the kernel and the stride of the same size (typically 4 to 16), which effectively downsamples the image while increasing the number of channels in the tensor to 40%, where % is the model depth multiplier controlling the complexity of the network; (b) model block, a typical MobileNetV3 (Howard et al., 2019), or Isometric model residual block (Sandler et al., 2019) with the “expansion”, depthwise and “projection” convolutions and a squeeze-and-excite block (Hu et al., 2020); (c) output adapter, a sequence of two nonlinear kernel-1 convolutions with an average pooling layer in between and finally (d) logits layer, a fully-connected layer producing model predictions. In the following, we will rely on two specific designs: (a) larger design more suitable for complex tasks that uses 40-channel tensors (or % = 1) and the embedding size of 1280 (as produced by the output adapter) and (b) smaller design typically using 24-channel tensors5 and the embedding size of 256.
In our implementation of networks with mixture modules, we used mixture weights ξ ∈ RK for mixing all variables appearing in every component of the block except for batch normalization (BN) parameters: this included all convolutional weights and biases, and all components of the squeezeand-excite block. In some experiments, we also used the same ξ to produce a linear combination of batch normalization β and γ parameters (but never moving mean variables). In the following, we will refer to such experiments as experiments with “mixed batch normalization parameters”.
Notice that generating block parameters from templates and mixture weights can be expensive for a large number of templates inF . However, since in this work we perform this calculation for an entire
4i.e., a state with maxi(ξ`)i ≈ 1 for most layers ` 5We used the number of channels divisible by 8 for training efficiency.
batch and not for individual samples, this procedure did not introduce a significant computational overhead.

heading: 4.2 LEARNING A SINGLE TASK

text:
We began our study of networks with mixture weights by training these models on individual supervised tasks. In our initial experiments, we studied models with 4 to 32 layers and 2 to 16 templates forming F . We trained these models on the CIFAR-100 dataset using a 24-channel tensors and a smaller output adapter (see Appendix B.1).
When training a model on a single task, we observe that even in the absence of regularization, the mixture weights Ξ frequently approach a collection of one-hot vectors, i.e., maxi(ξ`)i ≈ 1 for most layers `. Discovered model signatures (with one regularized example having a signature (1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4)) are discussed in Appendix C in detail. After studying multiple such signatures, we observed that they tend to repeat the same modules in contiguous uninterrupted sequences and the number of repetitions grows towards the end of the model. In other words, early modules use fewer repetitions. We call such pattern progressive.
Once the optimal signature is identified, we investigated whether this particular model architecture is truly optimal for solving a particular task. We compared networks with mixture weights to those with hand-designed signatures of three types: (a) sequential signatures (fL/KK ) ◦ · · · ◦ (f L/K 1 ) repeating each of K templates L/K times, where L is the number of layers; (b) cyclic signatures corresponding to (fK ◦· · ·◦f1)L/K and (c) random signatures. Our experimental results (for details see Appendix C) appear to indicate that mixture-weight results are comparable to those obtained with a sequential signature and overall outperform results for other considered model signatures. However, it remains an open question whether progressive signatures are actually optimal, or are an artifact of the modular network training.
In Table 1 we show our results of improving IMAGENET accuracy by increasing the number of layers while keeping the number of parameters intact. Due to hardware and time limitations we only experiment with progressive pattern and leave more detailed study for future work. More details on IMAGENET architecture is included in the Appendix.

heading: 4.3 MULTI-TASK AND TRANSFER LEARNING

text:
Multi-task learning Another potential application of compositional models is in multi-task learning, where we expect the computation to be partially shared across different tasks. In this application, each task Tν uses its own private set of mixture weights Ξν , but the template blocks F and input, output adapters are shared across all tasks. As the number of tasks grows, the total number of blocks in F should also be scaled up to achieve sufficiently high accuracy on all {Tν}. Also note that modular networks provide a convenient tool for training sets of tasks parameterized by additional arguments like those, for example, used in basic visual question answering modules (finding specific objects, transforming and filtering attention regions, etc.) (Hu et al., 2018; 2017a).
Notice that in its current form, this approach is a generalization of ADASHARE (Sun et al., 2019), in which reused modules always retain their relative order with respect to each other6. Since multitask training is not the primary application explored in this paper and the direct comparison is not possible without re-implementing the method, we do not compare our results with ADASHARE.
While the tasks {Tν} could be very diverse and along with conventional supervised tasks could also include unsupervised tasks, or even generative and image-to-image translation objectives, here we
6and cannot be reused later in the same network
will only consider a collection of image classification tasks. For our experiments, we chose a collection of natural image datasets including: IMAGENET, CIFAR-100, PLACES365, SUN397 and FOOD101. Here we used model depth multiplier % of 1 and a larger output adapter (see Section 4.1). We compared multiple different ways of co-training different classification tasks: (a) baseline model with or without per-task batch normalization β and γ variables (see linear patches in Mudrakarta et al. (2019)) and (b) model with mixture-weights with or without per-task batch normalization variables.
In our experiments summarized in Appendix D, we noticed that the mixture-weight model with K = 32 templates achieved a lower overall cross-entropy loss and thus fitted training data better than the baseline models. This did translate to a superior IMAGENET performance, but in some cases resulted in worse accuracies for simpler datasets, on which overfitting seemed to occur.
Self-Organization of Computational Blocks Perhaps one the most interesting aspects of networks trained with mixture modules is that they essentially involve self-organization of atomic computational blocks. The properties of emergent block configurations are insightful both when we train a model on a single task and when performing multi-task learning.
In the case of a multi-task learning, we discover that the location of blocks re-used across tasks agrees with our intuition regarding the nature of the task similarity. Specifically, if all tasks operate on a similar domain, first few layers of all models appear to share the same mixture weights (see Figure 2(a)) thus sharing early-stage image processing. For similar tasks operating on different image domains, we see the opposite situation (see Figure 2(b)), where the first few layers differ, but the late sample processing is shared across all tasks performing inference on the same dataset (EMNIST, MNIST and KMNIST in our experiments). In this case, first few layers essentially adapt network input to a particular common format shared across all related tasks and processed by the following layers.
Transfer learning A set of blocks F pretrained on a collection of tasks can later be used for transferring extracted knowledge onto a new task. Adjusting mixture weights to form a new model potentially gives us more flexibility than tuning logits or batch normalization parameters alone.
In our experiments, we used models pretrained on 5 datasets: IMAGENET, CIFAR-100, PLACES365, SUN397 and FOOD101 (see Appendix D). We then used a learned collection of templates F for transferring knowledge to different new tasks such as STANFORD DOGS, AIRCRAFT, CARS196 and EMNIST. We compared our results with the baseline obtained by fine-tuning logits and BN parameters of a baseline model trained with a common model backbone, but per-task logit layers and BN parameters (“Baseline with patches” in Table 7). Proper initialization of mixture weights Ξ proved to be extremely important for achieving best performance. Model initialized with random Ξ performed significantly worse than the models initialized with Ξ corresponding to one of the datasets in the original mixture. In the following, we will denote by Ξν a set of mixture weights used for a task ν in the original mixture-module network trained on a collection of datasets. Initializing the model at some particular Ξν , we compared model performance with and without fine-tuning Ξ (fine-tuning BN parameters and logits in both cases).
The final results are presented in Table 2. Experimental observations suggest that a proper choice of the dataset ν for Ξ = Ξν is extremely important: using Ξ values from the FOOD196 model resulted in a model that achieved the same > 99% training accuracy, but would frequently lead to a 10%− 20% smaller validation accuracy. We also observe that the validation accuracy improvement in the final model can be partly attributed to a higher validation accuracy on the chosen model Ξν and partly to the result of fine-tuning this Ξ ≈ Ξν value.

heading: 4.4 DOMAIN ADAPTATION

text:
In this section, we explore the use of compositional models for unsupervised multi-source domain adaptation which aims to minimize the domain shift between multiple labeled source domains and an unlabeled target domain. To our knowledge, we are the first to show that it is possible to perform domain adaptation with modular networks by simply changing the order of modules in the network. To illustrate the ability of compositional networks to perform distribution alignment, we adopted two existing unsupervised domain adaptation methods: Adversarial Discriminative Domain Adaptation (ADDA) by Tzeng et al. (2017) and Moment Matching for Multi-Source Domain Adaptation (M3SDA) by Peng et al. (2019). We performed experiments with two sets of datasets: digits datasets (MNIST, corrupted MNIST, SVHN and USPS) and DomainNet dataset introduced by Peng et al. (2019). The code used for the domain adaptation experiments can be found on our project page (to be added if accepted). The details of implementation of the compositional domain adaptation and a review of related work on domain adaptation can be found in the Appendix E.
Multi-source adversarial classification In the compositional adversarial classification method, the shared classifier is trained to perform accurate classification on all source datasets. Additionally, a domain discriminator is trained to distinguish between the late features of source and target domains (in our implementation, the output of the layer before the output adapter). As in the multitask learning scenario, the model contains separate mixture weights for each domain; the mixture weights
of source domains are trained along with templates to minimize the classification loss, while the target domain mixture weights are trained to ”fool” the domain discriminator by aligning the target feature with the source features.
Moment Matching As in the original paper introduced by Peng et al. (2019), the method performs domain adaptation by minimizing the distance between the source and target domain statistics (more details can be found in the original paper). In the compositional moment matching setting, only target domain mixture weights are trained to perform moment matching. In addition to that, the model has two logits layer instead of one which are used to perform minmax discrepancy optimization (see Peng et al. (2019) for more detail). In the compositional case, only the mixture weights are affected by the discrepancy minimization.
Results As we can see in the Table 3, fine-tuning only the mixture weights for the domain adaptation objective improves the classification accuracy on target domain. To compare the results of conversational domain adaptation, we trained the adversarial and moment matching models with an isometric model without weight sharing as a backbone, and for fair comparison we only fine-tuned the input adapter (Saito et al. (2019) point out that finetuning the first layers is appropriate for the case when the domain shift is mostly in the low-level visual features). The results show that although only fine-tuning mixture weights of the conversational model does not result in the same adaptation efficiency as the baseline probably due to the fact that mixture weights contain more than ten times parameters than the input adapter, such approach can still significantly increase the target accuracy.

heading: 5 DISCUSSION

text: In this paper, we introduced a novel approach to modular neural networks that decouples modules from their position in the network thus allowing reuse of modules not only across tasks, but also across layers. Our method beats the baseline on multitask learning with additional templates and achieves better transfer learning results on most of the datasets we explored, which we partly attribute to less aggressive parameter over-sharing and partly to the ability of our model to fine-tune the module order itself. We then apply our method to training parameter-efficient networks and show that we can scale SOTA IMAGENET classification models to achieve higher accuracy while keeping the number of parameters fixed. We also show that simple fine-tuning of mixture weights allows unsupervised domain adaptation improving model’s performance on target domain by a large margin. Since, by design, the compositional models divides the task into simple subtasks, it can provide greater understanding of the computation process. For example, we notice that co-training multiple tasks on the same image domain, all trained models share early image processing modules. At the same time, when the dataset contains multiple image domains of the otherwise identical task, early layers bring network activations to the common format, while the final stage of the computation defined by the sequence of participating modules is shared.