Finding relationships between multiple views of data is essential both in exploratory analysis and as pre-processing for predictive tasks. A prominent approach is to apply variants of Canonical Correlation Analysis (CCA), a classical method seeking correlated components between views. The basic CCA is restricted to maximizing a simple dependency criterion, correlation, measured directly between data coordinates. We introduce a new method that finds dependent subspaces of views directly optimized for the data analysis task of neighbor retrieval between multiple views. We optimize mappings for each view such as linear transformations to maximize cross-view similarity between neighborhoods of data samples. The criterion arises directly from the well-defined retrieval task, detects nonlinear and local similarities, measures dependency of data relationships rather than only individual data coordinates, and is related to well understood measures of information retrieval quality. In experiments the proposed method outperforms alternatives in preserving cross-view neighborhood similarities, and yields insights into local dependencies between multiple views.
1 INTRODUCTION
Finding dependent subspaces across views (subspaces where some property of data is statistically related or similar across views) is a common data analysis need, where Canonical Correlation Analysis (CCA) (Hotelling, 1936) is a standard unsupervised tool. Preprocessing to find dependent subspaces is useful both for prediction and for analysis: in predictive tasks, such subspaces help if non-dependent parts of each view may arise from noise and distortions. In some data analysis tasks, finding the dependent subspaces may itself be the main goal; for example in bioinformatics domains dependency seeking projections have been used to identify relationships between different views of cell activity (Tripathi et al., 2008; Klami et al., 2013); in signal processing a similar task could be identifying optimal filters for dependent signals of different nature, e.g., speech and the corresponding tongue movements of the speakers as in Westbury (1994).
Methods like CCA maximize simple correlations between data point coordinate features across the projected subspaces. However, in many data domains the coordinates may not be of main interest but rather the data relationships that they reveal. It is then of great interest to develop dependency seeking methods that directly focus on the data relationships. For example, consider a database of scientists, defined in one view by their level of interest in various research topics, and in another view by their level of interest in various hobbies. In a database like this, finding relationships of people is the common interest, e.g. to find nearest colleagues for a scientist, having the most similar (neighboring) research interests; or to find hobby partners having the most similar (neighboring) hobby interests; the question is then, can we predict the research colleagues from hobby partners or vice versa? Research topics and hobbies are very dissimilar views, and not all of their variation will be related, but we can try to find subspaces of research and hobby interests, so that research neighbors and hobby neighbors are as highly related as possible in those subspaces.
In this paper we propose a method that solves this task: we present a novel method for seeking dependent subspaces across multiple views, preserving neighborhood relationships of data. Our ∗Ziyuan Lin and Jaakko Peltonen contributed equally to the paper.
method directly maximizes the between-view similarity of neighborhoods of data samples, a natural measure for similarity of data relationships among the views. The method detects nonlinear and local dependencies, has strong invariance properties, is related to an information retrieval task of the analyst, and performs well in experiments.
Relating data items is one of the main elementary tasks in Shneiderman’s taxonomy of tasks in visual data analysis (Shneiderman, 1996). Our method is optimized for finding related (neighboring) data items, formulated as optimizing an information retrieval task. Since our method directly serves the task of relating data items (across views) in Shneiderman’s taxonomy, in this sense it arguably comes closer to needs of data analysts than maximizing some variant of coordinate correlation.
We find linear projections (linear subspaces) of views. Linear projections have advantages of simplicity and easy interpretability with respect to original data features. Even if projections are linear, the dependency criterion we optimize is flexible and detects nonlinear dependencies across views.
We present our method in Section 2, properties and extensions in Section 3, related work in Section 4, experiments in Section 5, and conclusions in Section 6.
2 METHOD: DEPENDENT NEIGHBORHOODS OF VIEWS
Our method finds similar neighborhood relationships between views. We define the neighborhood relationships and then discuss how to measure their cross-view similarity. Instead of hard neighborhoods where two points are or are not neighbors, we use more realistic probabilistic neighborhoods.
Assume input data items xi = (xi,1, . . . ,xi,NViews) have paired features xi,V in each view V . We consider transformations of each view by a mapping fV which is typically a dimensionality reducing transformation to a subspace of interest; in this paper, for simplicity and interpretability we use linear mappings fV (xi,V ) =W TV xi,V with W T V ∈ Rdim
low(V )×dimorig(V ) as the to-be-optimized parameters, where dimorig(V ) and dimlow(V ) are the number of dimensions of V and its subspace respectively. The local neighborhood of a data item i in any transformation of view V can be represented by the conditional probability distribution pi,V = {pV ( j|i; fV )} where pV ( j|i; fV ) tells the probability that data item j 6= i is picked as a representative neighbor of i; that is, the probability that an analyst who inspected item i will next pick j for inspection. The pV ( j|i; fV ) can be defined in several ways, as a decreasing function of distance dV (i, j; fV ) between features of i and j in view V . Here we define it by a simple exponential falloff with respect to squared distance of i and j, as
pV ( j|i; fV ) = exp(−d2V (i, j; fV )/σ2i,V ) · (∑ k 6=i exp(−d2V (i,k; fV )/σ2i,V ))−1 (1)
where σi,V sets the falloff rate around i in the view. We tried two simple ways to set the σi,V : one is as a fraction of maximal pairwise distance so σi,V = 0.05 ·max j,k ‖x j,V − xk,V‖, or alternatively, set
σi,V = (dimlow(V )/dimorig(V ))1/2 ·mean j,l=kNN( j)‖xl,V − x j,V‖ (2)
i.e., calculate the average distance between x j,V and its k-th nearest neighbor xl,V , then give the average a heuristic correction factor √ dimlow(V )/dimorig(V ) since the average distance is obtained from the original space yet σi,V is used in a subspace. We use the first simple σi,V for artificial data experiments and the more data-driven second σi,V from (2) with k = 5 for the other experiments. Both choices give good results. Other local choices to e.g. achieve a desired entropy are possible, see Venna et al. (2010). With linear mappings the probabilities become
pV ( j|i; fV ) = exp(−‖W TV (xi,V − x j,V )‖2/σ2i,V ) · (∑ k 6=i exp(−‖W TV (xi,V − xk,V )‖2/σ2i,V ))−1 (3)
where the matrix WV defines the subspace of interest for the view and also the distance metric within the subspace. Our method learns the mapping parameters WV for each view.
2.1 COMPARISON OF NEIGHBORHOODS ACROSS VIEWS
Neighborhoods represented as probability distributions can be compared by difference measures. We discuss two measures for different purposes, and their information retrieval interpretations.
Kullback-Leibler divergence. For two distributions p = {p( j)} and q = {q( j)}, the KullbackLeibler (KL) divergence is an information-theoretic asymmetric difference measure defined as
DKL(p,q) = ∑ j p( j)(log p( j)/q( j)) . (4)
The KL divergence is nonnegative and zero if and only if p = q. Traditionally it is interpreted to measure the amount of extra coding length needed when coding examples with codes generated for distribution q when the samples actually come from distribution p. We treat views symmetrically and compute the symmetrized divergence (DKL(p,q)+DKL(q, p))/2.
KL divergence is related to an information retrieval criterion: DKL(p,q) is the cost of misses in information retrieval of neighbors, when neighbors are retrieved using retrieval distribution q but they actually follow distribution p. DKL(p,q) is also the cost of false neighbors when neighbors are retrieved using p but they actually follow q. The relationships were shown in Venna et al. (2010) and used to compare a reduced-dimensional neighborhood to an original one; we use it in a novel way to compare neighborhoods across (transformed) views of data. The symmetrized divergence is the total cost of both misses and false neighbors when neighbors following the distribution in one transformed view are retrieved from the other transformed view with its neighbor distribution.
The value of the KL divergence can depend highly on differences between individual probabilities p( j) and q( j). A single missed neighbor can yield a high divergence value: for any index j if p( j) > ε for some ε > 0, DKL(p,q)→ ∞ as q( j)→ 0. In real-life multi-view data differences between views may be unavoidable, so we prefer a less strict measure focusing more on overall similarity of neighborhoods than severity of individual misses. We discuss such a measure below.
Angle cosine. A simple similarity measure between discrete distributions is the angle cosine between the distributions as vectors, that is, Cos(p,q) = (∑ j p( j)q( j))/ √ (∑ j(p( j))2)(∑ j(q( j))2),
which can be seen as the Pearson correlation coefficient between elements of p and q; it is thus a neighborhood correlation—a neighborhood based analogue of the coordinate correlation cost function of CCA.1 The angle cosine is bounded above and below: it has highest value 1 if and only if p = q and lowest value 0 if supports of p and q are nonoverlapping.
Similarity of neighborhoods by itself is not enough. The KL divergence and angle cosine (neighborhood correlation) measures only compare similarity of neighborhoods but not potential usefulness of the found subspaces. In high-dimensional data it is often possible to find subspaces where neighborhoods are trivially similar. For example, in data with sparse features it is often possible to find two dimensions where all data is reduced to a single value; in such dimensions neighborhood distributions would become uniform across all data since, hence any two such dimensions appear similar. To avoid discovering trivial similarities we wish to complement the measures of similarity between neighborhoods with terms favoring nontrivial (sparse) neighborhoods. A simple way to prefer sparse neighborhoods is to omit the normalization from neighborhood correlation, yielding
Sim(p,q) = ∑ j p( j)q( j) (5)
which is the inner product between the vectors of neighborhood probabilities. Unlike Cos(p,q), Sim(p,q) favors sparse neighborhoods: it has highest value 1 if and only if p= q and p( j)= q( j)= 1 for only one element j, and lowest value 0 if the supports of p and q are nonoverlapping.
The information retrieval interpretation is: Sim(p,q) is a proportional count of true neighbors from p retrieved from q or vice versa. If p has K neighbors with near-uniform high probabilities p( j) ≈ 1/K and other neighbors have near-zero probabilities, and q has L neighbors with high probability q( j) ≈ 1/L, then Sim(p,q) ≈ M/KL where M is the number of neighbors for which both p and q are high (retrieved true neighbors). Thus Sim(p,q) rewards matching neighborhoods and favors sparse neighborhoods (small K and L). One advantage of this formulation is to avoid matching two neighborhoods that seem to match only because they are highly uninformative: for example if p and q are both uniform over all neighbors, they have the same probability values and would be “similar” in a naive comparison of probability values, but both are actually simply uninformative about the choice of neighbors. Sim(p,q) would prefer a more sparse, more informative match, as desired.
1To make the connection exact, typically correlation is computed after substracting the mean from coordinates; for neighbor distributions of n data items, the mean neighborhood probability is the data-independent value 1/(n−1)2 which can be substracted from each sum term if an exact analogue to correlation is desired.
2.2 FINAL COST AND OPTIMIZATION TECHNIQUE
We wish to evaluate similarity of neighborhoods between subspaces of each view, and optimize the subspaces to maximize the similarity, while favoring subspaces having sparse (informative) neighborhoods for data items. We then evaluate similarities as Sim(pi,V , pi,U ) where pi,V = {pV ( j|i; fV )} is the neighborhood distribution around data item i in the dependent subspace of view V and fV is the mapping (parameters) of the subspace, and pi,U = {pU ( j|i; fU )} is the corresponding neighborhood distribution in the dependent subspace of view U having the mapping fU . As the objective function for finding dependent projections, we sum the above over each pair of views (U,V ) and over the neighborhoods of each data item i, yielding
C( f1, . . . , fNviews) = Nviews
∑ V=1
Nviews
∑ U=1,U 6=V
Ndata ∑ i=1 Ndata ∑ j=1, j 6=i pV ( j|i; fV )pU ( j|i; fU ) (6)
where, in the setting of linear mappings and neighborhoods with Gaussian falloffs, pV is defined by (3) and is parameterized by the projection matrix WV of the linear mapping.
Optimization. The function C( f1, . . . , fNviews) is a well-defined objective for dependent projections and can be maximized with respect to the projection matrices WV of each view. We use gradient techniques for optimization, specifically limited memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS). Even with L-BFGS, (6) can be hard to optimize due to several local optima. To find a good local optimum, we optimize over L-BFGS rounds with a shrinking added penalty driving the objective away from the worst local optima during the first rounds; we use the optimum in each round as initialization of the next. For the penalty we use KL divergence based dissimilarity between neighborhoods, summed over neighborhoods of all data items i and view pairs (U,V ), giving
CPenalty( f1, . . . , fNviews) = Nviews
∑ V=1
Nviews
∑ U=1,U 6=V
Ndata ∑ i=1 (DKL(pi,V , pi,U )+DKL(pi,U , pi,V )) 2
(7)
which is a function of all mapping parameters and can be optimized by L-BFGS; (7) penalizes severe misses (pairs (i, j) with nonzero neighborhood probability in one view but near-zero in another) driving the objective away from bad local optima. In practice KL divergence is too strict about misses; we use two remedies below.
Bounding KL divergence by neighbor distribution smoothing. To bound the KL divergence, one way is to give the neighbor distributions (1) a positive lower bound. In the spirit of the well-known Laplace smoothing in statistics, we revise the neighbor probabilities (1) as
pV ( j|i; fV ) = (exp(−d2V (i, j; fV )/σ2i,V )+ ε) · (∑ k 6=i exp(−d2V (i,k; fV )/σ2i,V )+(Ndata−1)ε)−1 (8)
where ε > 0 is a small positive number. To keep notations simple, we still denote this smoothed neighbor distribution as pV ( j|i; fV ). To avoid over-complicated formulation and for consistency, we also use this version of neighbor probabilities in our objective function (6), even though the value of the objective is bounded by itself. We simply set ε = 1e−6 which empirically works well. Shrinking the penalty. Even with bounded KL divergence, optimization stages need different amounts of penalty. At end of optimization, nearly no penalty is preferred, as views may not fully agree even with the best mapping. We shrink the penalty during optimization; the objective becomes
CTotal( f1, . . . , fNviews) =C( f1, . . . , fNviews)− γCPenalty( f1, . . . , fNviews) (9)
where γ controls the penalty. We initially set γ so the two parts of the objective function are equal for the initial mappings, C( f1, . . . , fNviews) = γCPenalty( f1, . . . , fNviews), and multiply γ by a small factor (0.9 in experiments) at the start of each L-BFGS round to yield exponential shrinkage.
Time complexity. We calculate the neighbor distributions for all views, and optimize the objective function involving each pairs of views, thus the naive implementation takes O(dN2dataN 2 views) time, with d the maximal dimensionality among views. Acceleration techniques (Yang et al., 2013; Van Der Maaten, 2014; Vladymyrov & Carreira-Perpinán, 2014) from neighbor embedding could be adopted to reduce time complexity of a single view from O(N2data) to O(Ndata logNdata) or even O(Ndata). But scalability is not our first concern in this paper, so we use the naive O(N2data) implementation for calculating the neighbor distributions for each view involved.
3 PROPERTIES OF THE METHOD AND EXTENSIONS
Information retrieval. Our objective measures success in a neighbor retrieval task of the analyst: we maximize count of retrieved true neighbors across views, and penalize by severity of misses. Invariances. For any subspace of any view, (1) and (3) depend only on pairwise distances and are thus invariant to global translation, rotation, and mirroring of data in that subspace. The cost is then invariant to differences of global translation, rotation, and mirroring between views and finds view dependencies despite such differences. If in any subspace the data has isolated subsets (where data pairs from different subsets have zero neighbor probability) invariance holds for local translation/rotation/mirroring of the subsets as long as they preserve the isolation. Dependency is measured between whole subspaces. Unlike CCA where each canonical component of one view has a particular correlated pair in the other view, we maximize dependency with respect to the entire subspaces (transformed representations) of each view, as neighborhoods of data depend on all coordinates within the dependent subspace. Our method thus takes into account within-view feature dependencies when measuring dependency. Moreover, dependent subspaces do not need to be same-dimensional, and in some views we can choose not to reduce dimensionality but to learn a metric (full-rank linear transformation). Finding dependent neighborhoods between feature-based views and views external neighborhoods. In some domains, some data views may directly provide neighborhood relationships or similarities between data items, e.g., friendships in a social network, followerships in Twitter, or citations between scientific papers. Such relationships or similarities can be used in place of the feature-based neighborhood probabilities pV ( j|i; fV ) above. This shows an interesting similarity to a method (Peltonen, 2009) used to find similarities of one view to an external neighborhood definition; our method contains this task as one special case. Other falloffs. Exponential falloff in (1) and (3) can be replaced with other forms like t-distributed neighborhoods (van der Maaten & Hinton, 2008). Such replacement preserves the invariances. Other transformations. Our criterion is extensible to nonlinear transformations in future work; replace linear projections by another parametric form, e.g. neural networks, optimize (9) with respect to its parameters; the transformation can be chosen on a view-by-view basis. Optimization difficulty of transformations varies; the best form of nonlinear transformation is outside the paper scope.
4 RELATED WORK
In general, multi-view learning (Xu et al., 2013) denotes learning models by leveraging multiple potentially dependent data views; such models could be built either for unsupervised tasks based on the features in the views or for supervised tasks that involve additional annotations like categories of samples. In this paper we concentrate on unsupervised multi-view learning, and our prediction tasks of interest are predicting neighbors across views.
The standard Canonical Correlation Analysis (CCA) (Hotelling, 1936) iteratively finds component pairs maximizing correlation between data points in the projected subspaces. Such correlation is a simple restricted measure of linear and global dependency. To measure dependency in a more flexible way and handle nonlinear local dependency, linear and nonlinear CCA variants have been proposed: Local CCA (LCCA, Wei & Xu 2012) seeks linear projections for local patches in both views that maximize correlation locally, and aligns local projections into a global nonlinear projection; its variant Linear Local CCA (LLCCA) finds a linear approximation for the global nonlinear projection; Locality Preserving CCA (LPCCA, Sun & Chen 2007) maximizes reweighted correlation between data coordinate differences in both views. In experiments we compare to the well known traditional CCA and LPCCA as an example of recent state of the art.
As a more general framework, Canonical Divergence Analysis (Nguyen & Vreeken, 2015) minimizes a general divergence between distributions of data coordinates in the projected subspace.
The methods mentioned above work on data coordinates in the original spaces. There are also nonlinear CCA variants (e.g., Lai & Fyfe 2000; Bach & Jordan 2003; Verbeek et al. 2003; Andrew et al. 2013; Wang et al. 2015; Hodosh et al. 2013) for detecting nonlinear dependency between multiple views. Although some variants above are locality-aware, they introduce locality from the original space before maximizing correlation or other similarity measures in the low-dimensional
subspaces. Since locality in the original space may not reflect locality in the subspaces due to noise or distortions, such criteria may not be suited for finding local dependencies in subspaces.
The CODE method (Globerson et al., 2007) creates an embedding of co-occurrence data of pairs of original categorical variables, mapping both variables into a shared space. Our method is not restricted to categorical inputs – its main applicability is to high-dimensional vectorial data, with several other advantages. In contrast to CODE, we find dependent subspaces (mappings) from multiple high-dimensional real-valued data views. Instead of restricting to a single mapping space we find several mappings, one from each view, which do not need to go into the same space; our output spaces can even be different dimensional for each view. Unlike CODE our method is not restricted to maximizing coordinate similarity: we only need to make neighborhoods similar which is more invariant to various transformations.
The above methods and several in Xu et al. (2013) all maximize correlation or alternative dependency measures between data coordinates across views. As we pointed out, in many domains coordinates are not of main interest but rather the data relationships they reveal; we consider neighborhood relationships and our method directly finds subspaces having similar neighborhoods.
5 EXPERIMENTS
We demonstrate our method on artificial data with multiple dependent groups between views, and three real data sets: a variant of MNIST digits (LeCun & Cortes, 2010), video data, and stock prices. In this paper we restrict our method to find linear subspaces, important in many applications for interpretability, and compare with two prominent linear subspace methods for multiple views, CCA and LPCCA. To our knowledge, no other information retrieval based approaches for finding linear subspaces is known up to the time when we did the experiment. Future work could compare methods for nonlinear mappings (Xu et al., 2013) to variants of ours for the same mapping; we do not focus on the mapping choice, and focus on showing the benefit or our neighborhood based objective.
On the artificial data set, we measure performance by correspondence between found projections and the ground truth. On the real data we use mean precision-mean recall curves, a natural performance measure for information retrieval tasks, and a measure for dependency as argued in Section 2.
Experiment on artificial data sets. We generate an artificial data set with 2 views with multiple dependent groups in each pair of corresponding dimensions as follows. Denote view V (∈ {1,2}) as X (V ) ∈ R5×1000, and its i-th dimension as X (V )i . For each i, we divide the 1000 data points in that dimension into 20 groups {gi j}20j=1 with 50 data points each. For each gi j and view V , we let x̂(V )i jk , Fi jm (V ) i j + εi jk (1 ≤ k ≤ 50), with m (V ) i j ∼N (0,5), εi jk ∼U [−0.5,0.5] and Fi j ∈ {−1,1} a random variable allowing positive or negative correlation inside the group. We assemble x̂(V )i jk into X̂ (V ) ∈ R5×1000, and randomly permute entries of X̂ (1)i and X̂ (2) i in the same way but differently for different i, to ensure cross-dimension independency. Lastly we perform a PCA between X̂ (1)i and X̂ (2)i for each i, to remove cross-dimension correlation. We assemble the resulting X (V ) i into X (V ).
We pursue 2 transformations mapping from the 5D original space to a 1D latent space for each of the two views. Ground truth projections for both views will then be W (i) = (δi j)5j=1 ∈R1×5. Results are in Fig. 1: compared with CCA, our method successfully finds one of the ground truth transformations (= the 5th one), despite mirroring and scale, recovering the between-view dependency.
We measure performance by correspondence between the found projections and the ground truth transformation: let W1,W2 ∈ R1×5 be projections found by a method, define
Corr(W1,W2) = max i
( |W (i)W T1 |/‖W1‖2 + |W (i)W T2 |/‖W2‖2 ) /2 (10)
as the correspondence score. High score means good match between the found projections and ground truth. We repeat the experiment calculating correspondence on 20 artificial data sets generated in the same way. The table in Figure 1 summarizes the statistics. Our method outperforms CCA and LPCCA (with k = 5), finding the dependency on all 20 data sets.
Experiment on real data sets with two views. We show our method helps match neighbors between the subspaces of two views after transformation. We use the following 3 data sets.
MNIST handwritten digit database (MNIST). MNIST contains 70000 gray-scale hand-written digit images of size 28× 28. We create a training set and a testing set with 2000 images each. In the training set, we randomly choose 200 images from each digit to balance the distribution, while the testing set is another 2000 random images without balancing. We apply nonlinear dimensionality algorithm on both the left half and the right half of the images to create the two views to simulate a scenario where views have nonlinear dependency; we use Neighbor Retrieval Visualizer (NeRV, Venna et al. 2010) embedding to 5 dimensions with λ = 0.1 and λ = 0.9 respectively. The experiment is repeated 17 times, covering 68000 images.
Image patches from video (Video). We take image patches from a subset of frames in a video (db.tt/rcAS5tII). Starting from frame 50, we take 5 consecutive frames as a short clip at every 50 frames until frame 5200, then create two views from image patches in two fixed rectangles in those frames, rect1 = [800,900]× [250,350] and rect2 = [1820,1920]× [800,900]. We measure 5-fold cross-validation performance after randomly permuting the clips.
Stock prices (Stock), from the Kaggle “Winton stock market challenge” (goo.gl/eqdhKK). It contains prices of a stock at different times. We split the time series in the given training set into two halves, and let view 1 be the amplitudes from the Fourier transform results of the first half, and view 2 be the phases from the Fourier transform results of the second half.
For each data set we seek a pair of transformations onto 2D subspaces for the views. We measure performance by mean precision-mean recall curves of neighbor retrieval between 1) the two subspaces from the transformations, and 2) one of the original views and the subspace from the transformation for the other view. The better the performance is, the more to the top (better mean precision) and right (better mean recall) the curve will be in the figure. We set the number of neighbors in the ground truth as 5 for MNIST and Stock, 4 for Video, and let the number of retrieved neighbors vary from 1 to 10 as we focus on the performance of the matching for the nearest neighbors. We compare the performance with CCA and LPCCA. Figure 2 (column 1–3) shows the results.
We now show our method can find dependent subspaces for multiple (more than two) views. In this experiment we use Cell-Cycle data with five views. The views are from different measurements of cell cycle-regulated gene expression for the same set of 5670 genes (Spellman et al., 1998). We preprocess data as in Tripathi et al. (2008) with an extra feature normalization step. We seek five two-dimensional subspaces from the five views, comparing to the PCA baseline with 2 components. We again use mean precision-mean recalls curves as the performance measure, additionally average the curves across the 10 view pairs or view-transformed coordinate pairs, besides averaging over the five repetitions in 5-fold cross-validation. Figure 2 (column 4) shows we outperform the baseline.
Finding subspaces with different dimensions. We show our method can find dependent subspaces with different dimensions. We create three two-dimensional Lissajous curves Lk(t) = (cos √ 2k−1t +2π(k−1)/3, cos √
2kt + 2π(k− 1)/3), k = 1,2,3. We create the first view X (1) ∈ R6×1000 as X (1)1,1:1000 = (0, · · · ,999) and X (1) d≥2,1:1000
i.i.d∼ N (0,1), and the second view X (2) ∈R6×1000 as the concatenation of the coordinates in the Lissajous curves. We seek a one-dimensional subspace
from X (1), and a two-dimensional subspace from X (2); the aim is to find the nonlinear dependency between one-dimensional timestamps, and a two-dimensional representation for the three trajectories summarizing the two-dimensional movements of the three points along Lissajous curves. Figure 3 shows the Lissajous curves, found subspaces, and optimized projection pair. Our method successfully finds the informative feature in X (1), and keeps transformed coordinates of X (2) smooth, with roughly the same amount of “quick turns” as in original Lissajous curves. The magnitudes in the optimized projections also suggest they capture the correct dependency.
6 CONCLUSIONS
We presented a novel method for seeking dependent subspaces across multiple views, preserving neighborhood relationships of data. It has strong invariance properties, detects nonlinear dependencies, is related to an information retrieval task of the analyst, and performs well in experiments.

The authors develop a way learn subspaces of multiple views such that data point neighborhoods are similar in all of the views.  This similarity is measured between distributions of neighbors in pairs of views. The motivation is that this is a natural criterion for information retrieval.
I like the idea of preserving neighborhood relationships across views for retrieval tasks. And it is nice that the learned spaces can have different dimensionalities for different views.  However, the empirical validation seems preliminary.
The paper has been revised from the authors' ICLR 2016 submission, and the revisions are welcome, but I think the paper still needs more work in order to be publishable.  In its current form it could be a good match for the workshop track.
The experiments are all on very small data sets (e.g. 2000 examples in each of train/test on the MNIST task) and not on real tasks.  The authors point out that they are not focusing on efficiency, and presumably computation requirements keep them from considering larger data sets.  However, it is not clear that there is any conclusion that can be drawn that would apply to more realistic data sets.  Considering the wealth of work that's been done on multi-view subspace learning, with application to real tasks, it is very hard to see this as a contribution without showing that it is applicable in such realistic settings.
On a more minor point, the authors claim that no other information retrieval based approaches exist, and I think this is a bit overstated.  For example, the contrastive loss of Hermann & Blunsom "Multilingual models for compositional distributed semantics" ACL 2014 is related to information retrieval and would be a natural one to compare against.
The presentation is a bit sloppy, with a number of vague points and confusing wordings.  Examples:
- the term "dependency" gets used in the paper a lot in a rather colloquial way.  This gets confusing at times since it is used in a technical context but not using its technical definition.
- "an information retrieval task of the analyst": vague and not quite grammatical
- "the probability that an analyst who inspected item i will next pick j for inspection" is not well-defined
- In the discussion of KL divergence, I do not quite follow the reasoning about its relationship to the "cost of misses" etc.  It would help to make this more precise (or perhaps drop it?  KL divergence is pretty well motivated here anyway).
- Does C_{Penalty} (7) get added to C (6), or is it used instead?  I was a bit confused here.
- It is stated that CCA "iteratively finds component pairs".  Note that while CCA can be defined as an iterative operation, it need not (and typically is not) solved that way, but rather all projections are found at once.
- How is PCA done "between X_i^1 and X_i^2"?
- "We apply nonlinear dimensionality algorithm": what is this algorithm?
- I do not quite follow what the task is in the case of the image patches and stock prices.
Other minor comments, typos, etc.:
- The figure fonts are too small.
- "difference measures" --> "different measures"
- "...since, hence any two...": not grammatical
- "between feature-based views and views external neighborhoods": ?


BEYOND FINE TUNING: A MODULAR APPROACH TO LEARNING ON SMALL DATA
In this paper we present a technique to train neural network models on small amounts of data. Current methods for training neural networks on small amounts of rich data typically rely on strategies such as fine-tuning a pre-trained neural network or the use of domain-specific hand-engineered features. Here we take the approach of treating network layers, or entire networks, as modules and combine pre-trained modules with untrained modules, to learn the shift in distributions between data sets. The central impact of using a modular approach comes from adding new representations to a network, as opposed to replacing representations via fine-tuning. Using this technique, we are able surpass results using standard fine-tuning transfer learning approaches, and we are also able to significantly increase performance over such approaches when using smaller amounts of data.
1 INTRODUCTION
Training generalizable models using only a small amount of data has proved a significant challenge in the field of machine learning since its inception. This is especially true when using artificial neural networks, with millions or billions of parameters. Conventional wisdom gleaned from the surge in popularity of neural network models indicates that extremely large quantities of data are required for these models to be effectively trained. Indeed the work from Krizhevsky et al. (2012) has commonly been cited as only being possible through the development of ImageNet (Russakovsky et al. (2015)). As neural networks become explored by practitioners in more specialized domains, the volume of available labeled data also narrows. Although training methods have improved, it is still difficult to train deep learning models on small quantities of data, such as only tens or hundreds of examples.
The current paradigm for solving this problem has come through the use of pre-trained neural networks. Bengio et al. (2012) were able to show that transfer of knowledge in networks could be achieved by first training a neural network on a domain for which there is a large amount of data and then retraining that network on a related but different domain via fine-tuning its weights. Though this approach demonstrated promising results on small data, these models do not retain the ability to function as previously trained. That is, these models end up fine tuning their weights to the new learning task, forgetting many of the important features learned from the previous domain.
The utility of pre-training models extends beyond training on small data. It is also used as an effective initialization technique for many complicated models (Jaderberg et al. (2015); Lakkaraju et al. (2014)). This, in addition to the continuing trend of treating specific network layer architectures as modular components to compose more advanced models (He et al. (2015); Larsson et al. (2016); Szegedy et al. (2015); Abadi et al. (2016)) informs our work as we seek to use pre-trained models as ∗Seattle, WA †Richland, WA; AA and NOH contributed equally
an architectural framework to build upon. Instead of overwriting these models and fine-tuning the internal representations to a specific task, we propose composing pre-trained models as modules in a higher order architecture where multiple, potentially distinct representations contribute to the task. With this approach, useful representations already learned are not forgotten and new representations specific to the task are learned in other modules in the architecture.
In this paper we present our neuro-modular approach to fine-tuning. We demonstrate how modules learn subtle features that pre-trained networks may have missed. We quantitatively compare traditional fine-tuning with our modular approach, showing that our approach is more accurate on small amounts of data (<100 examples per class). We also demonstrate how to improve classification in a number of experiments, including CIFAR-100, text classification, and fine-grained image classification, all with limited data.
2 RELATED WORK
Transferring knowledge from a source domain to a target domain is an important challenge in machine learning research. Many shallow methods have been published, those that learn feature invariant representations or by approximating value without using an instance’s label (Pan & Yang (2010); Sugiyama et al. (2008); Pan et al. (2011); Zhang et al. (2013); Wang & Schneider (2014); Gong et al. (2016)). More recent deep transfer learning methods enable identification of variational factors in the data and align them to disparate domain distributions (Tzeng et al. (2014); Long et al. (2015); Ganin & Lempitsky (2014); Tzeng et al. (2015)). Mesnil et al. (2012) presents the Unsupervised and Transfer Learning Challenge and discusses the important advances that are needed for representation learning, and the importance of deep learning in transfer learning.Oquab et al. (2014) applied these techniques to mid-level image representations using CNNs. Specifically, they showed that image representations learned in visual recognition tasks (ImageNet) can be transferred to other visual recognition tasks (Pascal VOC) efficiently. Further study regarding the transferability of features by Yosinski et al. (2014) showed surprising results that features from distant tasks perform better than random features and that difficulties arise when optimizing splitting networks between co-adapted neurons. We build on these results by leveraging existing representations to transfer to target domains without overwriting the pre-trained models through standard fine-tuning approaches.
Long et al. (2015) developed the Deep Adaptation Network (DAN) architecture for convolutional neural networks that embed hidden representations of all task-specific layers in a reproducing kernel Hilbert space. This allows the mean of different domain distributions to be matched. Another feature of their work is that it can linearly scale and provide statistical guarantees on transferable features. The Net2Net approach (Chen et al. (2015)) accelerates training of larger neural networks by allowing them to grow gradually using function preserving transformations to transfer information between neural networks. However, it does not guarantee that existing representational power will be preserved on a different task. Gong et al. (2016) consider domain adaptation where transfer from source to domain is modeled as a causal system. Under these assumptions, conditional transferable components are extracted which are invariant after location-scale transformations. Long et al. (2016) proposed a new method that overcomes the need for conditional components by comparing joint distributions across domains. Unlike our work, all of these require explicit assumptions or modifications to the pre-trained networks to facilitate adaptation.
We note that while writing this paper, the progressive network architecture of Rusu et al. (2016) was released, sharing a number of qualities with our work. Both the results we present here and the progressive networks allow neural networks to extend their knowledge without forgetting previous information. In addition, Montone et al. (2015) discusses a semi-modular approach. Montone et al. also froze the weights of the original network, although it did not focus on the small data regime, where only a few tens of examples could be available. However, our modular approach detailed here focuses on leveraging small data to adapt to different domains. Our architecture also complements existing network building strategies, such as downloading pre-trained neural networks to then be fine-tuned for domain adaptation.
3 MODULAR ARCHITECTURE
Generically, modular neural networks are directed graphs of pre-trained networks linked together with auxiliary, untrained networks. Depicted in Fig. 1, one only trains the new components of the network. The architecture could take the form of simply placing two networks in parallel (the twotowers approach), shown in Fig. 1. In addition, the architecture could interleave the modules with the layers of the pre-trained network (the stitch approach), shown in Fig. 2
This allows the network as a whole to retain the original representational power of the pre-trained network. Thus, our modular approach bounds from below the performance of transfer learning. Here, we explore some of the properties of these modular architectures, including how they learn new representations and how they perform on small amounts of data.
3.1 LEARNED FILTERS
In the case of convolutional networks, we posit that adding modules to networks helps them learn new domains because the original modules contribute well-trained filters, allowing the untrained modules to learn more subtle features that may perhaps be more discriminating. Even slight regularization on the module network will encourage the network to avoid redundancy with the base network.
To visualize images that maximally stimulate each filter, we followed the approach of Zeiler & Fergus (2014). We set the objective function to be the activation of the filter we were querying. We then conducted back-propagation. Instead of using the gradients to alter the weights, we used the gradients at the input layer to alter the pixels themselves. We initialized with an image of noise smoothed with a Gaussian filter of radius 1. The gradient was normalized, so the input image, X was updated according to Xt+1 = Xt + 0.01 ∗ ∇/|∇|, where ∇ is the induced gradient at the input layer. This was repeated 500 times, at which point the image largely had converged.
After training a simple neural network on MNIST with 3 convolutional layers, (8 × 8 × 8) − maxpool2 − (8 × 4 × 4) − (8 × 3 × 3) − Dense128, which was done using ADAM Kingma & Ba (2014) and augmenting the images with 10% shifts and zooms, we reached an accuracy of 98.8%. We then added an even simpler module to the neural network, (4 × 8 × 8) −maxpool2 − (4×4×4)− (4×3×3)−Dense32. This module is trained on the same input as the original model but it is tied together with the output features of the original model, as illustrated in Fig. 1. After training the module, the combined network achieves 99.2% accuracy. The models were intentionally kept small, with the original model only having 8 filters per layer, and the module only having 4 filters per layer.
As we can see in Fig. 3, the module does not learn filters that merely duplicate the original network. As is common, the first layer learns typical edge and orientation detectors, but the module is more sensitive to high-frequency diagonal components and details around the edge of the image. In the second layer, we see that the module is sensitive to diagonal components near the boundary. And the third layer shows that the module has indeed concentrated its effort on detecting strokes near the edge of the image. As we can see from inspecting Figure 3c, while the original network concentrated its efforts on the center of the images (as it should), the module was then able to focus more around the edges of the image and catch some of the mistakes made by the original network.
3.2 SMALL DATA
Although the modular approach can be used to extend and improve a network on its original task, its value comes from its ability to facilitate transfer learning. If a network has been trained on thousands or even millions of examples and hand-tuned for weeks or months, one would not want to throw away this valuable representational power by training the network with 100 examples from an out-of-domain dataset. Instead, the modular approach keeps the original, unaltered network, in addition to learning supplementary representations specific to the distribution of the new data.
This allows the modular approach to more robustly handle small data sets than naive fine-tuning. To demonstrate this, we trained a network on CIFAR-10 and used it to apply to CIFAR-100 for varying
amounts of training data. The CIFAR-10 network was trained until it was 88.9% accurate, using the network in He et al. (2016) with 3 residual units, for a total of 28 layers.
We then compared two approaches. For the first approach, we simply fine tuned the CIFAR-10 network by using training data from the CIFAR-100 dataset and replacing the final softmax. Second, we froze the original CIFAR-10 network and added an identical copy as a module, which would be trained on the same batches of data as the first approach. That is, we have: Network 1 – fine-tuning the base network and Network 2 – freezing the base network and fine-tuning a module. This doubles the amount of weights in the second network, but Network 1 and Network 2 have an identical number of weights to be trained and those weights have the same starting value. More formally, we present these two approaches in equations 1 and 2 below.
yft = softmax(NN(x;w0 = {C10})) (1) ymod = softmax([NN(x;w? = {C10}), NN(x;w0 = {C10})]) (2)
where yft denotes predictions made from a fine-tuned network and ymod denotes predictions made from our modular architecture. NN denotes the neural network without softmax activation trained on CIFAR-10, and w0 is the initialization of the weights, which are learned from training on CIFAR10, i.e. w0 = {C10}. Note that in our modular architecture pre-trained weights are locked as denoted by w? = {C10} in Equation2, i.e.,∇wNN(w?) ≡ 0. To train, we used the ADAM optimization algorithm (Kingma & Ba (2014)). We added an activity L2 regularization of 1e−6 to the module to help break degeneracy. We used batches of 200, where each batch contained two images per class. Each batch was iterated over five times, before the next batch was used. This iteration allowed simulating multiple epochs over small data. We recorded the results of the performance on the test set after each batch, in Fig. 4a.
We observe that for all amounts of training data, but particularly for small amounts of training data, the modular approach outperforms traditional fine-tuning. Of course, we chose to make the module
a complete copy of the original CIFAR-10 network. This ensured we could compare with the same number of weights, same initialization, same data, etc. Further research will certainly reveal more compact module networks that outperform our example.
4 EXPERIMENTS
4.1 TRANSFER LEARNING FROM CIFAR-10 TO CIFAR-100 WITH STITCH NETWORKS
To investigate the effectiveness of modular networks for transfer learning, we explore a second example of transfer learning from CIFAR-10 in order to model CIFAR-100. As we were able to show above, a modular network is able to outperform traditional fine-tuning because it learns additional features that may complement those captured by the pre-trained model. However, there is no reason why a module needs to only accept input from the input layer nor a reason why it needs to send its output directly to the softmax layer. Here, we describe stitch networks, where the modules are actually interwoven with the original network.
We believe that in modular networks, the untrained module learns representations that capture the difference from the original distribution of data to the distribution of data under the new task. Expanding upon this idea, instead of learning the shift in distributions only at the softmax layer as with our other modular networks, we integrate the signal from the learned modules much more tightly with the paired untrained modules by using a Stitch Network. Use of the Stitch Network allows for the model to learn to correct the distribution difference after each transformation made by the learned module, shown in Fig. 2.
The stitch network we explore is comprised of layer pairs between a single learned and unlearned module. The learned module is a five layer convolutional neural network where the first two layers are 3x3 convolutional layers with 32 filters, followed by max pooling and two more 3x3 convolutions with 64 filters. The convolutions are followed by a fully connected layer with 512 outputs and finally a softmax for classification. This model is pre-trained on CIFAR-10 then stripped of the softmax layer, has its weights locked and is then used as the learned module for the stitch network. The untrained module is composed in a similar fashion, with four 3x3 convolutions with maxpooling, and a fully connected layer each with 1/4 the number of outputs as the corresponding pre-trained layers. The outputs of each layer pair are concatenated and fed as the input for each proceeding layer of the untrained module. Both modules feed into the final softmax layer of the composite network which then classifies over the new data set. A sketch of this is shown in Fig. 2
Dense Layer (512) Feature Extraction Layer
Softmax Layer
}
} }}
Input Car Image
Dense Layer (196)}
We train our entire composite model on a randomly selected subset of ten CIFAR-100 classes. We compare the accuracy over the validation set of the selected classes against traditional fine-tuning using only the learned module, as well as against an uninitialized version of the learned module. We were additionally interested in comparing across all models the effect of limiting the amount of data available for training. We repeat the experiment with the same subset of classes, varying the amount of available training data such that the networks are shown only a fraction of each class for training. Note that there are 500 available training examples per class in CIFAR-100.
We find by using the stitch network, we are able to match or improve upon classification results (Fig. 5) obtained using traditional fine-tuning over a pre-trained model. We also outperform training from scratch, regardless of the amount of training data used. We note that we find significant gains over traditional methods as the number of available training examples drops below 200 examples per class.
4.2 STANFORD CARS DATA SET
The Stanford Cars data set (Krause et al. (2013)), which features 16,185 images of 196 classes of cars, is an example of a data set for fine-grained categorization. Rather than train a classifier to distinguish between fundamentally different objects like horses and planes, as required in the Large Scale Visual Recognition Challenge (Russakovsky et al. (2015)), fine-grained categorization requires the classifier to learn subtle differences in variations of the same entity. For example, a classifier trained on the Stanford Cars data set would have to learn distinguishing features between a BMW X6 SUV from 2012 and an Isuzu Ascender SUV from 2008.
In this research two models are trained on the Stanford Cars data set. Both models utilize a transfer learning approach by leveraging the non-fully connected output from the VGG16 model (Simonyan & Zisserman (2014)). The “fine-tuned” model passes the VGG16 features to a fully connected layer of length 512 followed by a softmax layer of length 196, as seen in Fig. 6. Gradient descent via RMSPROP is used to train the dense layers. The “module” model merges the fixed VGG16 features with a ResNet (He et al. (2015)) model, whose output is then fed to two consecutive dense layers of length 256 capped by a softmax layer of length 196. The module model architecture is shown in Fig. 7. Again, RMSPROP is used to train ResNet and post-merge dense layer weights, but the VGG features are unchanged.
As seen in Fig. 4b, after 50 epochs the module model appears to significantly outperform the finetuned model in validation accuracy. However, it should be noted that while the module model carries
19,537,990 trainable parameters the fine-tuned model only has 12,946,116 parameters. Furthermore, no hyperparameter optimization is performed on either model.
4.3 MODULE FOR LSTM TEXT CLASSIFICATION
We further investigate the effects of our modular network approach by applying this method to a different modeling problem - text classification. Similar to image data, text represents an unstructured data-type that often exhibits long-term and interconnected dependencies within the data that are difficult to model with simpler classifiers. Whereas in the case of images neighboring pixels may represent semantically related concepts or objects, in text words may exhibit long-term semantic or syntactic dependencies that can be modeled sequentially. These characteristics make text classification particularly well-suited to recurrent neural networks such as long short-term memory (LSTM) networks, but these learning methods typically require a great deal of data to be learned efficiently and to avoid overfitting.
To test our methodology, we evaluate a modular recurrent network against two individual recurrent neural networks on the IMDB sentiment dataset. Previous work has shown deep learning methods to be effective at sentiment classification performance on this dataset (Maas et al. (2011)), however we add to this past work by presenting an analysis that demonstrates the effectiveness of modular networks in the case of extremely small training sets. To this end, we sample only 500 training examples from the original 25,000 available in the full training set, and evaluate on the full 25,000 validation examples. We use the same 500 training examples for each model evaluated in our experiments for consistency, and report accuracy for each model on the full validation set.
We evaluate three models in our text-classification experiments, two of which are individual recurrent networks and the final which is our modular recurrent network. The first model consists of three layers - an initial layer that projects sequences of words into an embedding space, a second LSTM layer with 32 units, and a final sigmoid layer for computing the probability of the text belonging to the positive class. Our second model is identical to the first except that we fix the weights of the embedding layer using pre-trained GloVe word vectors1. In particular, we use 100-dimensional vectors computed from a 2014 version of Wikipedia.
Finally, we detail our modular network, which leverages both individual recurrent neural networks described above. To construct our modular network, we take the embedding and LSTM layers from our individual networks, and concatenate the output of both LSTM layers into a single tensor layer in the middle of our modular network. Additionally, we modify the output of each of these component LSTM layers by forcing each to output a weight matrix that tracks the state of the LSTM layer across all timesteps throughout the dataset. In this way, we seek to fully leverage the sequential dependencies learned by this layer, and this method outperforms the simpler alternative method of simply outputting the final state of each of the LSTM layers. We then feed this concatenated layer to a gated recurrent unit (GRU) layer with a sigmoid activation function for calculation of class probabilities. We experimented with an LSTM and densely connected layers after the tensor concatenation layer, but found best performance with the GRU. All models were optimized with the ADAM algorithm, and trained for 15 epochs. An outline of this architecture can be seen in Figure 8.
Here, we report results for our classification experiments with the three networks described above. We see an accuracy of 61.9% for our first model which is trained directly from the data without any pre-training. This is significantly lower than previously reported results, however we are training on only 2% of the available data to test our method’s application to small training sets. We see slightly better performance in terms of accuracy (64.9%) from our second model initialized with GloVe vectors. This seems to indicate that despite being trained on more formally written language in Wikipedia, these vectors can still boost performance on a task modeling text that is inherently subjective and opinion-based. Finally, we see an accuracy of 69.6% from our modular network, an increase of almost 5% accuracy over the next best performing model. Because weight initializations of recurrent networks can greatly affect model performance, we ran the classification experiments with our modular network 10 times, and report the average accuracy across these 10 runs. As can be seen here, our modular approach improves on the best performing individual network suggesting that
1http://nlp.stanford.edu/projects/glove/
this approach is useful in the domain text classification, and that our modular approach overcomes the poor performance shown by one of its component models.
5 CONCLUSIONS
We have presented a neuro-modular approach to transfer learning. By mixing pre-trained neural networks (that have fixed weights) with networks to be trained on the specific domain data, we are able to learn the shift in distributions between data sets. As we have shown, often the new modules learn features that complement the features previously learned in the pre-trained network. We have shown that our approach out-performs traditional fine-tuning, particularly when the amount of training data is small – only tens of examples per class. Further research will explore more efficient architectures and training strategies, but we have demonstrated that our approach works well for MNIST, CIFARs, the Stanford Cars dataset, and IMDB sentiment. Thus, the modular approach will be a valuable strategy when one has a large pre-trained network available but only a small amount of training data in the transfer task.
ACKNOWLEDGMENTS
This work was supported by the US Government.

This paper proposes a method of augmenting pre-trained networks for one task with an additional inference path specific to an additional task, as a replacement for the standard “fine-tuning” approach.
Pros:
-The method is simple and clearly explained.
-Standard fine-tuning is used widely, so improvements to and analysis of it should be of general interest.
-Experiments are performed in multiple domains -- vision and NLP.
Cons:
-The additional modules incur a rather large cost, resulting in 2x the parameters and roughly 3x the computation of the original network (for the “stiched” network).  These costs are not addressed in the paper text, and make the method significantly less practical for real-world use where performance is very often important.
-Given these large additional costs, the core of the idea is not sufficiently validated, to me.  In order to verify that the improved performance is actually coming from some unique aspects of the proposed technique, rather than simply the fact that a higher-capacity network is being used, some additional baselines are needed:
(1) Allowing the original network weights to be learned for the target task, as well as the additional module.  Outperforming this baseline on the validation set would verify that freezing the original weights provides an interesting form of regularization for the network.
(2) Training the full module/stitched network from scratch on the *source* task, then fine-tuning it for the target task.  Outperforming this baseline would verify that having a set of weights which never “sees” the source dataset is useful.
-The method is not evaluated on ImageNet, which is far and away the most common domain in which pre-trained networks are used and fine-tuned for other tasks.  I’ve never seen networks pre-trained on CIFAR deployed anywhere, and it’s hard to know whether the method will be practically useful for computer vision applications based on CIFAR results -- often improved performance on CIFAR does not translate to ImageNet.  (In other contexts, such as more theoretical contributions, having results only on small datasets is acceptable to me, but network fine-tuning is far enough on the “practical” end of the spectrum that claiming an improvement to it should necessitate an ImageNet evaluation.)
Overall I think the proposed idea is interesting and potentially promising, but in its current form is not sufficiently evaluated to convince me that the performance boosts don’t simply come from the use of a larger network, and the lack of ImageNet evaluation calls into question its real-world application.
===============
Edit (1/23/17): I had indeed missed the fact that the Stanford Cars does do transfer learning from ImageNet -- thanks for the correction.  However, the experiment in this case is only showing late fusion ensembling, which is a conventional approach compared with the "stitched network" idea which is the real novelty of the paper.  Furthermore the results in this case are particularly weak, showing only that an ensemble of ResNet+VGG outperforms VGG alone, which is completely expected given that ResNet alone is a stronger base network than VGG ("ResNet+VGG > ResNet" would be a stronger result, but still not surprising). Demonstrating the stitched network idea on ImageNet, comparing with the corresponding VGG-only or ResNet-only finetuning, could be enough to push this paper over the bar for me, but the current version of the experiments here don't sufficiently validate the stitched network idea, in my opinion.


TAMING THE WAVES: SINE AS ACTIVATION FUNCTION IN DEEP NEURAL NETWORKS
Most deep neural networks use non-periodic and monotonic—or at least quasiconvex— activation functions. While sinusoidal activation functions have been successfully used for specific applications, they remain largely ignored and regarded as difficult to train. In this paper we formally characterize why these networks can indeed often be difficult to train even in very simple scenarios, and describe how the presence of infinitely many and shallow local minima emerges from the architecture. We also provide an explanation to the good performance achieved on a typical classification task, by showing that for several network architectures the presence of the periodic cycles is largely ignored when the learning is successful. Finally, we show that there are non-trivial tasks—such as learning algorithms—where networks using sinusoidal activations can learn faster than more established monotonic functions.
1 INTRODUCTION
Most activation functions typically used nowadays in deep neural networks—such as sigmoid, tanh, ReLU, Leaky ReLU, ELU, parametric ReLU, maxout—are non-periodic. Moreover, these functions are all quasiconvex, and more specifically either monotonic (sigmoid, tanh, ReLU, Leaky ReLU, ELU) or piece-wise monotonic with two monotonic segments (parametric ReLU, maxout).
Monotonicity makes sense from an intuitive point of view. At any layer of a network, neurons learn to respond to certain patterns, i.e. those that correlate with their weights; in case of monotonic functions, to a stronger positive correlation corresponds a stronger (or equal) activation, and viceversa, to a weaker positive correlation corresponds a weaker (or equal) activation. Neurons using piece-wise monotonic functions with two monotonic segments can be viewed as two separate neurons, each equipped with one of the two monotonic segments, and therefore independently looking for either the positive or the negative correlation between the weights and the input.
Excluding the trivial case of constant functions, periodic functions are non-quasiconvex, and therefore non-monotonic. This means that for a periodic activation function, as the correlation with the input increases the activation will oscillate between stronger and weaker activations. This apparently undesirable behavior might suggest that periodic functions might be just as undesirable as activation functions in a typical learning task.
But is this really the case? As shown in Section 2, there are several examples from the literature where sinusoidal functions were successfully used in neural networks. Moreover, as noted already in Gaynier & Downs (1995), networks using simple monotonic activation functions—such as sigmoids, tanh, ReLU—tend to have smaller VC dimension than those using non-monotonic functions. More specifically, even a network with a single hidden neuron using sinusoidal activation has infinite VC dimension1.
Neural networks using sinusoidal activation functions have been regarded as difficult to train (Lapedes & Farber (1987)) and have been largely ignored in the last years. There are a few questions
1I.e., it can correctly classify any set of points.
that naturally arise and make an analysis of deep neural networks using periodic activation functions interesting:
• What makes them in theory difficult to train? • Why do they still often manage to learn in practice? • How does the learned representation differ from the one of similar quasi-convex functions? • Are there tasks where periodic activation functions are more apt than quasiconvex ones?
In this paper we shed some light on these questions. In Section 2 we review relevant works on the topic of periodic activation functions. Starting from a simple example, in Section 3 we show what makes learning with sinusoidal activations a challenging task. In Section 4 we run a series of corroborative experiments, and show that there are tasks where sinusoidal activation functions outperforms more established quasi-convex functions. We finally present our conclusions in Section 5.
2 RELATED WORK
Periodic activation functions, and more specifically sinusoids, have received a tiny fraction of the attention that the research community reserved to the more popular monotonic functions. One of the first notions of a neural network with one hidden layer using sine as activation comes from (Lapedes & Farber, 1987, pp. 25-26). The authors define it as a generalized Fourier decomposition, and while recognizing the potential in their approximation capacity, they report that in their experiments these networks often exhibited numerical problems or converged to local minima.
In Sopena et al. (1999) the authors show on several small datasets that a multi layer perceptron with one hidden layer using sinusoids improves accuracy and shortens training times compared to its sigmoidal counterpart. For similar networks, improvements are shown in Wong et al. (2002) for a small handwritten digit recognition task and in McCaughan (1997) for the validity of logical arguments.
Some works have concentrated on mixing periodic and non periodic activations. In Fletcher & Hinde (1994) the authors propose to learn a coefficient that weighs each activation between sine and sigmoid. More recently, in Gashler & Ashmore (2016) the authors used sinusoids, linear and ReLU activations in the first layer of a deep network for time-series prediction.
Some theoretical results were presented in Rosen-Zvi et al. (1998), where the authors analyze the learning process for networks with zero or one hidden layers, and sinusoidal activations in all layers. In Nakagawa (1995) the author shows that a chaotic neuron model using a periodic activation function has larger memory capacity than one with a monotonous function.
Concerning recurrent neural networks (RNNs), in Sopena & Alquezar (1994) and Alquézar Mancho et al. (1997) the activation function for the last fully connected layer of a simple RNN was sine instead of sigmoid, which led to higher accuracy on a next-character prediction task. Choueiki et al. (1997) and Koplon & Sontag (1997) used sinusoidal activations in a RNN for short-term load forecasting and fitting sequential input/output data respectively. Liu et al. (2016) studied the stability of RNNs using non-monotonic activation functions, trying also sinusoids along others. No work so far—to the best of the authors’ knowledge—has investigated the use of periodic activation functions in convolutional neural networks (CNNs).
A separate line of research has focused on networks that closely mimic Fourier series approximations, so called Fourier series neural networks (Rafajłowicz & Pawlak (1997); Halawa (2008)). Here the hidden layer is composed of two parts: each input node is connected to an individual set of hidden nodes using sines and cosines as activations. The input-to-hidden connections have independent and fixed weights (with integer frequencies 1...K) for each input dimension. Then, the product is computed for each possible combinations of sines and cosines across dimensions. After that, only the hidden-to-output connections—which correspond to the Fourier coefficients—are learned. Despite the good theoretical properties, the number of hidden units grows exponentially with the dimensionality of the input (Halawa (2008)), rendering these networks impractical in most situations.
3 ANALYSIS OF SINUSOIDAL ACTIVATION FUNCTIONS
Let us start with a definition of the framework studied. In this section we analyze a deep neural network (DNN) with one hidden layer and linear activation at the output. The network receives as input a vector x—that has an associated target y—and computes an hidden activation h and a prediction ŷ as
h = F(Wx+ bW) (1) ŷ = Ah+ bA (2)
where W and A are weight matrices, bW and bA are bias vectors, and F is an activation function. As noted already in previous works, there is a clear interpretation of the variables in the network when F = sin, in terms of a Fourier representation. The weights W and the biases bW are respectively the frequencies and phases of the sinusoids, while A are the amplitudes associated, and bA the DC term. As shown in Cybenko (1989); Jones (1992) such a network can approximate all continuous functions on C(In), i.e. on the n-dimensional hypercube.
3.1 LEARNING WITH SINES AND LOCAL MINIMA
We can encounter issues with local minima even when learning the network parameters to solve a very simple optimization problem. Let us assume we are trying to learn the target function g(x) = sin(νx) for −m < x < m and some frequency ν ∈ R. x is the input to the network, and for this analysis we treat the case of continuous and uniformly distributed data, but we argue later in the section that similar conclusions can be expected with a limited amount of randomly distributed samples. By training a network with a single hidden neuron, fixed hidden-to-output connection A = a = [1] and no biases, i.e. no phase nor DC term to learn, our problem is reduced to learning the frequency ν as the weight W = [w].
Formally, we are minimizing the squared loss (sin(νx)− sin(wx))2. For a fixed choice of ν and m, the loss landscape L(ν, w,m) wrt to w has the form
L(ν, w,m) = ∫ m −m (sin(νx)− sin(wx))2dx
= −2 sin (m (w − ν)) w − ν + 2 sin (m (w + ν)) w + ν − sin (2mw) 2w + c(ν,m)
(3)
where c(ν,m) is a constant term. As illustrated in Fig. 1, for a fixed choice of ν and m, the three main terms in L(ν, w,m) are three cardinal sines (or sincs): the first is negative and centered at w = ν, which is the only global minimum and where the loss is 0; the second term is positive and centered at w = −ν, and is the only global maximum; the third sinc is negative and centered in w = 0. The latter creates a local minimum for small values of w and large values of m and ν, where the function expressed by the network is a constant sin(0) = 0.
We can already spot two culprits responsible for the difficulty of this learning problem:
(i) the deep local minimum centered in w = 0, produced by the sinc centeed in 0, which “traps” small weights around zero
(ii) the infinitely many ripples created by all three sincs, each of which is a shallow local minimum.
Also note that away from the main lobes the overall shape of the loss is almost flat, and therefore if the optimization starts far from the global optimum the gradients will tend to be small.
Let us now make the result more general, by including again the amplitudes and bias terms, and trying to learn a more complex function. After adding a bias/phase term to the neuron and to the target function g(x) (b and φ respectively) and a hidden-to-output weight/amplitude term, (a and γ respectively), we are trying to minimize (γ sin(νx+φ)− a sin(wx+ b))2. From the solution of the integral, the equation describing the second summand in Eq. 3 gains a term aγ cos(b+φ), while the third summand gains a term a2 cos(2b). Therefore all the sincs are still present (as shown in Fig. 2), and so are the aforementioned side effects.
Moreover, the local minimum centered in zero comes from the integral∫ sin2(wx)dx = x
2 − sin(2wx) 4w + c, (4)
which appears after expanding the square of the sum and applying linearity to the integral in L(ν, w,m). Note that this term is not related to the function to be learned g(x), nor to the fact that there is a single hidden neuron, and therefore will always appear in any network with a single layer of sinusoids trained using mean squared error.
Finally, since any function in the class that we are considering can be approximated to desired precision using a finite sum of sinusoids g(x) ≈ ∑M i=0 γi sin(νix + φi), we can
turn our analysis to any target function g(x). The resulting function to be minimized[∑M i=0 γi sin(νix+ φi)− ∑N i=0 ai sin(wix+ bi) ]2 is again the square of the sum of multiple sinu-
soids. After squaring and applying linearity, every term will either be sin2(·) or sin(·) sin(·) (with some amplitude terms). The former produces a sinc centered in zero, while the latter an odd pair of sincs.
Despite all this, the problems we just described are typically not an issue for many tasks. Going back to the example with a single sinusoid to learn, we can notice that the central local minimum disappears when the frequency ν is small enough that the main lobe of the rightmost sinc incorporates the main lobe of the central sinc (see Fig. 3). This happens when the data has a frequency representation with a large amount of low frequencies, which we assume to be often the case for many realistic datasets. The size of the support m also has an effect on the width and depth of the sincs. In a practical case at training time the integral is replaced by a sum—since only a limited amount of training samples is available—, the sampling is typically not uniform, and there might be noise in the data. Moreover, in the analysis we assumed that the loss surface (and therefore the
gradient) is calculated on the full training set, while in practice only mini batches of training samples are typically used. All these factors can contribute to smooth the loss surface L, potentially making the task easier to solve (see Fig. 3).
On these premises, we can expect that learning will be difficult when g(x) has large high frequency components (disjoint sincs). If network weights are initialized with small enough values, the weights might remain trapped inside the local minima of the central sincs. For large initialization the network might still be unable to find the global minimum due to the absence of overall curvature and the presence of shallow local minima. The optimization will be hard also if g(x) has low frequency components and the weights are initialized with large values. We speculate that a large initialization of the weights, typical in the past, was the main reason why these networks were regarded as difficult to train even with a single hidden layers.
Extending the analysis to deeper networks using sinusoids is not as simple. Already for two hidden layers the resulting function is of the form sin(sin(·)), whose integral is not known analytically in closed form.
3.2 INITIALIZATION AND USE OF PERIODICITY
As a consequence of the results presented in Section 3.1, the correct initialization range of the weights using sine might be very different from the one used for other activation functions. If the weights are very small, the sinusoid acts in its central linear region (Fig. 4).
While for inherently periodic tasks it is reasonable to assume that the network might indeed perform better, several tasks analyzed in Section 2 are not clearly periodic. None of the aformentioned works has analyzed the possibility that the network used mostly the monotonic segment of the sinusoid around zero, which is very similar to the tanh (Fig. 4). Especially in the typical training scenario— where the input data x is normalized to have zero mean and unit variance, and the network initialization is done using small weights W and zero biases—most pre-activations z = Wx+ b are likely to be such that |z|< π/2. In Section 4 we run a series of experiments to investigate if and how much a network trained using sine as activation actually relies on the periodic part.
4 EXPERIMENTS
In this section we train several networks using sin as activation function on the MNIST and Reuters dataset. We then investigate how much of the periodicity is actually used by replacing the activation function in the trained network with the truncated sin, (abbreviated as tr. sin), defined as
tr. sin =  0, if − π/2 < x sin(x), if − π/2 ≤ x ≤ π/2 1, if x > π/2
(5)
We also train the same networks using the monotonic function tanh for comparison. We then run experiments on a couple of algorithmic task where the nature of the problem makes the periodicity of the sinusoid potentially beneficial.
4.1 MNIST
We experiment with the MNIST dataset, which consists of 8-bits gray-scale images, each sized 28 x 28 pixel, of hand-written digits from 0 to 9. The dataset has 60,000 samples for training and 10,000 samples for testing. It is simple to obtain relatively high accuracy on this dataset, given that even a linear classifier can achieve around 90% accuracy. Since the data is almost linearly separable, it is reasonable to expect that using sine as activation function will not make much use of the periodic part of the function. We test a DNN, a CNN and an RNN on this problem, using sine as activation function, and compare the results to the same network trained using tanh.
On all experiment on MNIST we scale the images linearly between 0 and 1. All networks have an output layer with 10 nodes, use softmax and are trained with cross-entropy as loss. The batch size is 128 and the optimizer used is Adam (Kingma & Ba (2015)) with the hyper-parameters proposed in the original paper.
DNN We use a DNN with 1 to 2 hidden layers, each with 256 hidden neurons. We initialize the weights in all layers using a normal distribution with standard deviation σ in the set 1, 0.1, 0.01. The input images are flattened to vectors of size 28× 28 = 784, which makes the task referred to as permutation invariant MNIST. The networks are trained for 20 epochs.
RNN The input images are presented as a sequence of 28 rows, each containing 28 values, starting from the top to the bottom. We use a RNN with 1 hidden layer with 128 hidden neurons. We experiment separately with vanilla RNNs and LSTMs. When the latter are used with sine, the function is used in place of the inner tanh activation. We initialize the weights in all recurrent layers using a normal distribution with standard deviation of 0.1.
The DNN results are reported in Table 1. As expected, replacing the activation from tanh to truncated sin does not affect much the results. For this reason we will not report this value on the following tables. For the same reason, switching sin to either tanh or to truncated sin has almost the same effect, so we will only report the latter from here onwards. When using small values of σ = {0.1, 0.01} for the initialization, all networks equipped with sines obtained very similar results to the networks trained with tanh. Even though for these networks between 27% to 47% of the activations fall outside of the range [−π/2, π/2], replacing sin with tr. sin does not reduce the accuracy by more than 2.5%. We can therefore conclude that the network is ignoring for the most part the periodicity of the function. On the contrary, the tanh is more significantly relying on the saturated part, and as σ increases so does the drop in the accuracy when switching the activation function to sine (reaching random guessing accuracy for σ = 1).
As expected from the results presented in Section 3, the networks with sine had difficulty to converge for large initialization σ = 12. Also notice that adding weight decay allowed the same network with 1 hidden layer to converge, reaching a solution that scarcely uses the periodic part of the function. Finally, results show that even for deeper networks with eight hidden layers, sinusoid can learn the task quite effortlessly, and still does so scarcely relying on the segment of the function outside [−π/2, π/2]. A somewhat similar but less evident behavior emerged from the RNNs, as shown in Table 2. Especially for the LSTMs, the network using tanh relied on larger pre-activations much more than the network using sin.
Similar experiments with the Reuters dataset (ref) showed the same behavior, as seen in table 3. Each sequence of words corresponding to a data sample from the dataset is first converted to a vector of size 1000, where the ith entry represents the amount of times that the ith most frequent word in the dataset appears in the sentence. The DNN has 128 hidden neurons, networks are trained for 20 epochs and test results are computed on a held-out 20% of the data.
4.2 LEARNING ALGORITHMIC TASKS
We test the networks using sine as activation on a couple of algorithmic tasks, such as sum or difference of D-digits numbers in base 10. In both tasks the data is presented as a sequence of one-hot encoded vectors, where the size of the vector at each timestep is 12: the first 10 entries correspond to the numbers from 0 to 9, the last two entries correspond to the operator symbol—‘+’
2The network with sin, 1-L and σ = 1 reaches an accuracy of 40% on the training data after 20 epochs and 83% after 1000 epochs. With two hidden layers it has random guessing accuracy on the training data after 20 epochs, and after 100 epochs 100% accuracy on training data and random guessing accuracy on test data.
or ‘−’ in case of sum or difference respectively—and the ˙blank˙ symbol used for padding. The length of an input sequence is D + 1 +D, while the output sequence has length D + 1. If a string is shorter than the total length, the remaining entries are padded with the ˙blank˙ symbol.
For the task sum (difference) the network is expected to produce the result of the sum (difference) of two positive integers fed as input. We run experiments with the number of digits D = 8. The order of the digits of each number is inverted, which was shown to improve the performance in several tasks using encoder-decoder (ENC-DEC) architectures.
We use an encoder-decoder architecture based on vanilla RNN or LSTM. The networks have 128 hidden units in every layer, one recurrent layer for encoding and one recurrent layer for decoding. The decoder has also a fully connected output layer with softmax at each step. The encoder ”reads” the input sequence one symbol at a time and updates its hidden state. At the end of the input sequence, the hidden state from the encoder is fed at each step for D + 1 times as input to the decoder. The decoder produces the output, one digit at a time.
The networks are trained for 5000 iterations3 using Adam as optimizer, cross-entropy as loss and a batch size of 128. The feed-forward and recurrent weights are initialized using a normal distribution with the widely used schemes proposed in Glorot & Bengio (2010) and Saxe et al. (2013) respectively, we clip gradients at 1 and decay the learning rate by 10−5 at every iteration. Samples are generated at each iteration and we do not use a separate validation or test set, since the number of possible samples is so large that overfitting is not an issue. The accuracy for a given prediction is 1 only if every digit in the sequence is correctly predicted. The results reported in Fig. 5 are computed at every iteration on the newly generated samples before they are used for training.
The networks using sine learn the tasks faster and with higher accuracy than those using tanh. While in vanilla RNNs the difference is quite evident, the improvement is less striking for the LSTM. In all the models switching the activation from sine to truncated sine, or from tanh to sine brings the accuracy almost to 0, indicating that the network is effectively using the periodic part of the function.
5 CONCLUSIONS
Neural networks with a single hidden layer using sinusoidal activation functions have been largely ignored and regarded as difficult to train. In this paper we analyzed these networks, characterizing the loss surface, and showing in what conditions they are especially difficult to train. By looking into the hidden activations of networks succesfully trained on a simple classification task, we showed that when learning is successful the networks often scarcely rely on the periodicity of the sinusoids.
Finally, we showed on a pair of simple algorithmic tasks where the periodicity is intuitively beneficial, that neural networks using sinusoidal activation functions can potentially learn faster and better than those using established monotonic functions on certain tasks. This encourages future
3Here we refer to one iteration as the processing of 128 minibatches.
work to investigate the use of periodic functions, the effect at different layers, and the potential of incorporating these functions in other models using quasi-convex functions.
ACKNOWLEDGMENTS
The authors wish to acknowledge CSC IT Center for Science, Finland, for computational resources.
A APPENDIX
We report here similar experiments to Section 4.2 but using uniform sampling of the addends, instead of uniform sampling of the number of digits of each addend.
As shown on the left plots in Fig. 6, for D = 8 the networks using sin learn faster and reach higher accuracy than the network using tanh. For the case of D = 16 and 3 recurrent layers in the encoder, sine reaches almost 80% accuracy, while tanh never takes off in the 5000 epochs of training. A similar behavior emerges on the task dif, as shown in Fig. 7, although with overall lower accuracy and with none of the networks successfully learning the task with D = 16. Surprisingly, the LSTMs almost completely fail to learn the tasks under these training setting.
More hidden neurons When the number of hidden units is doubled from 128 to 256, the standard LSTM using tanh as activation learns faster and reaches higher accuracy than the network trained with sine, while the vanilla RNN using sin still outperforms both the vanilla RNN and the LSTM using tanh. These results are reported in Fig. 8 for the case of D = 8 only, since for D = 16 all networks are stuck at zero accuracy. Further investigation would be required to explain how doubling the amount of neurons in the tanh LSTM changed the learned representation, providing such a boost in performance.
Curriculum learning experiments For the D = 16 case, we also experiment with a curriculum learning approach: while keeping the length of the input and output sequences fixed to 16+1+16 and 17 respectively, we start training by limiting the maximum number of digits to D = 8 and increase D by 2 every 1000 iterations, so that by the 4000th iteration D = 16. As shown in Fig. 9, by using this approach the network using sine as activation function reaches an accuracy close to 1 by the end of the training. As shown by the steep drops in performance when D is increased, the network has only learned to correctly perform the operation within the number of digits it was trained upon, but it can adapt very quickly to the longer addends. The network using tanh takes more time to learn the case with D = 8 and after that does not adapt to larger number of digits.

Summary:
In this paper, the authors explore the advantages/disadvantages of using a sin activation function.
They first demonstrate that even with simple tasks, using sin activations can result in complex to optimize loss functions.
They then compare networks trained with different activations on the MNIST dataset, and discover that the periodicity of the sin activation is not necessary for learning the task well.
They then try different algorithmic tasks, where the periodicity of the functions is helpful.
Pros:
The closed form derivations of the loss surface were interesting to see, and the clarity of tone on the advantages *and* disadvantages was educational.
Cons:
Seems like more of a preliminary investigation of the potential benefits of sin, and more evidence (to support or in contrary) is needed to conclude anything significant -- the results on MNIST seem to indicate truncated sin is just as good, and while it is interesting that tanh maybe uses more of the saturated part, the two seem relatively interchangeable. The toy algorithmic tasks are hard to conclude something concrete from.