ERTIES OF CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
1 INTRODUCTION
The overwhelming success of Convolutional Neural Networks (CNNs) is generally attributed to their ability to learn task-specific representations from large quantities of data. This has led to many state of the art results in areas such as image classification (He et al., 2015), semantic segmentation (Liu et al., 2015), and game-playing Go agents (Silver et al., 2016). However, our current understanding of CNN representations is limited, though there is a growing body of literature on visualization techniques for interpreting internal representations (Mahendran & Vedaldi, 2016; Dosovitskiy & Brox, 2015; Nguyen et al., 2016).
In this work, we aim to understand the effect that the training data has on three key properties of representations: invariance, equivariance, and equivalence (Lenc & Vedaldi, 2015). Invariance, with respect to a particular transformation (e.g. rotation by θ = 5), is achieved when the feature vectors for input images at a particular layer do not change when the inputs are transformed. This can be seen as a measure of the robustness of the representation. Equivariance is a generalization of invariance and allows the representation to change in predictable ways in response to input transformations. Equivariance is a rough measure of the structure of the representation space because we can reason about input space transformations in the more abstract representation space. Likewise, measuring equivalence among network representations trained on either the same or different data distributions gives insight into how the training data shapes the representation.
Data augmentation has been considered essential for top CNN performance since the seminal work of Krizhevsky et al. (2012). When input images are stochastically perturbed, less overfitting is observed because all inputs are unique. Since then, others have experimented with various data augmentation strategies with great success (Howard, 2013; Wu et al., 2015; He et al., 2014). While others have noted that data augmentation leads to greater representation invariance (Lenc & Vedaldi, 2015; Wu et al., 2015; Peng et al., 2014), we measure the amount of invariance achieved in response to 9 types of transforms at various magnitudes. We additionally measure the equivariance and pairwise representation distance of the resulting CNNs.
In this work we quantify the invariance and equivariance properties of 70 CNNs on two datasets trained using different input transformations. We find the representation at the penultimate layer
(fc7) is structured wrt almost all input transformations when no augmentation is used. Applying data augmentation effectively collapses the structure of the representation space, leading to input transformation invariance for all 9 transforms, including magnitudes of transforms that were not observed during training.
We measure the pairwise distances of the CNN representations by measuring the error of learned mappings between CNN representations. There is a strong bias towards similarity among CNNs trained with the same type of transformation, and we can group transformations based on their mutual average distances. Similar types of transformations (e.g. Color Jitter, Gaussian Noise) yield more similar representations compared to other dissimilar transform pairs.
We also propose a way to increase the equivariance of a CNN by finetuning on a novel loss function that simultaneously minimizes classification error for both transformed and untransformed inputs. This leads to an increase in equivariance while improving or slightly decreasing performance on the untransformed images (dataset dependent), though this trade-off can be controlled by weighting the terms of the loss function.
2 RELATED WORKS
Lenc & Vedaldi (2015) studied image representations and gave definitions for representation invariance, equivariance, and equivalence. They measured these properties wrt a limited number of transforms for the convolutional layers of the popular AlexNet architecture. We provide extensions to the definitions of these properties to measure the relative degree to which representations possess these properties. We also consider a wider variety of transformations and use the fc7 layer of AlexNet because it is the most invariant layer.
Convergent learning is the idea that identical networks (differing in random initialization) converge to the same representation (i.e.. representation equivalence). In Li et al. (2015), the authors show that there are many corresponding convolutional filters among CNNs for one-to-one, one-to-many, and many-to-many relationships. In complementary fashion, we measure representation distances between last fully connected layers in CNNs trained over different transformations of their inputs.
In Krizhevsky et al. (2012), applying random crops, horizontal flips and color jittering during training reduced the top-1 error by over 1%. Howard (2013) applied additional brightness, contrast, crop, and scale transformations at both training and test time to reduce validation top-1 error from 40.7% to 37.0%. Wu et al. (2015) adds rotation and photo filter effects such as lens distortion, vignetting, and color casting for further improvement. Similarly, we explore a series of 10 types of transformations applied at training time; however, we focus on how they affect the learned representation instead of optimizing classifier performance.
In Peng et al. (2014), synthetic images rendered from 3D CAD models are used to explore invariance in object detection networks for factors such as pose, object texture, and foreground/background color. Training a CNN on images that vary in these aspect results in greater invariance than training with less data variety. 3D CAD models are also used in Aubry & Russell (2015) to show that upper layers of CNNs disentangle and (locally) linearize independent object factors such as object viewpoint, style, color, and scale.
ManiTest (Fawzi & Frossard, 2015) defines classifier invariance as the average magnitude of the minimal transform that causes predictions to change. In contrast, we examine internal CNN representations and measure both representation distance and classifier performance.
3 INVARIANCE, EQUIVARIANCE, AND EQUIVALENCE
In this section, we provide definitions and intuition on the three properties of interest, borrowing basic definitions and notation from Lenc & Vedaldi (2015). Though we focus on CNN image representations, the following applies to any kind of data representation.
3.1 EQUIVARIANCE AND INVARIANCE
If the domain of interest (e.g. natural images) is X ⊂ Rnxn, a function φ : Rnxn → Rm assigns a feature vector to every input image and thus defines an image representation. We will use X to
denote the input space and Z the output space (i.e. φ : X → Z). φ may be equivariant to some transformations, but not others, so we say that φ is equivariant wrt g : X → X iff
∃Mg : Z → Z, ∀x ∈ X : φ(gx) ≈Mgφ(x) (1) This means that transformation by g in X corresponds to a transformation by Mg (if it exists) in Z. Thus if Mg exists, it informs us of the structure of the abstract space Z in terms of the concrete space X . If Mg is the identity transform, then φ maps x and gx to the same point in Z, which leads to invariance wrt g (i.e. φ(gx) ≈ φ(x)). This is related to the idea of whether CNNs collapse (invariance) or linearize (equivariance) view manifolds of 3D objects (Bakry et al., 2015; Aubry & Russell, 2015).
While the previous definitions suggest equivariance and invariance are all-or-nothing properties, we use the following definition to precisely define the ≈ used in Eq 1. We measure the equivariance of φ wrt a transform g as
Equivarainceφ(g;L) = min Mg
1
N N∑ i=1 L(φ(xi),Mgφ(gxi)) (2)
whereL is a suitable loss function such as classification error or L2 distance (Section 3.3). Invariance is similarly measured by fixing Mg as the identity function.
In practice, it is difficult to maximize Mg over the space of all functions, so we restrict Mg to be a simple parametric function (e.g. linear) and learn the parameters from data. Thus our results can be considered a lower bound on the actual equivariance of the representation. Eq. 2 also uses the paradigm that Mg attempts to undo in Z the transformation g performed in X . This is also done in Lenc & Vedaldi (2015) and allows us to compute classification based metrics by applying the same classifier to both φ(x) and Mgφ(gx).
3.2 EQUIVALENCE AND REPRESENTATION DISTANCE
In Lenc & Vedaldi (2015), φ1 is equivalent to φ2 if φ1 has the same information in the sense that ∃E1 : Z1 → Z2, ∀x ∈ X : φ2(x) ≈ E1φ1(x) (3)
However, this relation is only symmetric (as the name equivalence implies) iff E1 is invertible. If E1 is invertible, then E−11 satisfies φ1(x) ≈ E −1 1 φ2(x). In the other direction, if the relation is symmetric, then ∃E2 such that φ1(x) ≈ E2φ2(x) and such an E2 is an inverse of E1. In light of this, we propose renaming Eq 3 to be the sub-representation property with the the terminology that φ2 is a sub-representation of φ1. We then define φ1 and φ2 to be equivalent iff an invertible E1 exists. One example of sub-representation are two reprensetations characterized by (1) the RGB pixels of an image and (2) the corresponding grayscale intensities. The grayscale is a sub-representation of the RGB because we can recover the grayscale from the RGB, but not vice versa.
As with equivariance, we are interested in measuring the extent that two representations are distinct. Thus we define the distance between two representations as
D(φ1, φ2) = min E1 [L(E1φ1(x), φ2)] + min E2 [L(φ1, E2φ2(x))] (4)
As with Mg , finding the optimal E1, E2 is difficult, so we learn linear models from data. Choices for the loss function, L, are discussed in Section 3.3.
3.3 DISTANCE METRICS
The two distance metric we employ are normalized euclidean distance in the representation space and classification error.
LL2(ri, r ′ i) = ||ri − r′i||2 ||ri||2 , Lerr(ri, r ′ i) = 1− δ(yi, ŷi), ŷi := argmax j C(r′i)j
where ri = φ(xi), r′i =Mg(gxi), C(ri) is the predicted distribution of class labels, yi is the ground truth label of ri, and δ is the Kronecker delta function. For LL2, we normalize each dimension of r to have similar magnitude, similar to the normalization of Li et al. (2015). The classifier, C, is composed of the remaining layers of the CNN after φ.
4 IMPROVING EQUIVARIANCE
In this section, we propose a loss function for training a network to increase its invariance and equivariance. CNNs for classification tasks are typically trained with a cross entropy loss, i.e.
Ld(X,Y ) = 1
N N∑ i=1 H(yi, C(φ(xi))) (5)
where X is the training images, Y is the training labels, and H computes cross entropy.
Ld can be augmented by a term that biases the representation φ to be equivariant to some set of transforms G = {g1, g2, ..., gJ}. We propose the loss function, Le, which is minimized when the transformed images are both correctly classified and spatially close to the untransformed image inZ.
Le(X,Y ) = 1
NJ N∑ i=1 J∑ j=1 [ H(yi, C(x ′ ij)) + λ1||x′ij − φ(xi)||22 + λ2||φ(gjxi)− φ(xi)||22 ] (6)
where x′ij = Mgjφ(gjxi). The first term of the sum is the classification loss of the transformed images, while the second and third terms respectively enforce equivariance and invariance. In our experiments, we set λ1 = 504096 , λ2 = 25 4096 , though the results do not appear highly sensitive to this particular setting. Each equivariance mapping Mgj constitutes a new portion of the network that attempts to undo the transformation gj in the Z. As such, the parameters of Mgj must be learned along with the parameters of the network.
Combining Eq. 5 and 6, we arrive at our combined loss:
L(X,Y ) = Ld(X,Y ) + λ3Le(X,Y ) (7)
We found that λ3 = 0.3 provides good compromise between performance on the transformed and untransformed data, though λ3 can be tuned for individual performance requirements.
5 EXPERIMENTAL SETUP
Here we detail the experiments we conduct including the transformations, networks, and datasets.
For the first set of experiments, we simply measure the invariance and equivariance of the AlexNet1 architecture trained under various data augmentation schemes. We measure the representation of layer fc7 (denoted φfc7) after applying the ReLU non-linearity and dropout2.
1We used the reference CaffeNet architecture, which differs slightly from Krizhevsky et al. (2012) 2We follow the test time convention of halving neuron activations instead of zeroing out half the activations.
5.1 TRANSFORMATIONS
We examine a set of 10 transformation types. See Table 1 for brief descriptions or Appendix A for full details. We train 4 CNNs for each type of transform (only 2 for blurring, 1 for baseline) for a total of 35 networks per dataset. Each of the 4 CNN within a transform type is trained with different magnitudes of transformations. Training with data augmentation is performed by stochastically transforming each network input with a transform taken from a specified range.
After training, we measure invariance and equivariance wrt fixed transforms for each type. For example, the 4 CNNs trained on different ranges of rotations (θ ∈ [−5, 5], θ ∈ [−10, 10], θ ∈ [−15, 15], θ ∈ [−20, 20]) all have their equivariance measured wrt rotations of θ = ±{2.5, 5, 10, 15, 20, 25, 30, 40}. As a baseline, we trained a model with no data augmentation and measured the invariance and equivariance of that network wrt all transforms.
5.2 EQUIVARIANCE MAPPING
To measure equivariance wrt gj , we learn Mgj (Eq. 2) and measure the classification accuracy over the transformed images. Due to data augmentation during training, the network may already be invariant wrt gj , so we model Mgj as a linear residual mapping to bias it towards an identity mapping (He et al., 2015). That is Mgj (r) = σ(r + R(r)), where R is to be estimated from data and σ(x) = max(x, 0) is the ReLU non-linearity. We experimented with R as a linear mapping and as a neural network with a single hidden layer. In nearly all cases, the linear mapping outperformed the neural network, so we present results only for the linear mappings. Therefore, we have
Mgj (r) = σ(r +Wjr + bj) = σ((Wj + I)r + bj) (8)
where (Wj , bj) are parameters to be estimated from data.
Each Mgj is trained using fully online SGD for 10 epochs with a static learning rate of 0.001, using an L2 weight decay penalty of 10−5. Momentum with β = 0.9 is used. For learning Mgj , only (Wj , bj) are updated. The rest of the network parameters are treated as fixed. The loss to be minimized is ||Mgj (φfc7(gjx))− φfc7(x)||22.
5.3 REPRESENTATION DISTANCE MAPPING
Here we are interested in learning E1 (or equivalently E2) from Eq. 4 for each ordered pair (φi, φj) in a set of of network representations {φi}. We consider linear mappings with E1(r) = σ(Wr + b) and estimate (W, b) from data. Hyper-parameters for learning are the same as in Section 5.2. The loss to be minimized for each ordered pair is ||φi(x)− E1φj(x)||22.
5.4 FINETUNING EQUIVARIANCE
For this experiment, we take the CNNs trained with the most extreme parameter ranges for each transform type and finetune the representation using Eq. 7 as the loss function. For each CNN, we selected 4-6 transformations within the training range to be the G in Eq. 6.
We finetune for 150,000 weight updates with a mini-batch size of 10 for RVL-CDIP and 100 for ILSVRC. The learning rate is 0.0005 and decays by a factor of 10 every 60,000 updates. Momentum of β = 0.9 was used with no weight-decay regularization.
5.5 DATASETS
We use two large datasets with distinct properties. The first is the popular ILSVRC 2012 dataset (1.2M train / 50K validation), composed of natural images. The second is the RVL Complex Document Information Processing (RVL-CDIP) dataset (Harley et al., 2015) (320K train / 40K val / 40K test). It is composed of scanned tobacco litigation documents in grayscale format and each document image is labeled with one of 16 categories (e.g. letter, memo, email, form, news article, scientific publication). Examples images can be found in Figure 4 in Appendix A In contrast to ILSVRC, document images in RVL-CDIP are intrinsically 2D objects, have fixed zoom and location, and have little background area. We aim to compare and contrast the invariance and equivariance of the various transforms for these two datasets.
In our experiments CNNs are trained using the training splits, using the validation split for model selection. For learning equivariance mappings and representation distance mappings, we use a static randomly chosen 50K training images for ILSVRC and the validation split for RVL-CDIP. Thus the reported metrics are over the validation split for ILSVRC and test split for RVL-CDIP. We subsample for ILSVRC for computational reasons and use the validation split for RVL-CDIP because the CNN representation may have overfit the training data.
6 RESULTS
6.1 MEASURING INVARIANCE AND EQUIVARIANCE
Figure 1 shows the invariance and equivariance measurements for each type of transform on ILSVRC. Additional results for both ILSVRC and RVL-CDIP can be found in Appendix B.1. For each transform type, we compare the CNN trained with the most extreme transforms to the baseline model trained with no data augmentation. The transformations we measured range from no transform to approximately double the transformation magnitude seen during training.
In many instances (e.g. Blur, Noise), the baseline model’s performance deteriorates rapidly even for mild transformations. The large difference between the baseline invariance (black line) and equivariance (red line) indicate that Z is structured wrt most transforms (Color is one exception).
For CNNs trained with data augmentation, the equivariance mapping does not improve performance for transforms that are observed during training. This suggests that Z may not be (linearly) struc-
tured wrt those particular transforms possibly because the structure has been collapsed due to the training object. Some transforms (e.g. Noise, Blur, Shear), at large magnitudes, do show a gap between the equivariance and invariance lines, suggesting that Z is structured wrt these transforms. Equivariance performance does drop off outside the training range, but not near as sharply as the baseline model (e.g Figs. 7a, 7c, 7f), so CNN representations generalize somewhat to unseen transforms.
Figure 2 shows how various magnitudes of training augmentation affect the invariance/equivariance properties of the network for select transforms. Other transforms behaved similarly. Larger training ranges yields greater robustness for a wider range of transforms.
6.2 REPRESENTATION DISTANCE
In this second experiment, we measured the pairwise representation distances of 37 CNNs (34 w/ transforms and 3 baseline networks). Using equation 4 (L = LL2), we computed a pairwise distance matrix for the 37 network representations. We attempted a T-SNE embedding (Maaten & Hinton, 2008) visualization, but the distances seem intrinsically difficult to embed in 2D (see Appendix B.3). Comments on the asymmetry of the distances measured can be found in Appendix B.3.
There is, however, a strong bias towards networks of the same training transformation to be nearer to each other compared to random chance. We performed a K-NN analysis by counting the percentage of same-transform pairs that are K nearest neighbors (Figure 3a). Approximately 80% of sametransform pairs are within 5 neighbors of each other for RVL-CDIP (10 neighbors for ILSVRC). This indicates that networks trained with the same transform end up with more similar representations, though the strength of the transformation and network initialization also play a role.
We also visualize the average distance between CNNs grouped by transformation type (Figs 3c and 3b). Patterns emerge in both datasets. In RVL-CDIP, Crop is the most unique transform because the CNNs learn features as a different static image scale. On the other hand, Crop is not as unique for ILSVRC because objects appear at multiple sizes, so the features learned are not targeted at a single size. Blur, especially for ILSVRC, seems to give unique representations because it changes the local textures from which the representation is derived in a bottom-up fashion. Refining the representation in a top-down manner may be a further avenue for improving CNN representation robustness.
Rotation, Shear, and Perspective transforms are mutually similar for both datasets. This is likely because all three move local pixels in a rigid fashion. Another mutually similar group is Baseline, Color Jitter, and Gaussian Noise, which all operate on pixels independently. Elastic Deformations are somewhat similar to Shear, but not to other transforms.
6.3 IMPROVED EQUIVARIANCE
For RVL-CDIP, finetuning using Eq. 7 improves both equivariance and performance on untransformed images by ∼0.5% (Table 2). This may be because the original network was trained with a wide range of transform parameters, so the representation caters to no particular set of transform parameters. By introducing Mgj during finetuning, the representation can individualize to the untransformed image, while still avoiding the overfit that occurs without data augmentation. Results for ILSVRC were mixed, with finetuning improving equivariance for 3 of 9 transformations (see Table 4 in Appendix B).
7 CONCLUSION
This work gives the results from a large scale empirical study into the invariance and equivariance properties of CNNs trained under different input transformations. We quantify these properties for 70 CNNs across 10 types of transforms for 2 large datasets. CNNs are able to learn invariance to all transforms tried, and this invariance/equivariance extends somewhat to transforms outside the training range. We show evidence that the CNN linearizes its representation wrt Rotation, Perspective, and Shear transforms. In general, the baseline CNN showed very little invariance to even moderate transformations.
We also measured CNN representation distance between all pairs of 37 networks. There is a bias towards CNNs trained with the same type of transformation to have more similar representations. The analysis also revealed that similar types of transforms (e.g. Rotation, Shear) lead to more similar representations. We also proposed a joint loss function that moderately increases accuracy on both untransformed and transformed images, leading to an increase in equivariance for most transforms.
APPENDIX
A TRANSFORM DETAILS
For space reasons, exact details and parameterizations of the transforms used to train the CNNs were omitted from the main text, but are included here. Table 1 gives a brief explanation of the 9 transforms, but each transform will be explained in detail hereafter. Pixels that move into the image dimensions from outside the image (e.g. pixels rotated in) are set to be intensity 0 (see Figure 6b).
A.1 COLOR JITTER
Color Jitter adds a random value to each color channel (RGB or grayscale) of the input image. The random value drawn is separate for each channel, but the same value is applied to each spatial location, essentially making each color either brighter or darker. If the addition of the random value causes a pixel to go outside the normal [0, 255] range, it is truncated back to the range.
During CNN training, the random values are drawn from a Gaussian Distribution parameterized by a mean µ and standard deviation σ. Four CNNs were trained, with µ = 0 and σ ∈ {5, 10, 15, 20}. When measuring invariance/equivariance, we apply a deterministic color jitter transformation that is the same for all input images. We simply adjust the brightness of the image (all color channels tied) by a fixed amount in ±{2, 5, 10, 15, 20, 25, 30, 45}.
A.2 CROP
During training with Crop transformations, we first resize the input images to a larger size (such as 240x240, 256x256, 288x288, or 320x320), and then take a 227x227 crop from the larger imgae. The larger the original image, the more detail the CNN gets to see, but also a smaller percentage of the original image is captured in the input window. CNNs trained with other transformations have their input images resized to 227x227.
For measuring invariance/equivariance, the baseline transform for the Crop CNNs is the center crop, and the transforms are crops at other locations. For example, for invariance to upper left corner crops, we measured the difference in CNN activations between the network applied to the center crop and the network applied to the upper left crop. We measured 25 spatial locations arranged in a 5x5 grid.
A.3 ELASTIC DEFORMATIONS
Elastic Deformations are a way of locally distorting images by applying a smoothed random displacement field (Simard et al., 2003). Image transformations can be characterized by a backward mapping of pixel locations in the output image to locations in the input image. For elastic deformations, first an random displacement field is sampled that would map output pixels to random input locations in their local neighborhood. The size of the local neighborhood is controlled by an α parameter. This displacement field is then smoothed using Gaussian filtering with a specified σ. This causes local regions of the displacement field to point in the same direction.
For training CNNs, we used a fixed σ and sampled α for each input image. The four CNNs we trained with Elastic Deformations can be described as (α ∈ [0, 5], σ = 2), (α ∈ [0, 10], σ = 2), (α ∈ [0, 5], σ = 3), (α ∈ [0, 10], σ = 3). When measuring equivariance/invariance (in contrast to training) we apply the same exact transform to every image. For elastic deformations, this means we applied the same displacement field to each image. We did this for a number of displacement fields of various parameter settings. As shown in Figure 7c These settings included every combination of α ∈ {5, 10, 15} and σ ∈ {2, 2.5, 3, 3.5}.
A.4 GAUSSIAN BLUR
We used the standard Gaussian Blur transform which replaces each pixel by a weighted average of the neighboring pixels, where weight magnitudes are Gaussian wrt spatial distance. The shape of the Gaussian weighting function is controlled by σ, which is measured in pixels. We trained two CNNs with Gaussian Blur, where for each input image, we sampled a sigma from a uniform distribution (i.e. σ ∈ [0, 1.5] and σ ∈ [0, 3]). For measuring equivariance/invariance, we used σ ∈ [0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5].
A.5 GAUSSIAN NOISE
Gaussian Noise is similar to Color Jitter except that we sample different random values for each spatial location. The strength of the noise is controlled by σ, which is measured in pixel intensity ([0, 255] scale). When training the four CNNs, σ is sampled from a uniform distribution for each input image. The four ranges used were [0, 5], [0, 10], [0, 15], [0, 20].
We measured the invariance/equivariance of Gaussian Noise transforms defined by σ ∈ [2.5, 5, 7.5, 10, 12.5, 15, 20, 25, 30, 40].
A.6 MIRROR
Mirroring refers to reflecting the input image over the horizontal or vertical image axes. We trained 3 CNNs with mirroring. The first performed horizontal mirroring with probability 0.5. The second performed vertical mirroring with probability 0.5. The third performed both types of mirroring with independent probability 0.5 for a total of 4 combinations of flips.
We measures the equivariance/invariance of all 4 combinations of flips.
A.7 PERSPECTIVE
Perspective transforms are a class of transforms with the constraint that straight lines in the input image remain straight in the output image. One way to parameterize (8 parameters) a perspective transform is to specify the output coordinates of the input unit square, so that the unit square is mapped to an arbitrary quadrilateral.
When training a CNN, for each image we first sampled σ from a uniform distribution. Then we sampled the displacement of the coordinates of the unit square from a Gaussian Distribution with mean µ = 0 and standard deviation σ. Thus the output coordinates for the upper left corner of the unit square are x = N(0, σ), y = N(0, σ). For the lower right corner, they would be x = 1 + N(0, σ), y = 1 + N(0, σ). Then the image is warped according to the perspective transform defined by the output coordinates of the unit square.
The ranges used for training are sigma = [0, 0.001], [0, 0.002], [0, 0.003], [0, .004]. For measuring invariance/equivariance, we sampled 10 perspective transforms from a range of equally spaced σ values in increasing magnitude along the x-axis of Figure 7f. Examples of these transforms are given in Figure 6.
A.8 ROTATION
A Rotation transform is specified by an angle θ and is always performed about the center of the image. During training, we sample θ from a uniform distribution for each image. We trained four CNNs with rotations with θ drawn from the following ranges: [−5, 5], [−10, 10], [−15, 15], [−20, 20]. For measuring equivariance/invariance, we used θ ∈ ±{2.5, 5, 10, 15, 20, 25, 30, 45}.
A.9 SHEAR
A Shear transform warps a square image into a parallelogram. We parameterize it by specifying a shear angle θ and an orientation that is either horizontal or vertical. While the original square image has corners that are 90 deg, the parallelogram that results has corners with angles of 90 + θ and 90− θ.
During training, we sample θ from a uniform distribution for each image. We trained four CNNs with shear with θ drawn from the following ranges: [−5, 5], [−10, 10], [−15, 15], [−20, 20]. Each shear has equal probability of horizontal or vertical orientation. For equivariance/invariance, we measured only horizontal shears with θ ∈ ±{5, 10, 15, 20, 25, 30}.
B ADDITIONAL RESULTS
B.1 INVARIANCE AND EQUIVARIANCE
In this section, we include additional results from the experiments in Section 6.1 that did not fit in the main text. Figure 1 presents invariance and equivariance measurements for the ILSVRC dataset (similar to Figure 7).
We observe several interesting differences to the results on RVL-CDIP. The baseline model for ILSVRC is significantly more invariant to vertical mirrors than horizontal mirrors, though the RVLCDIP baseline performs equally bad on both types of mirrors. Brightness increases affect ILSVRC trained CNNs more, though this is likely because natural images are more susceptible to image saturation because they occupy the full spectrum of pixel intensities, while document images tend to be bi-modal. The RVL-CDIP CNN trained on Gaussian Noise was able to learn invariance, while the corresponding ILSVRC CNN did not. However, both the baseline and noise CNN for ILSVRC can correct for the noise with an equivariant mapping.
Figures 8 and 9 show the invariance/equivariance for all 70 CNNs trained with data augmentation. In general, we see an overwhelming trend that training with greater variety of inputs transforms results in greater invariance and equivariance. The exception seems to be that the magnitude of color jittering does not highly affect the CNN robustness. Early CNN layers can easily become invariant to overall brightness changes by extracting information based on pixel differences, and prior work has shown that later layers encode information mostly by which neurons are non-zero, rather than the magnitude of the neuron activations (Agrawal et al., 2014).
For Crop transforms (Figure 8c), we examined 25 evenly spaced crops arranged in a 5x5 grid. The x-axis of the figure shows these crops ordered in scan-line order. The periodic nature of the graph shows that crops nearer the center of the image yield higher performance. There is virtually no difference between the invariance and equivariance measurements for the Crop CNNs, showing that the high level CNN features trained for classification are not predictive for image extrapolation. This is because the equivariance mapping for Crop transforms uses the CNN activations for some un-centered crop to predict the CNN activations over the center crop.
B.1.1 CNN ACCURACY
We also report the accuracy of each CNN in Table 3. While all transforms help improve invariance, we see that some input transformations help performance on the untransformed images, while other hurt performance. In particular, Crop and Shear transforms improve performance the most, while Elastic Deformations and Gaussian Blur seem to hurt performance most.
B.2 CROSS DATASET INVARIANCE/EQUIVARIANCE MEASUREMENTS
In this section, we describe and give results for a new experiment. While all previous experiments measure invariance/equivariance on the same dataset that the CNN was trained on (though with different splits), here we use a new dataset to measure the invariance/equivariance of CNNs trained on RVL-CDIP. The new dataset, ANDOC, is also composed of grayscale document images, though while RVL-CDIP is composed of scanned office documents, ANDOC is composed of digitized
historical documents, which differ significantly from modern office documents. As the RVL-CDIP CNNs were not trained to classify ANDOC documents, we resort to measuring the LL2 difference in the feature representations.
CNN RVL-CDIP ILSVRC CNN RVL-CDIP ILSVRC
In general, the measured invariance/equivariance is lower for ANDOC than for RVL-CDIP (see Figure 10 for examples). This makes sense because the CNNs learn to encode discriminative information about RVL-CDIP documents. However, the invariance and equivariance properties are not lost when switching to a new domain of data. In fact, for Mirror, Rotation, and Shear transforms, the invariance is higher, though this is likely due to less information about the input image being encoded for the new domain.
B.3 REPRESENTATION DISTANCE
In this section, we include additional results from the experiments in Section 6.2 that did not fit in the main text. Figure 11 shows an example T-SNE embedding of the CNN representations. There appears to be very little cluster structure and the T-SNE embeddings appear to be sensitive to the random initialization of the embedding vectors. The results from multiple runs of the T-SNE
embedding in general do not agree on either which points are outliers or on nearest neighbor pairs. This is likely because representation distances do not seem to follow the triangle inequality axiom.
While Eq. 4 yields a symmetric function by averaging two one-way distances, we find that for some pairs of CNNs, there is significant difference between the magnitudes of those one-way distances. In other words,
D′(φ1, φ2) = ∣∣∣min E1 [L(E1φ1(x), φ2)]−min E2 [L(φ1, E2φ2(x))] ∣∣∣ (9)
is large. Figure 12 plots a heat map of Eq. 9 applied pairwise to each CNN for each dataset. Optimization difficulties are one possible cause of large values in Figure 12. However, a more likely explanation is that some CNNs encode unique information about the input that cannot be predicted from the representations learned from other CNNs.
B.4 IMPROVED EQUIVARIANCE
Table 4 shows the results of finetuning using Eq. 7 for CNNs trained with different types of data augmentation. Overall, finetuning improves 3 of 9 transform types. We believe that performance does not improve for some transforms because the dataset naturally contains examples of those transforms due to different object poses. This is not the case for RVL-CDIP because the document images are always scanned in the same manner.

This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent.
Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well.
Major issues are in the experiments with the representation distances:
* The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained).
* The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all...
* It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task.
* The proposed method does not seem to improve equivariance consistently on all tasks. Especially with \lambda_1 and \lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer?
In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field.
There are also few minor issues:
* It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results.
* The angles on page 1 and 5 are missing units (degrees?).
* On page three, "In practice, it is difficult... ", it is not M_g which is maximised/minimised, but the loss over the M_g
* Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights
* Is the network for RVL-CDIP the same architecture as Alexnet?
* On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).


Local trends of time series characterize the intermediate upward and downward patterns of time series. Learning and forecasting the local trend in time series data play an important role in many real applications, ranging from investing in the stock market, resource allocation in data centers and load schedule in smart grid. Inspired by the recent successes of neural networks, in this paper we propose TreNet, a novel end-to-end hybrid neural network that predicts the local trend of time series based on local and global contextual features. TreNet leverages convolutional neural networks (CNNs) to extract salient features from local raw data of time series. Meanwhile, considering long-range dependencies existing in the sequence of historical local trends, TreNet uses a long-short term memory recurrent neural network (LSTM) to capture such dependency. Furthermore, for predicting the local trend, a feature fusion layer is designed in TreNet to learn joint representation from the features captured by CNN and LSTM. Our proposed TreNet demonstrates its effectiveness by outperforming conventional CNN, LSTM, HMM method and various kernel based baselines on real datasets.
1 INTRODUCTION
Time series, which is a sequence of data points in time order, is being generated in a wide spectrum of domains, such as daily fluctuation of the stock market, power consumption records of households, performance monitoring data of clusters in data centres, and so on. In many applications, users are interested in understanding the evolving trend in time series and forecasting the trend, since the conventional prediction on specific data points could deliver very little information about the semantics and dynamics of the underlying process generating the time series. For instance, time series in Figure 1 are from the household power consumption dataset1. Figure 1(a) shows some raw data points of time series. Though pointA andB have approximately the same value, the underlying system is likely to be in two different states when it outputs A and B, because A is in an upward trend whileB is in a downward trend (Wang et al., 2011; Matsubara et al., 2014). On the other hand, even when two points with the similar value are both in the upward trend, e.g, point A and C, the different slopes and durations of the trends where point A and C locate, could also indicate different states of the underlying process.
Particularly, in this paper we are interested in the local trend of time series which measures the intermediate local behaviour, i.e., upward or downward pattern of time series that characterized by the slope and duration (Wang et al., 2011). For instance, in Figure 1(b) the linear segments over raw data points of time series represent the local trends extracted from a real household power consumption time series. For the ease of presentation, we will use the term trend and local trend interchangeably in the rest of the paper. Learning and forecasting local trends are quite useful in a wide range of applications. For instance, in the stock market, due to its high volatility and noisy environment, in reality predicting stock price trends is preferred over the prediction of the stock market absolute values (Atsalakis & Valavanis, 2009). Predicting the local trend of stock price time series empowers
∗These two authors contributed equally. 1 https://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption
traders to design profitable trading strategies (Chang et al., 2012b; Atsalakis & Valavanis, 2009). In the smart energy domain, knowing the predictive local trend of power consumption time series enables energy providers to schedule power supply and maximize energy utilization (Zhao & Magoulès, 2012).
Meanwhile, in recent years neural networks have shown the dramatical power in a wide spectrum of domains, e.g., natural language processing, computer vision, speech recognition, time series analysis, etc. (Wang et al., 2016b; Sutskever et al., 2014; Yang et al., 2015; Lipton et al., 2015). For time series data, two mainstream architectures, convolutional neural network (CNN) and recurrent neural network (RNN) have been exploited in different time series related tasks, e.g., RNN in time series classification (Lipton et al., 2015) and CNN in activity recognition and snippet learning (Liu et al., 2015; Yang et al., 2015). RNN is powerful in discovering the dependency in sequence data (Jain et al., 2014; Graves, 2012) and particularly the Long Short-Term Memory (LSTM) RNN works well on sequence data with long-term dependencies (Chung et al., 2014; Hochreiter & Schmidhuber, 1997) due to the internal memory mechanism. CNN excels in exacting effective representation of local salience from raw data of time series by enforcing a local connectivity between neurons. (Yang et al., 2015; Hammerla et al., 2016).
In this paper, we focus on learning and forecasting the local trends in time series via neural networks. This involves learning different aspects of the data. On one hand, the sequence of historical local trends describes the long-term contextual information of time series and thus naturally affects the evolution of the following local trend. On the other hand, the recent raw data points of time series (Wang et al., 2011; Batal et al., 2012), which represent the local variation and behaviour of time series, affect the evolving of the following trend as well and have particular predictive power for abruptly changing local trends (Wang et al., 2011). For instance, in Figure 1(c), trend 1, 2 and 3 present a continuous upward pattern. Then when we aim at predicting the subsequent trend of time series at the end of the third local trend, the previous three successive upward trends outline a probable increasing trend afterwards. However, the local data around the end of the third trend, e.g., data points in the red circle, indicate that time series could stabilize and even decrease. The data points after the third trend indeed present a decreasing trend indicated by the red dotted segment. In this case, the subsequent trend has more dependency on the local data points. Therefore, it is highly desired to develop a systematic way to model such various hidden and complementary dependencies in time series for the local trend forecasting problem.
To this end, we propose a end-to-end hybrid neural network, referred to as TreNet. In particular, it consists of a LSTM recurrent neural network to capture the long dependency in historical local trends, a convolutional neural network to extract local features from local raw data of time series, and a feature fusion layer to learn joint representation to take advantage of both features drawn from CNN and LSTM. Such joint representation is used for the local trend forecasting. The experimental analysis on real datasets demonstrates that TreNet outperforms individual recurrent neural network, convolutional neural network and a variety of baselines in term of local trend prediction accuracy.
The rest of the paper is organized as follows. Section 2 presents related work, while Section 3 defines the problem to be solved and introduces the notations. In Section 4, we present the proposed TreNet. Section 5 demonstrates the performance of our method and baselines on real datasets. Finally, the paper is concluded in Section 6. Refer to Section 7 and Section 8 for more experiment results and discussion.
2 RELATED WORK
Traditional learning approaches over local trends of time series mainly make use of Hidden Markov Models (HMMs) (Wang et al., 2011; Matsubara et al., 2014). HMMs maintain short-term state dependences, i.e., the memoryless Markov property and predefined number of states, which requires significant task specific knowledge. RNNs instead use high dimensional, distributed hidden states that could take into account long-term dependencies in sequence data. Previous time series segmentation approaches (Keogh et al., 2001; Matsubara et al., 2014; Yuan, 2015) focus on achieving a meaningful segmentation and finding patterns, rather than modeling the relation in segments and therefore are not suitable for forecasting local trends. Multi-step ahead prediction is another way to realize local trend prediction by fitting the predicted values to estimate the local trend. However, multi-step ahead prediction is a non-trivial problem itself (Chang et al., 2012a). In this paper, we concentrate on directly learning local trends through neural networks.
RNNs have recently shown promising results in a variety of applications, especially when there exist sequential dependencies in data (Lyu & Zhu, 2014; Chung et al., 2014; Sutskever et al., 2014). Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997; Lyu & Zhu, 2014; Chung et al., 2014), a class of recurrent neural networks with sophisticated recurrent hidden and gated units, are particularly successful and popular due to its ability to learn hidden long-term sequential dependencies. (Lipton et al., 2015) uses LSTMs to recognize patterns in multivariate time series, especially for multi-label classification of diagnoses. (Chauhan & Vig, 2015; Malhotra et al., 2015) evaluate the ability of LSTMs to detect anomalies in ECG time series. Bidirectional LSTM (Graves & Schmidhuber, 2005) is usually intended for speech processing rather than time series forecasting problems. Our paper focuses on using LSTM to capture the dependency in the sequence of historical local trends and meanwhile the hidden states in LSTM are further used to learn joint feature representations for the local trend forecasting.
CNN is often used to learn effective representation of local salience from raw data (Vinyals et al., 2015; Donahue et al., 2015; Karpathy et al., 2014). (Hammerla et al., 2016; Yang et al., 2015; Lea et al., 2016) make use of CNNs to extract features from raw time series data for activity/action recognition. (Liu et al., 2015) focuses on the prediction of periodical time series values by using CNN and embedding time series with the potential neighbors in the temporal domain. Our proposed TreNet will combine the strengths of both LSTM and CNN and form a novel and unified neural network architecture for local trend forecasting.
Hybrid neural networks, which combines the strengths of various neural networks, are receiving increasing interest in the computer vision domain, such as image captioning (Mao et al., 2014; Vinyals et al., 2015; Donahue et al., 2015), image classification (Wang et al., 2016a), protein structure prediction (Li & Yu, 2016), action recognition (Ballas et al., 2015; Donahue et al., 2015) and so on. But efficient exploitation of such hybrid architectures has not been well studied for time series data, especially the trend forecasting problem. (Li & Yu, 2016; Ballas et al., 2015) utilize CNNs over images in cascade of RNNs in order to capture the temporal features for classification. (Bashivan et al., 2015) transforms EEG data into a sequence of topology-preserving multi-spectral images and then trains a cascaded convolutional-recurrent network over such images for EEG classification. (Wang et al., 2016a; Mao et al., 2014) propose the CNN-RNN framework to learn a shared representation for image captioning and classification problems. In our proposed TreNet, LSTM and CNN first respectively learn the trend evolution and local raw data of time series and then TreNet fuses the features captured by LSTM and CNN to predict the trend.
3 PROBLEM FORMULATION
In this section, we provide the formal definition of the trend learning and forecasting problem in this paper.
We define time series as a sequence of data points X = {x1, . . . , xT }, where each data point xt is real-valued and subscript t represents the time instant. The corresponding local trend sequence of X is a series of piecewise linear representations of X , denoted by T = {〈`k, sk〉}. Each element of T , e.g., 〈`k, sk〉 describes a linear function over a certain subsequence (or segment) of X and corresponds to a local trend in X . Such local trends in T are extracted from X by time series segmentation and fitting a linear function w.r.t. time t over each segment (Keogh et al., 2001; Wang
et al., 2011). `k and sk respectively represent the duration and slope of trend k. `k is measured in terms of the time range covered by trend k. Local trends in T are time ordered and non-overlapping. The durations of all the local trends in T address ∑ k `k = T . In addition, a local trend sequence
ending by time t is denoted by T (t) = {〈`k, sk〉 | ∑ k `k ≤ t}.
Meanwhile, as we discussed in Section 1, local raw data of time series affects the varying of trend as well and thus we define the local data w.r.t. a certain time instant t as a sequence of data points in a window of size w, denoted by L(t) = {xt−w, . . . , xt}. At certain time t, trend forecasting is meant to predict the duration and slope of the following trend based on a given sequence of historical trends T (t) and local data set L(t). The predicted duration and slope at time t are denoted by ˆ̀t and ŝt. Our proposed TreNet can be trained for predicting either ˆ̀t or ŝt. For simplicity, we use ŷt to represent the predicted value of TreNet throughout the paper.
Therefore, given the training datasetD = X∪T , we aim to propose a neural network based approach to learn a function ŷt = f(T (t),L(t)) for the trend forecasting. In this paper, we focus on univariate time series. The proposed method can be naturally generalized to multivariate time series as well by augmenting the input to the neural network. Refer to Section 8 for more discussion.
4 HYBRID NEURAL NETWORKS FOR TREND LEARNING AND FORECASTING
In this section, we first present an overview about the proposed TreNet for the trend forecasting. Then we will detail the components of TreNet.
Overview.
The idea of our TreNet is to combine CNN with LSTM to utilize their representation abilities on different aspects of training data D (D = X ∪ T ) and then to learn a joint feature for the trend prediction. Technically, TreNet is designed to learn a predictive function ŷt = f(R(T (t)), C(L(t))). R(T (t)) is derived by training the LSTM over sequence T to capture the dependency in the trend evolving, while C(L(t)) corresponds to local features extracted by CNN from L(t). The long-term and local features captured by LSTM and CNN, i.e., R(T (t)) and C(L(t)) convey complementary information pertaining to the trend varying. Therefore, the feature fusion layer is supposed to take advantages of both features to produce a fused representation for improved performance. Finally, the trend prediction is realized by the function f(·, ·), which corresponds to the feature fusion and output layers in Figure 2.
Learning the dependency in the trend sequence. During the training phase, the duration `k and slope sk of each local trend k in sequence T are fed into the LSTM layer of TreNet. Each j-th neuron in the LSTM layer maintains a memory cjk at step k. The output hjk or the activation of this neuron is then expressed as (Hochreiter & Schmidhuber,
1997; Chung et al., 2014):
hjk = o j ktanh(c j k) (1)
where ojk is an output gate and calculated as:
ojk = σ(Wo[`k sk] +Uohk−1 + Vock) j (2)
where [`k sk] is the concatenation of the duration and slope of the trend k, hk−1 and ck are the vectorization of the activations of {hjk−1} and {c j k}, and σ is a logistic sigmoid function. Then, the memory cell cjk is updated through partially forgetting the existing memory and adding a new memory content c̃jk:
cjk = f j kc j k−1 + i j k c̃ j k , c̃ j k = tanh(Wc[`k sk] +Uchk−1) j (3)
The extent to which the existing memory is forgotten is modulated by a forget gate f jk , and the degree to which the new memory content is added to the memory cell is modulated by an input gate ijk. Then, such gates are computed by
f jk = σ(Wf [`k sk] +Ufhk−1 + Vfck−1) j (4)
ijk = σ(Wi[`k sk] +Uihk−1 + Vick−1) j (5)
At each step k, the hidden activation hk is the output to the feature fusion layer. Specifically, given a T (t) containing n local trends (i.e., |T (t)| = n), the output of R(T (t)) is R(T (t)) = hn.
Learning features from the local raw data of time series. When the k-th trend in T is fed to LSTM, the corresponding local raw time series data input to
the CNN part of TreNet is L(t), where t = k∑
i=1
`i. CNN consists of H stacked layers of 1-d
convolutional, activation and pooling operations. Denote by ai the input signal of layer i and thus at the first layer a1 = L(t). Each layer has a specified number of filters ni of a specified filter size di. Each filter on a layer sweeps through the entire input signal to exact local features as follows:
vi,jm = φ(b i,j + m+di/2∑ z=m−di/2 W i,jz a i z) ,∀m = 1, . . . , |ai| (6)
where vi,jm is the activation of j-th filter of layer i on m position of the input signal. Here φ is the Leaky Rectified Linear Unit, which is shown to perform better (Xu et al., 2015). Then the maxpooling is performed over the vi,jm of each filter.
Finally, the output of CNN in TreNet is the concatenation of max-pooling of each filter on the last layer H , namely:
C(L(t)) = [p1, . . . , pn H
], pj = [ max 1≤z≤q ({vH,jm+z})], ∀j = 1, . . . , nH (7)
where q is the pooling size.
Feature fusion and output layers. The feature fusion layer combines the representations R(T (t)) and C(L(t)), to form a joint feature. Then, such joint feature is fed to the output layer to provide the trend prediction. Particularly, we first map R(T (t)) and C(L(t)) to the same feature space and add them together to obtain the activation of the feature fusion layer (Mao et al., 2014). The output layer is a fully-connect layer following the feature fusion layer. Mathematically, the prediction of TreNet is expressed as:
ŷt = f(R(T (t)), C(L(t))) =W o · φ(W r ·R(T (t)) +W c · C(L(t)))︸ ︷︷ ︸ feature fusion +bo (8)
where φ(·) is element-wise leaky ReLU activation function and + denotes the element-wise addition. W o and bo are the weights and bias of the output layer.
To train TreNet, we adopt the squared error function plus a regularization term as:
J(W , b ; T ,X ) = 1 |T | |T |∑ k=1 (ŷk − yk)2 + λ‖W ‖2 (9)
where W and b represent the weight and bias parameters in TreNet, λ is a hyperparameter for the regularization term and yk is the true value of trend slope or duration.
The cost function is differentiable and the architecture of TreNet allows the gradients from the loss function (9) to be backpropagated to both LSTM and CNN parts. TreNet can be trained respectively for the slope and duration of local trends using T and X . When performing forecasting, T (t) and L(t) are fed to TreNet and the prediction value ŷk could be either the slope or duration depending on the training target.
5 EXPERIMENTAL ANALYSIS
In this section, we conduct extensive experiments to demonstrate the prediction performance of TreNet by comparing to a variety of baselines. Due to the page limit, refer to Section 7 for more experiment results.
5.1 EXPERIMENT SETUP
Dataset: We test our method and baselines on three real time series datasets.
• Daily Household Power Consumption (HousePC). This dataset2 contains measurements of electric power consumption in one household with a one-minute sampling rate over a period of almost 4 years. Different electrical quantities and some sub-metering values are available. We use the voltage time series throughout the experiments.
• Gas Sensor (GasSensor). This dataset3 contains the recordings of chemical sensors exposed to dynamic gas mixtures at varying concentrations. The measurement was constructed by the continuous acquisition of the sensor array signals for a duration of about 12 hours without interruption. We mainly use the gas mixture time series regarding Ethylene and Methane in air.
• Stock Transaction (Stock): This dataset is extracted from Yahoo Finance and contains the daily stock transaction information in New York Stock Exchange from 1950-10 to 2016-4.
All datasets are preprocessed by (Keogh et al., 2001) to extract local trends. Alternative time series segmentation and local trend extraction approaches can be used as well. We choose (Keogh et al., 2001) here due to its high efficiency. Totally, we obtain 42591, 4720 and 1316 local trends respectively from above datasets. For the ease of experimental result interpretation, the slope of extracted local trends is represented by the angle of the corresponding linear function and thus in a bounded value range [−90, 90]. The duration of local trends is measured by the number of data points within the local trend. Then, the obtained trend sequences and the set of local data are split into training (80%), validation (10%) and test (10%) datasets.
Baselines: We compare TreNet with the following six baselines:
• CNN. This baseline method predicts the trend by only using CNN over the set of local raw data of time series to learn features for the forecasting. The size of local data is set at w as is defined in Section 3.
• LSTM. This method uses LSTM to learn dependencies in the trend sequence T and predicts the trend only using the trained LSTM.
• Support Vector Regression (SVR). A family of support vector regression based approaches with different kernel methods is used for the trend forecasting. We consider three
2 https://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption 3 https://archive.ics.uci.edu/ml/datasets/Gas+sensor+array+under+dynamic+gas+mixtures
commonly used kernels (Liu et al., 2015), i.e., Radial Basis kernel (SVRBF), Polynomial kernel (SVPOLY), Sigmoid kernel (SVSIG). The trend sequence and the corresponding set of local time series data are concatenated as the input features to such SVR approaches.
• Pattern-based Hidden Markov Model (pHMM). (Wang et al., 2011) proposed a patternbased hidden Markov model (HMM), which segments the time series and models the dependency in segments via HMM. The derived HMM model is used to predict the state of time series and then to estimate the trend based on the state.
• Naive. This is the naive approach which takes the duration and slope of the last trend as the prediction for the next one.
• ConvNet+LSTM(CLSTM). It is based on the cascade structure of ConvNet and LSTM in (Bashivan et al., 2015) which feeds the features learnt by ConvNet over time series to a LSTM and obtains the prediction from the LSTM.
Evaluation metric: We evaluate the predictive performance of TreNet and baselines in terms of Root Mean Square Error (RMSE). The lower the RMSE, the more accurate the predictions.
Training: The training procedure of TreNet and baselines in our paper follows the schema below.
The CNN and LSTM components in TreNet share the same network structure (e.g., number of layers, neurons in each layer) as CNN and LSTM baselines. CNN has two stacked convolutional layers, which have 32 filters of size 2 and 4. The number of memory cells in LSTM is 600. For baseline CNN and LSTM, we tune the learning rate for each approach from {10−1, 10−2, 10−3, 10−4, 10−5} (Sutskever et al., 2013), in order to achieve the least prediction errors and then fix the learning rate. For TreNet, in addition to the learning rate, the number of neurons in the feature fusion layer is chosen from the range {300, 600, 900, 1200} to achieve the best performance. We use dropout and L2 regularization to control the capacity of neural networks to prevent overfitting, and set the values to 0.5 and 5 × 10−4 respectively for all datasets (Mao et al., 2014). The Adam optimizer (Kingma & Ba, 2014) is chosen to learn the weights in neural networks.
Regarding the SVR based approaches, we carefully tune the parameters c (error penalty), d (degree of kernel function), and γ (kernel coefficient) for kernels. Each parameter is selected from the sets c ∈ {10−5, 10−4, . . . , 1, . . . , 104, 105}, d ∈ {1, 2, 3}, γ ∈ {10−5, 10−4, . . . , 1, . . . , 105} respectively. We iterate through candidate values of each combination of c, d and γ to train our model, and keep the parameters that generate the lowest RMSE on the validation set, and then use them to predict on the test set.
The training datasets of SVR and pHMM baselines are consistent as that of TreNet. Likewise, CNN and LSTM baselines are respectively fed by the set of local data and the trend sequence of the same size as TreNet. In addition, since the window size of local data is tunable, we vary the window size of local data, i.e. w, from the range {100, 300, 500, 700, 900}, so as to investigate how the size of local data influences the predication performance. The results will be presented in Section 5.2. The model’s performance on the validation set will be evaluated after each epoch of training. Each model is trained for at least 50 epochs. Meanwhile, the training process adopts early stopping if no further improvement in the performance of validation shows up after 50 epochs.
5.2 EXPERIMENT RESULTS
Table 1 studies the prediction performances of TreNet and baselines. For each dataset, the window size of local data is constant for approaches (i.e., CNN, SVRBF, SVPOLY, SVSIG, pHMM and TreNet) that take local data as input. Then, the results of each approach are obtained by tuning the corresponding parameter as described in Section 5.1.
In Table 1, we observe that TreNet consistently outperforms baselines on the duration and slope prediction by achieving around 30% less errors at the maximum. It verifies that the hybrid architecture of TreNet can improve the performance by utilizing the information captured by both CNN and LSTM. Specifically, pHMM method performs worse due to the limited representation capability of HMM. On the slope prediction, SVR based approaches can get comparable results as TreNet.
In the following group of experiments, we investigate the effect of local data size (i.e., w) on the prediction. In particular, we tune the value of local data size for the approaches whose input fea-
tures contains local data and observe the prediction errors. Such approaches include CNN, SVRBF, SVPOLY, SVSIG, pHMM and TreNet. LSTM only consumes the trend sequence and thus is not included. Due to the page limit, we report the results on the HousePC dataset in Table 2 and Table 3. The results on Stock and GasSensor datasets can be referred to Section 7.
Baseline Naive has no original time series data as input CLSTM works on the whole time series and has no local data. Thus they are excluded from this set of experiments.
In Table 2, we observe that compared to baselines TreNet has the lowest errors on the duration prediction across different window sizes. pHMM requires sufficient data points to model the relations of segments and fails to work on 100 size. As the window size increases and more local data points are fed to the training process, the prediction errors of CNN and TreNet decrease or nearly stabilize. This could be because only the certain amount of local data has predictive power. The filtering and pooling mechanism enables CNN to focus on the certain local data having strong predictive power and thus giving more local data only gives rise to marginal improvements. Such similar phenomenon is observed on the slope prediction as is shown in Table 3. For more results and discussion, please refer to Section 7.
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
6 CONCLUSION
In this paper we propose TreNet, a novel hybrid neural network to learn and predict the local trend behaviour of time series. The experimental results demonstrate that such a hybrid framework can indeed utilize complementary information extracted by CNN and LSTM to enhance the prediction performance. Moreover, such architecture is generic and extendible in that additional exogenous time series can be fed to TreNet, so as to boost the performance and investigate the effect of different data sources on the trend evolving.
7 APPENDIX
7.1 DATA PRE-PROCESSING
In this part, we describe the data pre-processing, which extracts the local trend sequence from raw time series data for the subsequent neural network training and testing.
We convert the raw time series data into a piecewise linear representation, namely consecutive segments (Keogh et al., 2001; Wang et al., 2011). Each segment corresponds to a local trend and is fitted by a linear function of time series value w.r.t. time, e.g., xt = β1t+β0+ over the time range [t1, t2) of this segment. Then, the slope and duration are derived from the coefficient β1 and [t1, t2).
Technically, we adopt the bottom-up approach in (Keogh et al., 2001), since it can achieve lower approximate errors compared with top-down and sliding window methods. The process is illustrated in Figure 3. Initially, we approximate time series X with bT2 c line segments (T is the length of the
time series). Then, we iteratively merge the neighbouring segments to build longer ones. In each iteration, neighbouring segments with the minimal approximation error are merged into a new one. The merging process repeats until every possible merge gives rise to a segment with errors above a specified threshold. We use the relative mean squared error as the error metric and specify the threshold as 0.05.
7.2 ADDITIONAL EXPERIMENT RESULTS
In this group of experiments, we visualize the trend prediction using the sample testing data instance from each dataset in Figure 4. We can observe that in HousePC TreNet successfully predicts the changed trend, though there are successive upward trends before. In Stock and GasSensor datasets, the succeeding upward and downward trends are correctly predicted as well.
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
Then, we provide the RMSE w.r.t. the varying window size on Stock and GasSensor datasets in Table 4, Table 5, Table 6 and Table 7.
From the results, we observe that TreNet outperforms baselines almost on all window sizes. Meanwhile, the prediction errors often present the decreasing and stable pattern as the window size varies.
Window size of local data: The observation in above experiments w.r.t. the varying window size provides inspiration for choosing the window size of local data. Given the training dataset, we can find out the maximum duration of local trends and takes it as the local data size. This is because doing so can ensure that the range of local data in each training instance can cover the most recent local trend, whose raw data is believed to have strong predictive power for the subsequent trend. Additionally, we observe that setting the window size of local data of CNN and TreNet in this way can achieve comparable prediction errors compared to the cases with larger window sizes .
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
Window Size CNN SVRBF SVPOLY SVSIG pHMM TreNet
8 DISCUSSION
For multivariate time series, we can augment the input of TreNet by including the trend sequences and local data of exogenous time series and then train TreNet for a certain target time series to predict its trend. Another line of research is to explore equipping TreNet with multi-task learning. This is motivated by the observation that if we decompose the trend forecasting problem into classification and regression respectively for the slope and duration, we can utilize the correlation between slope
and duration to boost the prediction performance. In addition, there could be alternative frameworks to combine the outputs of CNN and LSTM and our work opens the door for applying hybrid neural networks for trend analysis in time series.

Updated review: the authors did an admirable job of responding to and incorporating reviewer feedback. In particular, they put a lot of effort into additional experiments, even incorporating a new and much stronger baseline (the ConvNet -> LSTM baseline requested by multiple reviewers). I still have two lingering concerns previously stated -- that each model's architecture (# hidden units, etc.) should be tuned independently and that a pure time series forecasting baselines (without the trend preprocessing) should be tried. I'm going to bump up my score from a clear rejection to a borderline.
-----
This paper is concerned with time series prediction problems for which the prediction targets include the slope and duration of upcoming local trends. This setting is of great interest in several real world problem settings (e.g., financial markets) where decisions (e.g., buy or sell) are often driven by local changes and trends. The primary challenge in these problems is distinguishing true changes and trends (i.e., a downturn in share price) from noise. The authors tackle this with an interesting hybrid architecture (TreNet) with four parts: (1) preprocessing to extract trends, (2) an LSTM that accepts those trends as inputs to ostensibly capture long term dependencies, (3) a ConvNet that accepts a local window of raw data as its input at each time step, and (4) a higher "feature fusion" (i.e., dense) layer to combine the LSTM's and ConvNet's outputs. On three univariate time series data sets, the TreNet outperforms the competing baselines including those based on its constituent parts (LSTM + trend inputs, CNN).
Strengths:
- A very interesting problem setting that can plausibly be argued to differ from other sequential modeling problems in deep learning (e.g., video classification). This is a nice example of fairly thoughtful task-driven machine learning.
- Accepting the author's assumptions as true for the moment, the proposed architecture seems intuitive and well-designed.
Weaknesses:
- Although this is an interesting problem setting (decisions driven by trends and changes), the authors did not make a strong argument for why they formulated the machine learning task as they did. Trend targets are not provided from "on high" (by data oracle) but extracted from raw data using a deterministic algorithm. Thus, one could just easily formulate this as plain time series forecasting problem in which we forecast the next 100 steps and then apply the trend extractor to convert those predictions into a trend. If the forecasts are accurate, so will be the extracted trends.
- The proposed architecture, while interesting, is not justified, in particular the choice to feed the extracted trends and raw data into separate LSTM and ConvNet layers that are only combined at the end by a shallow MLP. An equally straightforward but more intuitive choice would have been to feed the output of the ConvNet into the LSTM, perhaps augmented by the trend input. Without a solid rationale, this unconventional choice comes across as arbitrary.
- Following up on that point, the raw->ConvNet->LSTM and {raw->ConvNet,trends}->LSTM architectures are natural baselines for experiments.
- The paper presupposes, rather than argues, the value of the extracted trends and durations as inputs. It is not unreasonable to think that, with enough training data, a sufficiently powerful ConvNet->LSTM architecture should be able to learn to detect these trends in raw data, if they are predictive.
- Following up on that point, two other obvious baselines that were omitted: raw->LSTM and {raw->ConvNet,trends}->MLP. Basically, the authors propose a complex architecture without demonstrating the value of each part (trend extraction, LSTM, ConvNet, MLP). The baselines are unnecessarily weak.
One thing I am uncertain about in general: the validity of the practice of using the same LSTM and ConvNet architectures in both the baselines and the TreNet. This *sounds* like an apples-to-apples comparison, but in the world of hyperparameter tuning, it could in fact disadvantage either. It seems like a more thorough approach would be to optimize each architecture independently.
Regarding related work and baselines: I think it is fair to limit the scope of in-depth analysis and experiments to a set of reasonable, representative baselines, at least in a conference paper submitted to a deep learning conference. That said, the authors ignored a large body of work on financial time series modeling using probabilistic models and related techniques. This is another way to frame the above "separate trends from noise" problem: treat the observations as noisy. One semi-recent example: J. Hernandez-Lobato, J. Lloyds, and D. Hernandez-Lobato. Gaussian process conditional copulas with applications to financial time series. NIPS 2013.
I appreciate this research direction in general, but at the moment, I believe that the work described in this manuscript is not suitable for inclusion at ICLR. My policy for interactive review is to keep an open mind and willingness to change my score, but a large revision is unlikely. I would encourage the authors to instead use their time and energy -- and reviewer feedback -- in order to prepare for a future conference deadline (e.g., ICML).