Representation Learning via Global Temporal Alignment and Cycle-Consistency

We introduce a weakly supervised method for representation learning based on aligning temporal sequences (e.g., videos) of the same process (e.g., human action). The main idea is to use the global temporal ordering of latent correspondences across sequence pairs as a supervisory signal. In particular, we propose a loss based on scoring the optimal sequence alignment to train an embedding network. Our loss is based on a novel probabilistic path finding view of dynamic time warping (DTW) that contains the following three key features: (i) the local path routing decisions are contrastive and differentiable, (ii) pairwise distances are cast as probabilities that are contrastive as well, and (iii) our formulation naturally admits a global cycle consistency loss that verifies correspondences. For evaluation, we consider the tasks of fine-grained action classification, few shot learning, and video synchronization. We report significant performance increases over previous methods. In addition, we report two applications of our temporal alignment framework, namely 3D pose reconstruction and fine-grained audio/visual retrieval.

Introduction
Temporal sequences (e.g., videos) are an appealing data source as they provide a rich source of information and additional constraints to leverage in learning. By far the main focus on temporal sequence analysis has been on learning representations targeting distinctions at the global signal level, e.g., action classification, where abundant labeled data is available for training. In this paper, we target a weakly-supervised training regime for representation learning, capable of making fine-grained temporal distinctions.
Most previous approaches to temporally fine-grained understanding of sequential signals have considered fullysupervised training methods (e.g., [56]), where labels are provided at the sub-sequence level, e.g., frames. The major drawback of these methods is the expense in acquiring dense labels, and their subjective nature. In contrast, a key consideration in our work is the selection of training signals embedding space video 1 video 2 Figure 1. We introduce a representation learning approach based on (globally) aligning pairs of temporal sequences (e.g., video) depicting the same process (e.g., human action). Our training objective is to learn an element-wise embedding function that supports the alignment process. For example, here we illustrate the alignment (denoted by black dashed lines) in the embedding space between videos of the same human action (i.e., tennis forehand) containing significant variations in their appearances and dynamics. Empirically, we show that our learned embeddings are sensitive to both human pose and fine-grained temporal distinctions, while being invariant to appearance, camera viewpoint, and background. by taking a probabilistic path finding view of DTW that encompasses the following three key features. First, we introduce a differentiable smoothMin operator that effectively selects each successive path extension. Moreover, we show that this operator has a contrastive effect across paths which is missing in previous differentiable DTW formulations [30,8,6]. Second, the pairwise embedding similarities that form our cost function are defined as probabilities, using the softmax operator. Optimizing our loss is shown to correspond to finding the maximum probability of any feasible alignment between the paired sequences. The softmax operator over element pairs also provides a contrastive component which we show is crucial to prevent the model from learning trivial embeddings. This forgoes the need for a downstream discriminative loss and the corresponding non-trivial task of defining negative alignments, e.g., [8,6]. Third, as an additional supervisory signal, our probabilistic framework admits a straightforward global cycle-consistency loss that matches the alignments recovered through a cycle of sequence pairings. Collectively, our method takes into account long-term temporal information that allows us to learn embeddings sensitive to fine-grained temporal distinctions (e.g., human pose), while being invariant to nuisance variables, e.g., camera viewpoint, background, and appearance.
Contributions. We make the following key contributions: • A novel weakly supervised method for representation learning tasked with discovering the alignment between sequence pairings for the purpose of finegrained temporal understanding. • A differentiable DTW formulation with two novel features: (i) a smoothMin operation that admits a probabilistic path interpretation and is contrastive across alternative paths, and (ii) a probabilistic data term that is contrastive across alternative data pairs. • A global cycle consistency loss to further enforce the temporal alignment. • An extensive set of evaluations, ablations, and comparisons with previous methods. We report significant performance increases on several tasks requiring finegrained temporal distinctions. • Two downstream applications, namely 3D pose reconstruction and audio-visual retrieval. Our code and trained models will be available at: https://github.com/hadjisma/VideoAlignment.

Related work
Representation learning. Most focus in representation learning with videos has been cast in a fully supervised setting, e.g., [47,43,7,16]. Self-supervised learning with images or videos has emerged as a viable alternative to supervised learning, where the supervisory signal is obtained from the data. For video, a variety of proxy tasks have been defined in lieu of training with annotations, such as classifying whether video frames are in the correct temporal order (e.g., [32,17,28,4,55,54]), predicting whether a video is played at a normal or modified rate [3], solving a spatiotemporal jigsaw puzzle task [1], predicting figureground segmentation [35], predicting pixel [57,29,50,26] or region correspondences [52,53,25] across neighbouring video frames, and predicting some aspect of future frames conditioned on past frames [49,51,21]. Others have considered multimodal settings, such as predicting video-audio misalignment [34]. Similar to [15], our method is best characterized as weakly supervised, where sequence-level labels are used to determine sequence pairings for training.
Sequence alignment. Several methods [42,40] assume paired, temporally synchronized videos of the same physical event for the purpose of representation learning. In contrast, and more closely related to our work, are methods that seek the alignment between sequences capturing the same process. One approach [15] is to cast the learning objective as maximizing the number of elements between sequences that can be brought into one-to-one correspondence via (soft) nearest neighbours. This method does not leverage the long-term temporal structure of the sequences as done in dynamic time warping (DTW) [38]. Given a cost function, DTW finds the optimal alignment between two sequences defined between elements comprising the sequences. Recent efforts [13,30,5,8,6] have explored differentiable approximations of the discrete operations underlying DTW to allow gradient-based training. Similar to recent work [8,6], we also incorporate a relaxed DTW as our loss for sequence alignment. Our formulation is probabilistic and includes a contrastive definition of the elementwise similarities. A key distinction with these prior works and our own, beyond differences in the target application domain (e.g., video-transcript alignment [8]), is that rather than incorporate contrastive modelling after the DTW step (e.g., through the use of margin-based loss), our method includes contrastive signals in both the differentiable min approximation and the pairwise matching cost function used in our DTW framework.
Contrastive learning. One can also draw parallels with contrastive learning using the cross-entropy loss (i.e., negative log softmax) [20,33,48], where the goal is to learn a representation that brings different views of the same data together (i.e., positives) in the embedding space, while pushing views of different data (i.e., negatives) apart. This amounts to encoding information shared across the views, while eschewing unique factors to each view. To construct different views, previous work has explored a variety of augmentation and sampling schemes [14,45,10,21,19,23] and correspondences across different modalities (e.g., video-audio, video-text, and luminance-depth

Technical approach
In this section, we describe our weakly-supervised approach to representation learning based on the alignment of sets of sequence pairs that capture the same process. Our learning objective is the training of a shared embedding function applied to each sequence element. In the case of multimodal sequences (e.g., audio-video), we have separate encoders for each modality that map their inputs to a common embedding space. Figure 2 provides an illustrative overview of our alignment approach to representation learning, which we fully unpack in the following subsections.

Background
Dynamic time warping (DTW) computes the optimal alignment between two sequences, X and Y, subject to certain alignment constraints.
Let X = x 1 x 2 · · · x M ∈ R D×M and Y ∈ R D×N denote the two sequences, where D corresponds to the dimensionality of the constituent sequence elements and M , N the respective sequence lengths. Given a cost matrix, C ∈ R M ×N , with elements c i,j defined by a cost function, c(x i , y j ), that measures the cost of matching elements x i and y j , we seek a feasible path between c 0,0 and c M,N that minimizes the total accumulated cost. The feasible paths are subject to matching endpoints, monotonicity, and continuity constraints. While not used here, the endpoint constraint can be relaxed to allow for subsequence matching, e.g., [39,6].
The number of feasible alignments in DTW grows exponentially with the sequence lengths. Fortunately, the structure of DTW with an appropriate cost function, c(x i , y j ), is amenable to dynamic programming [2] which shares both quadratic time and space complexity. The optimal alignment (i.e., path through the cost matrix) is found by evaluating the following recurrence [38]: where R(0, 0) = 0, R(0, :) = R(:, 0) = ∞, and R(i, j) stores the partial accumulated cost along the optimal and feasible path ending with the alignment between x i and y j . The minimum operation amounts to a first-order Markov assumption, where the local path routing is deterministic. Due to the discrete nature of the min operator in (1), responsible for local correspondence decisions, several works [13,8,6] have considered smooth variants suitable for gradient-based training. In Sec. 3.2, we introduce a smooth relaxation of the min operator with favourable properties for our representation learning setting. Then in Sec. 3.3, we define our cost function which introduces a contrastive learning signal throughout the alignment process.

Local differentiable decisions
For brevity we use the notation to denote the incoming optimal accumulated costs from the feasible paths leading into (i, j). We first modify (1) as where s(r i,j ) is a smooth approximation of the minimum operator. The term d(r i,j ) can be seen as a (nondifferentiable) additional penalty on any path that reaches (i, j). With this added penalty term, (3) reduces to simply For an appropriate choice of s(·) the right hand side is now differentiable. Note that any path that is optimal according to (4) will correspond to a feasible path for the original DTW problem (although perhaps not optimal for that problem). Moreover, the cost of such a path according to (4) will be the original cost plus the sum of the penalties d(r i,j ) over all points (i, j) on that path. We are left with choosing a smooth approximation s(·) for the minimum operator. Here, we use a standard relaxation of the min operator [27] (specifically, the expected value where γ denotes a temperature hyper-parameter. We refer to solving the recurrence relation (4), with the function s(·) taken to be smoothMin, as the smoothDTW problem. Previous alignment methods [13,8,6] have instead used the following min γ formulation as a continuous approxima-  Figure 2. Our sequence alignment approach to representation learning begins by encoding each element comprising our sequences (e.g., image frames) using a trainable framewise backbone encoder plus embedding network, φ(·), yielding two sequences of embeddings, X and Y. The cost of matching these two sequences is expressed as negative log probabilities and consists of two parts: (i) alignment losses, smoothDTW(·, ·), from X to Y and Y to X based on the cumulative cost along the optimal respective paths and (ii) a global cyclicconsistency loss that verifies the correspondences computed between each ordered pair of sequences, where · denotes matrix multiplication and IM×M is the square identity matrix. Note, our alignment cost smoothDTW(·, ·) is not symmetric in its two arguments (due to the pairwise matching cost in (8)). Higher intensities in the cells comprising the accumulated cost matrices indicate lower values.   (5) and (6). For simplicity we can assume the ri's are in sorted order. We evaluate two extreme cases where r2 = r1 + ∆ for some ∆ ≥ 0 and the next largest value, r3, is either equal to r2 or much larger. It can be shown that d(r; γ) = γd(r/γ; 1) and therefore we need only plot one scale-invariant curve for each case.
tion of the min operator: where again γ denotes a temperature hyper-parameter. While both the min γ and smoothMin operators are differentiable approximations of the min operator (with the min operator subsumed as a special case), their different behaviours have profound effects on learning in our setting. These differences are illustrated in the plot in Fig. 3, where without loss of generality we assume the costs are sorted in increasing order, with r 2 = r 1 + ∆ for some ∆ ≥ 0 and r 3 ≥ r 2 . As can be seen, the min γ function is strictly monotonically increasing. As a result, with all other things being equal, minimizing this function encourages ties, i.e., the penalty is minimized only when ∆ = 0. This is an undesirable behaviour as we seek the resulting embeddings to yield a well-defined path (i.e., optimal alignment) in our cumulative cost matrix, R(i, j). In contrast, our smoothMin operator defines a contrastive watershed (at approximately 1.5γ), where values to the left of the watershed encourage ties, while to the right the values are encouraged to be well separated. Moreover, since d(r; γ) ≥ 0 for smoothMin, our smoothDTW approach always provides an upper bound on the cost of the optimal path. The supplemental presents an expanded discussion and comparison.

Contrastive cost function
To complete the definition of our smoothDTW recurrence in (4), we now specify the cost function c(x i , y j ). Specifically, for each element x i in X, we wish to express the cost of matching x i to any single item y j in Y, given that at least one of the elements in Y must match. Moreover, in keeping with our probabilistic path finding formulation, c(x i , y j ) should be the negative log probability of matching the given x i to a selected y j in Y. This leads to the (non-symmetric) contrastive formulation where the softmax 2 operator is defined as the standard softmax with a temperature hyper-parameter, β, over the second matrix dimension, i.e. columns. Also the notatioñ x i denotes L 2 normalization, sox i = x i /||x i || 2 , and so on. The use of the negative log softmax operator over the set of correspondences in (8) encourages a (soft) winnertake-all, where one pair (x i , y j ) has a significantly higher cosine similarity than all the other options. The normalization across the column penalizes situations without a clear winner.
With this cost function the resulting values R(i, j) are the optimum value of the negative log probability for any feasible path starting at (0, 0) and ending at (i, j). Here, this negative log probability is the sum of the matching cost, c(x m , y n ) (cf. (8)), and the smoothness penalty d(r m,n ) (cf. (14)), at each vertex, (m, n), along the optimal path ending at (i, j). Correspondingly, we define the alignment loss for matching X to Y as: We use the sum of the alignment losses for matching X to Y and vice versa as the overall alignment loss, as shown in the left panel in Fig. 2.
The combination of the softmax over elements in Y, in (8), and the use of smooth DTW to formulate the alignment cost (9), rewards embeddings that are both: a) contrastive across the elements in Y; and, moreover, b) have their best matching pairs (x i , y j ) arranged along a feasible path from (0, 0) to (M, N ). We show in our ablation study that the ability to leverage these two properties during training are key to our utilization of the temporal alignment proxy. In contrast, dropping the softmax and simply computing the inner-product between elements could lead to the collapse of the embeddings around a single point during training, thus allowing the network to trivially minimize the alignment cost. To avoid such collapse, previous DTW-based methods have resorted to adding a discriminative loss downstream [8,6].

Global cycle-consistency
An additional loss is based on the notion that the match from sequence X to Y , composed with the match from Y to X, should ideally be the identity. We formulate this directly in terms of the cumulative cost matrix R X,Y for matching sequence X to Y (as defined by (4), (5), and (8)), along with the cost matrix for matching Y to X, namely R Y,X . Given the interpretation that R X,Y (i, j) is the optimal negative log probability of a path from (0, 0) to (i, j) for matching X to Y , consider the implied conditional distribution for matching the prefix sequences Note that this distribution does not use any information for elements k > i from sequence X, and is only obtained from the forward pass of matching X to Y . We use the notation to denote the N × M matrix with elements (P X,Y ) n,m = p X,Y (n | m). Ideally, the contrastive matching in (8) is sharp and forms a feasible path from (0, 0) to (i, j), thereby providing a strongly peaked conditional distribution p X,Y (j|i) for each i. However, without knowing the ground truth matching {(i k , j k )} L k=0 , we cannot use an explicit log-likelihood loss. This issue can be avoided by considering the composed conditional distribution which is formed by treating p X,Y (k | i) and p Y,X (j | k) as conditionally independent distributions. It is easy to verify that this is indeed a distribution over elements j of X. Moreover, following the above matrix notation, it is represented by the M × M matrix P Y,X P X,Y .
In the ideal case, this transport from elements in one sequence to another and back again should return to the same starting element. From (12) this corresponds to P Y,X P X,Y equaling the identity matrix, I M ×M . Thus, our global cycle-consistency loss is the sum of cross-entropy losses: The right panel in Fig. 2 provides a summary of our global cycle-consistency loss.

Training and implementation details
Our final loss function is obtained by combining the contrastive alignment loss, (9), and the global cycle consistency loss, (13), according to (14) where λ g and λ s are weights used to balance the two losses and are empirically set to 1.0 and 0.1, respectively. The temperature hyper-parameters, γ and β used in (5) and (8) are both set to 0.1, while α in (10) is 1.
This overall loss is used to train a convolutional architecture composed of a backbone encoder applied framewise followed by an embedding network. Specifically, ResNet50-v2 [24] is used as our backbone encoder where we extract features from Conv4c layer. We adopt the same embedding network used in previous related work [15] comprised of two 3D convolutional layers, a global 3D max pooling layer, two fully connected layers, and a linear projection layer. The final embedding is L2 normalized. To learn over sequence pairs (X, Y), we randomly extract T = 20 frames from each sequence. Sampling of video frames is random to avoid learning potential trivial solutions that may arise from strided sampling. For a fair comparison to previous approaches using the same architecture (i.e., [15,40,32]), we use the same batch size of four sequences. Finally, our learning rate is fixed to 10 −4 for all our experiments.

Empirical evaluation
We evaluate the efficacy of our learned embeddings on challenging temporal fine-grained tasks, thereby going beyond traditional clip-level recognition tasks. In particular, our proposed loss is evaluated on fine-grained action recognition (i.e., action phase classification), few-shot finegrained classification, and video synchronization. In addition, we also show that learning to align temporal sequences supports different downstream applications such as synchronous playback, 3D pose reconstruction and finegrained audio/visual retrieval.

Datasets
To evaluate our method, we use the PennAction [58] and FineGym [41] video datasets. Both datasets contain a diverse set of videos of human-related sports or fitness activities. These datasets are selected as they allow for learning framewise alignments and evaluating on tasks where finegrained temporal distinctions are critical. PennAction [58] contains 2326 videos of humans performing 15 different sports or fitness actions. The videos are tightly cropped temporally around the start and end of the action. Notably, 13 categories contain non-repetitive actions. The dataset also includes ground truth 2D keypoint labels which we later use to demonstrate a 3D pose reconstruction application grounded on our alignment method. FineGym [41] is a recent large-scale fine-grained action recognition dataset that was specifically designed to evaluate the ability of an algorithm to parse and recognize the different phases of an action. Each video in FineGym is annotated according to a three-level hierarchy denoting the event being performed in the video, the different sets involved in performing the event, and the framewise elements (i.e., action phases) involved in each set. To perform any eventlevel action, a gymnast may perform the different sets in any order. To train embeddings using our alignment-based method, we re-organize the FineGym dataset such that all sets belonging to the same event appear in the same order in any given video. An example of this re-organization is provided in the supplemental. Also, it should be noted that while the videos of the vault event (VT) in FineGym depict the gymnast performing the action in three phases, the first two phases of the action are not explicitly labeled in the original dataset. For the sake of completeness we use the provided start and end times for these phases, thereby adding two new fine-grained action labels in FineGym. The  Table 1. Ablation study of the various components of our loss function, (14). Contrastive-Cost refers to our columnwise contrastive cost, (8), and GCC refers to our global cycle consistency loss, (13).
remaining events in FineGym are otherwise unchanged. To account for these minor additions to the annotations, we refer to the extensions of FineGym99 and 288 as FineGym101 and FineGym290, respectively. Notably, we also report results on the original FineGym99 and 288 in the supplemental. Importantly, we do not use the element-level (i.e., action phase) labels during training.

Baselines
We compare our approach to other weakly supervised [15,40,8] and self-supervised [3,32] methods that entail temporal understanding in their definition. A detailed description of the baselines is provided in the supplemental.

Ablation study
We first present an ablation study that validates the contribution of each component of our loss. For this purpose, we evaluate fine-grained action recognition performance on FineGym101. Following previous work [15], we use a Support Vector Machine (SVM) classifier [12] on top of the learned embeddings to report framewise fine-grained classification accuracy. Notably, the classifier is trained on the extracted embeddings with no additional fine-tuning of the network. The results in Table 1 show the pivotal role of our contrastive cost. In fact, turning off the contrastive component of our cost, (8), and simply relying on the cosine distance always leads to no improvement in learning from the onset of training. Also, these results show the advantage of adding our global cycle consistency, which further validates the correspondences. Finally, we also compare the performance of our smoothMin definition vs. the more widely used min γ . The superiority of the adopted smooth definition supports the laid out arguments in Section 3.2.

Fine-grained action recognition
We now compare our fine-grained action recognition performance to our baselines using the FineGym dataset and consider two training settings for the backbone framewise encoder. (i) scratch: the backbone ResNet50 [24] is trained from scratch with our proposed loss, (ii) only-bn: we fine-tune batch norm layers of ResNet50 from a model pre-trained on ImageNet [37]. The embedder is otherwise trained from scratch in both cases. A third setting where all layers are fine-tuned is presented in the supplemental. The results summarized in Table 2 (and the supplemental) speak decisively in favour of our method, where we outperform all other weakly and self-supervised methods with sizable margins. The gap is especially striking in the case of SpeedNet. This poor performance can largely be attributed to the fact that the task optimized in SpeedNet does not require detailed framewise understanding. On the other hand, the closest approach to ours is TCC as it also relies on pairwise local matchings between videos of the same class; however, the matchings are realized independently and thus ignores informative long-term sequence structure. This is in contrast to SaL which uses frames from the same videos to solve tasks requiring temporal understanding. Importantly, the superiority of our results compared to TCC demonstrates that the global nature of our loss makes the learned embeddings more robust to the presence of repeated sub-actions as is the case for most of the FineGym videos. Interestingly, the results obtained with D 3 TW* highlight the limits of the downstream discriminative loss, which requires an explicit construction of positive and negative examples for training. Notably, while our method outperforms all alternatives under both training settings, the best overall results are obtained under the only-bn setting and it is therefore used for all other experiments reported in this paper.
We also considered classification results of each event separately where we also outperformed all alternatives. Importantly, visualizations of the learned features suggests that the proposed loss learns to adapt and identify the most reliable cues to learn the alignments. Please see supplemental for detailed results, discussions, and visualizations.

Few-shot fine-grained action recognition
An advantage of our proposed weakly supervised method is that it does not rely on framewise labels for training. To evaluate this advantage, we also report few-shot classification results. In this case, the entire training set is used to learn the embeddings, but only a few videos per class are used to train the classifier. In particular, we use FineGym101 for this experiment with an increasing num-

Method
Kendall's Tau Phase classification TCN [40] 65 ber of videos per class to train the classifier, starting from the 1-shot setting all the way to using the entire dataset. The plot in Fig. 4 further confirms the superiority of our proposed method. As can be seen, our method outperforms all others across the range of number of labeled training videos used, with an especially strong performance even under the challenging 1-shot setting.

Video synchronization
To evaluate the quality of the synchronization (i.e., alignment) between two videos we use Kendall's Tau metric, which does not require framewise labels. However, this metric assumes little to no repetitions in the aligned videos. We therefore follow previous work [15] and report results only on the 13 classes without repetition in PennAction (i.e., strumming guitar and jumping rope are not included). Importantly, while previously reported results [15] were obtained by training a different network for each class of Pen-nAction, we consider the more challenging setting of learning a joint representation over all classes (i.e., as done in all previous experiments with FineGym). For a fair comparison, all baselines are also trained over all classes of Penn-Action rather than one network per class. Table 3 summarizes our video alignment results, where we once again achieve state-of-the-art performance compared to the baselines. Importantly, our performance is also superior to previously reported results under the multinetwork training setting [15]. We report results under this setting in the supplemental.
In addition to the quality of synchronization, we also re-  Table 3 state-of-the-art results on action phase classification. Notably, the action phase labels used here are provided by the original authors of the PennAction dataset [58] as the labels used previously [15] were not made publically available.

Downstream applications
3D pose reconstruction. As demonstrated by our evaluations, our proposed weakly-supervised method is capable of temporally aligning videos of similar actions even while they are captured in different environments, at different execution rates, and from different viewpoints. As a result, we can align poses of the same action from different viewpoints by forcing different videos to play synchronously. Examples of this video synchronization are provided in the supplemental. Importantly, the synchronization of videos taken at various viewpoints can serve as the basis for 3D pose reconstruction. To demonstrate this ability, we use videos from PennAction and their corresponding ground truth 2D keypoint labels. In particular, given a random query video from a given class in PennAction, we first start by aligning the remaining videos (from the same class) to it. Given these aligned frames and their corresponding 2D keypoints we use the Tomasi-Kanade factorization algorithm [46], followed by bundle adjustment [22], to compute a temporally aligned 3D model of the action performed. Fig. 5 presents an example of our 3D pose reconstructions; additional results are available in the supplemental material. Audio-visual alignment. Finally, we demonstrate that our alignment based representation learning method can be applied to other types of sequences by applying it on separate audio and visual inputs. Given that any audio-visual pair is by default aligned and to avoid learning trivial solutions, we sample audio segments and video frames differently to make the task of learning the alignment harder on the net- Figure 6. Sample fine-grained audio/visual retrieval using nearest neighbor matches in embedding space, showing that the audio of the explosion can be used to retrieve corresponding visual event.
works and consequently learn strong audio and visual embeddings. In particular, while the audio signal is uniformly sampled into consecutive one second long segments, video frames are on the other hand randomly sampled along the temporal dimension. This sampling strategy encourages our model to learn different alignment paths due to the randomness in the video frame selection. For visual features, the same backbone and embedding network described in Sec. 3.5 is used to encode video frames, while we use VGGish [18] to encode audio signals. In particular, each one second long audio segment is first converted into a log mel spectogram and used as an input for the VGGish network.
Training is otherwise performed as described in Sec. 3.5.
For this sample application, we use the firing cannon class from the VGGSound dataset [9] for training and testing. This class is selected as it is strongly visually indicated with an easily identifiable salient auditory signal (i.e., the explosion sound emitted upon firing of a cannon).
To demonstrate the quality of the learned audio and visual embeddings, we evaluate them on the task of finegrained audio/visual retrieval. In particular, given a one second long audio signal corresponding to the moment of firing a cannon, we extract its corresponding audio embedding from the VGGish network trained using our approach. This embedding is then used to query corresponding visual embeddings from the vision network. The top-5 nearest frame embeddings from all videos in the test set are extracted. Sample correspondences, shown in Fig. 6, clearly depict that the corresponding visual embeddings also capture the moment of the firing visually. More details and sample results are presented in the supplemental.

Conclusion
In summary, this work introduced a novel weakly supervised method for representation learning relying on sequence alignment as a supervisory signal and taking a probabilistic view in tackling this problem. Because the latent supervisory signal entails detailed temporal understanding, we judge the effectiveness of our learned representation on tasks requiring fine-grained temporal distinctions and show that we establish a new state of the art. In addition, we present two applications of our temporal alignment framework, thereby opening up new avenues for future investigations grounded on the proposed approach.

A. Supplementary material
Our supplementary material is organized as follows: Sec. B derives several properties of the two smooth minimum approximations introduced in the main manuscript. Sec. C provides details of our evaluation baselines. Sec. D further explains our re-organization of FineGym to satisfy our training requirements. For completeness, Secs. E and F provide additional fine-grained action recognition results. Similarly, Sec. G provides additional alignment results on PennAction. Finally, Sec. H probes our learned representations via visualizations to ascertain what information is captured. A supplemental video highlighting the different applications of the proposed loss is also provided.

B. Smooth Minimum Properties
We derive several properties of the two smooth minimum approximations, smoothMin and min γ . For more general applicability, we consider any vector-valued input, a ∈ R N . smoothMin properties. The smoothMin function is defined by where γ denotes a temperature hyper-parameter. Define a 0 to be the minimum coefficient in a, that is, a 0 = min(a).
Note that from (15) it follows that d(a; γ) ≥ 0, so we have smoothMin(a; γ) ≥ min(a), i.e., it provides an upper bound on the true minimum. The maximum possible value of the penalty d(a; γ) = γsmoothMin(a − a 0 )/γ; 1) is also of interest. Given (17) it is sufficient to maximize smoothMin(r; 1) subject to r 1 = 0 and r i ≥ 0 for i ∈ {2, . . . N }. By setting ∂ ∂ri smoothMin(r; 1) = 0 and simplifying, we find 1 − r i + smoothMin(r; 1) = 0 (18) for i ≥ 2. This implies all the r i 's for i > 1 are equal at the maximum. Using (15) in (18), setting r 1 = 0 and r i = x for i > 1, and simplifying, we find x must be the solution of For N ≥ 1, (19) has a unique solution x(N ) ≥ 1. This implies that the penalty function (17) only has maxima, a, for which there is exactly one distinct minimum value a j = min(a), and all the other values are in an (N − 1)-way tie for second largest at a i = a j + xγ for i = j, and where x is as in (19).
To compute the value of this maximum, we substitute the resulting vector r = 0 x(N ) . . . x(N ) into smoothMin(r; 1) and, by simplifying, we find where we used (19) in the second line above.
For concrete examples, we find x(2) ≈ 1.2785, x(3) ≈ 1.4631 for N = 2 and 3, respectively. Also, for N ≥ 4 it follows from (19) that x(N ) < log(N + 1) (since the left hand side less minus the right is negative at x = 0, positive for x = log(N + 1), and the derivative of this difference with respect to x is positive). Therefore, (20) implies that the maximum smoothness penalty when N = 2 is roughly 0.2785γ, and 0.4631γ for N = 3, which are in agreement with the plot in Fig. 3 of the main manuscript. Moreover, for large N the maximum penalty is O(γ log(N )). min γ properties. Similarly we examine min γ , which is defined by min γ (a) = min{a i }, For a 0 = min(a) and γ > 0 we have Recall that a 0 = min(a) so a i = a 0 for at least one i. Given that all the terms in the sum N i=1 e −(ai−a0)/γ are non-negative, and one of them is 1, we conclude that N i=1 e −(ai−a0)/γ > 1. Therefore, we have min γ (a) < min(a), i.e., it provides an lower bound on the true minimum.
Moreover, the maximum sum (22) occurs when a i = a 0 for all i (i.e., there is an N -way tie for the minimum). In this case Finally, we see that the smoothness penalty when min γ is used is given by Therefore, we have shown that d γ (a) is always negative and achieves its most negative value of −γ log(N ) if and only if the input vector a represents an N -way tie, i.e., a = a 0 . . . a 0 .

C. Baselines
We compare our approach to other weakly supervised [15,40,8] and self-supervised [3,32] methods that entail temporal understanding in their definition. SpeedNet [3] learns a video representation by learning to distinguish between videos played at different speeds. Time Contrastive Networks (TCN) [40] makes finegrained temporal distinctions by optimizing a contrastive loss that encourages embeddings of an anchor image and an image taken simultaneously from a different camera viewpoint to be similar, while the embedding of an image taken from the same sequence of the anchor but at a different time instant to be distant from that of the anchor. Shuffle and Learn (SaL) [32] learns to predict whether a triplet of frames are in the correct order or shuffled. Discriminative Differentiable Time Warping (D 3 TW) [8] was originally introduced to learn alignments between video frames and action labels. Its definition includes a discriminative component requiring explicit definitions of positive and negative pairs. A pair is considered positive if all action labels are used in the alignment table, whereas negative pairs are constructed by dropping some of the action labels. We adapt this loss to our video pair alignment setting and define D 3 TW* as follows: a positive pair is constructed by sampling frames from the entire duration of both video-1 and video-2 in any pair, whereas we randomly drop portions of video-2 to constitute a negative pair, thereby mimicking the dropping of action labels as proposed in the original paper. Temporal Cycle Consistency (TCC) [15] learns finegrained temporal correspondences between individual video frames by imposing a soft version of cycle consistency on the individual matches.

D. FineGym re-organization
As mentioned in the main mansucript, each video in Fin-eGym is annotated according to a three-level hierarchy denoting the event being performed in the video, the different sets involved in performing the event, and the framewise elements (i.e., action phases) involved in each set. To perform any event-level action, a gymnast may perform the different sets in any order. To train our embedding network using our alignment-based method, we re-organize the Fin-eGym dataset such that all sets belonging to the same event appear in the same order in any given video. For example, given floor exercise events, gymnasts can perform four different sets of exercises in any order, we re-organize the clips in each video according to a selected prototype order, as shown Figure 7. These organized event-level videos are used during training and testing.

E. Fine-grained action recognition
In Sec. 4.4 of the main manuscript, we compared our fine-grained action recognition performance to our baselines using FineGym with two training settings for the backbone framewise encoder. Here, we provide our complete comparison. In addition, to training from scratch (trainall) and fine-tuning the batch norm layers (only-bn), we include a third experiment consisting of fine-tuning all layers of a ResNet50 model pre-trained on ImageNet (train-all).
The full results of all experiments are summarized in Table 4. Consistent with our conclusions in Sec. 4.4, we outperform all the weakly and self-supervised baseline methods by significant margins under all training settings with the best results obtained under the only-bn setting. For completeness, we also provide results of training the SVM classifier for framewise fine-grained action recognition on the original FineGym99 and 288 short clips. The results of this experiment, summarized in Table 5, once again demonstrate the superiority of the proposed approach even under this more challenging setting. Notably, comparison between our method and those reported in [41] is not direct for two main reasons. First, we are targeting framewise accuracy, whereas [41] focuses on clip-level accuracy. Second, we are the first to report weakly supervised results on FineGym.

F. Details of fine-grained action recognition
To further investigate the utility of the learned embeddings, we also consider classification results of each event separately. In particular, using the embeddings learned on the entire FineGym101 dataset, we train a separate SVM classifier for each event. The results summarized in Table  6, further confirm the superiority of our approach. These results also show that classifying the sub-actions in the floor exercise (FX) event is the most challenging for all methods. Careful examination of videos in this class revealed wide variations in the way gymnasts perform each sub-action  Table 6. Detailed evaluation of fine-grained action recognition performance by looking at elements within each event separately.

G. Video synchronization
In the main manuscript, we provide results of video synchronization under the challenging setting of training a single network for all classes in PennAction, whereas [15] trained a different network for each class. For completeness, we provide additional results here where we also trained a network per class using our loss on PennAction to directly compare with [15]. The results summarized in Table 7 speak decisively in favor of our approach where we outperform all approaches with a sizeable margin.

Method
Kendall's Tau TCN [40] 73.28 SaL [32] 63.36 TCC [15] 73.53 Ours 78.29 Table 7. Video alignment results using Kendall's Tau metric on PennAction. Consistent with previous work, these results were obtained by training a separate network for each class separately.

H. Visualizing learned features
To investigate what our learned representation captures, we adapt the Class Activation Map (CAM) method [59] to visualize the learned features. In particular, we extract feature maps from the last convolutional layer of our embedding network and simply average them along the channel dimension. The resulting activation maps are then normalized between 0 and 1 framewise, upsampled to match the input dimensions, and superimposed on the input video frames. For PennAction, the heatmaps in Figure 8 (a) shows that our embeddings are selective to body parts most involved in performing an action. This can explained by the fact that videos in PennAction are carefully curated with relatively clean, similar actions with no repetitions. On the other hand, we can see from Figure 8 (b), that our embeddings are tuned to human contact with surfaces to learn alignments in FineGym. This is an especially desired behaviour as while there can be significant variations in the way gymnasts perform different phases of an action, they generally share some commonality in the manner that they make contact with surfaces. More generally, these visualizations suggest that the proposed loss learns to adapt and identify the most reliable cues to learn the alignments. Additional activation visualizations are provided in the supplemental video.

I. Downstream applications
Please see supplemental video 1 for various downstream application results.


extfieldsofstudy:
Computer Science
Engineering
Medicine

text:
Distributed Learning and Inference with Compressed Images

Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time, or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task.

I. INTRODUCTION
M ODERN intelligent devices such as smartphones, autonomous vehicles and robots are equipped with highquality cameras and powerful deep neural networks that enable advanced on-board visual analysis and understanding. These large models are trained with a large amount of data and require powerful hardware resources (e.g. GPUs). These models also require days or even weeks to train, which is not possible in resource-limited devices. Thus, training is often performed in a centralized server, which also allows using data captured by multiple devices to train better models (e.g. a fleet of autonomous cars). In this case, training and test take place in two physically separated locations, i.e. server and device, respectively. In other cases, such as in mobile cloud computing, the data is captured by the device, while the inference takes place in a server.
One important requirement in these scenarios is that, at some point, the visual data needs to be transmitted from the device to the server. Fig. 1a shows an archetypal scenario of autonomous driving, where each vehicle of the fleet captures and encodes data and transmits it to the server. The server decodes the data and uses it for training the analysis models. The trained models are then deployed to autonomous vehicles, where they perform inference. Captured data often requires to be annotated by humans in order to train supervised models, which adds to the reasons to process the data in a server.
The captured data can be stored on-board in a storage device and physically delivered to the server, or directly transmitted through a communication channel. In either case, storage space or channel capacity are constraints that condition the amount of collected samples in practice, and effective collection requires data compression to exploit the limited storage and communication resources efficiently.
The amount of data captured (possibly from multiple cameras) can be enormous, requiring high compression rates with lossy compression. However, this entails a certain degradation in the images, which depends on the bitstream rate (lower the rate, higher the degradation). In this paper, we study the impact of such degradation on the performance of the downstream analysis task. At times, the degradation affects only one of the training and test data. For instance, in Fig 1a, training data is degraded while test data (on-board) can be accessed without degradation.
When training data is compressed and test is not (or vice versa), a first effect we observe is covariate shift (i.e. train and test data are sampled from different distributions). For instance, the first column of Fig. 1b represents the original captured images, while the second represents the compressed images (i.e. reconstructed 1 ). A clear difference in terms of lack of details and blurred textures is observed, which causes covariate shift (e.g. original for test, compressed for training in the example of Fig 1a). A possible solution to this problem is compressing both training and test data at the same rate. For the autonomous driving scenario of Fig. 1a, this would mean deploying an image compressor in the car (including encoder and decoder) and performing inference on the reconstructed images. While this approach alleviates the covariate shift, it is not always effective and also increases the computational cost in the on-board system. The degradation caused by lossy compression not only induces covariate shift, but can also harm the performance of the downstream task through the means of semantic information loss. Here semantic information refers to the information that is relevant to solve a particular downstream task and it can be lost during the process of compression. Semantic information is task-dependent and its loss is typically irreversible. For example, the actual plate number WAF BA 747 in the second column of Fig. 1b is lost in the process of compression, and cannot be recovered. However, if the task is car detection, the actual plate number is not necessarily relevant semantic information.
In this paper, we study the effect of compression on downstream analysis tasks (focusing on semantic segmentation) under different configurations, which in turn can be related to real scenarios. We observe that both covariate shift and semantic information loss indeed result in a performance drop (see Fig. 1c) compared to training and test with original images (configuration OO). The performance depends on the compression rate and the particular training/test configuration. For instance, in the configuration of the autonomous driving scenario of Fig. 1a compressing the test data prior to inference (we refer to this approach as compression before inference, and corresponds to the training/test configuration compressedcompressed, or CC for short) degrades the performance more than not using the original data (configuration CO), showing that it is preferable to keep the test data more semantically informative than correcting the covariate shift.
The previous result also motivates us to explore whether there exists a solution that improves over the baseline CO configuration. As a result, we propose dataset restoration, an effective approach based on image restoration using generative adversarial networks (GANs) [1]. Dataset restoration is applied to the images in the training set without modifying the test images, effectively alleviating the covariate shift while keeping the test data semantically informative. In this case, we show that the configuration restored-original (RO) does improve performance over the baseline (see Fig. 1c). An additional advantage is that there is no computational cost penalty nor additional hardware or software requirements in the deployed on-board system (in contrast to compressing the test data). Note also that our approach is generic and independent of the particular compression (deep or conventional) used to compress the images.
Adversarial restoration decreases the covariate shift by hallucinating texture patterns that resemble those lost during compression while removing compression artifacts, both of which contribute to the covariate shift. The distribution of restored images is closer to the distribution of original images and thus the covariate shift is lower. Fig. 1b shows an example where the trees have lost their texture and appear essentially as blurred green areas. A segmentator trained with these images will expect trees to have this appearance, but during test they appear with the original texture and details of leaves and branches, which leads to poor performance. The restored image has textures that resemble real trees and contains less compression artifacts, which makes its distribution closer to that of the actual test images, contributing to a significant improvement in downstream performance (see Fig 1c). Note that adversarial restoration cannot recover certain semantic information. This example also illustrates the effect on semantic information. The license plate appears completely blurred due to compression. Note that adversarial restoration can recover the texture of digits (or even hallucinate random digits), which can be useful to improve car segmentation, but the original plate number is lost (i.e. semantic information), which makes it impossible to perform license plate recognition at that compression rate.
In summary, our contributions are as follows: • Systematic analysis of training/test configurations with compression and relation of downstream performance with rate, semantic information loss and covariate shift.
• Dataset restoration, a principled method based on our theoretical analysis, to improve downstream performance in on-board analysis scenarios. This method is task-agnostic and can be used alongside multiple image compression methods. It also does not increase the inference time and memory requirements of the downstream model.

A. Lossy compression
A fundamental problem in digital communication is the transmission of data as binary streams (bitstreams) under limited capacity channels [2], [3], a problem addressed by data compression. Often, practical compression ratios are achievable only with lossy compression, i.e. a certain loss with respect to the original data is tolerated. Traditional lossy compression algorithms for images typically use a DCT or a wavelet transform to transform the image into a compact representation, which is simplified further to achieve the desired bitrate. Examples of lossy image compression algorithms are JPEG [4], JPEG 2000 [5], [6], and BPG [7]. BPG is the current state-of-the-art and is based on tools from the HEVC video coding standard [8].
Recently, deep image compression [9]- [13] has emerged as a powerful alternative to the traditional algorithms. These methods also use a transformation based approach like the traditional methods, but use deep neural networks to parameterize the transformation [9]. The parameters of the networks are learned by optimizing for a particular rate-distortion tradeoff on a chosen dataset. Mean Scale Hyperprior (MSH) [13], a deep image compression method based on variational autoencoders and BPG are used as representative methods of deep learning based and traditional image compression respectively.

B. Visual degradation and deep learning
A loss in the quality of images can occur through many factors including blur, noise, downsampling and compression. Researchers have reported a drop in task performance of convolutional neural networks (CNN) models when such degradations are present in the test images [14]- [16]. Further, numerous methods have been proposed to make these CNN models robust to degradations [15], [17], [18]. These approaches include forcing adversarial robustness during training [15], modifying and retraining the network [17], and using an additional network altogether [18].
While the aforementioned works target robustness across degradations, there have been studies focusing exclusively on compression as well. These include [19] (on the deep compression method [12]), [20] (JPEG) and [21] (both deep [22] and JPEG). Unlike the previous methods, these works use the compressed images (in some form) for training the deep models and thus obtain a better performance on compressed images. Moreover, [19] and [20] encode the images using the compressors and the deep networks are trained to predict the task output using the encoded representation directly, resulting in faster inference.
A compressed image can be processed using a restoration method before using it for inference to improve its performance; although our analysis reveals that this is a sub-optimal approach. Galteri et al. [39] propose a GAN-based restoration network to correct JPEG compression artifacts. They also evaluate different restoration algorithms on the basis of the performance of restored images on a trained object detection network. They show that their GAN-based algorithm performs better than other methods compared in the paper. Our analysis provides an explanation for this observation.

D. Domain adaptation.
Domain adaptation [42] is a problem motivated by the lack of sufficient annotations. Typically, domain adaptation methods leverage the abundant annotated data available from a different yet related domain (called as the source domain) to improve performance on the domain of interest (target domain), where there is a lack of annotated data. Examples of source and target domains include synthetic images vs real images, images in the wild vs images on a webpage, etc. Domain adaptation methods can be divided into unsupervised [43], [44], semi-supervised [45] and supervised [46]- [50] depending on the quantity of available data (and annotations) in the target domain. Approaches for domain adaptation can be categorized into latent feature alignment using autoencoders [46], [47], adversarial latent feature alignment [48]- [51] and pixel-level adversarial alignment [44], [50].
The scenario when only compressed images are available at training time, with original images available at test time is related to domain adaptation. Dataset restoration, our proposed method for this scenario, corrects the covariate shift and domain adaptation algorithms account for domain shift in some form. Probably, the closest domain adaptation method to dataset restoration is a unsupervised method that addresses alignment only at pixel-level [44]. However, an important distinction is that domain adaptation tackles problem arising due to lack of annotations for the images in the target domain, while for us the concern lies in the non-availability of images themselves. Thus, we study the effectiveness of dataset restoration, using a varying number of original images for training the restoration model.

A. Problem definition
We are concerned with downstream understanding tasks where we want to infer from an input image x ∼ p X (x) the corresponding semantic information y ∼ p Y |X (y). In the rest of the paper, we will assume that y is a semantic segmentation map, but our approach can be applied to other semantic inference tasks, such as image classification or object detection. The objective is to find a parametric mapping φ : x → y by supervised learning from a training dataset , where each image x (i) has a corresponding ground truth annotation y (i) . The mapping is typically implemented as a deep neural network. The performance of the resulting model is evaluated on a test set X ts = x . Under conventional machine learning assumptions, X tr ∼ p X (x) and X ts ∼ p X (x), i.e. both training and test sets are sampled from the same underlying distribution p X .
In our setting, we consider that X tr and/or X ts undergo a certain degradation ψ : x →x. In our case, the degradation is related with the lossy compression process necessary to transmit the image to the remote location where the actual training or inference takes place; and so we havex is the image decoder 2 and z is the compressed bitstream. The resultx is the reconstructed image, which follows a new distribution pX of degraded images, i.e.x ∼ pX (x). Note that parallels can be drawn from the arguments in this section for other image degradations such as blur, downsampling, noise, color and illumination changes, etc.
Lossy compression is characterized by the distortion D (x,x) of the reconstructed image and the rate R (z) of the compressed bitstream. The encoder and decoder are designed to operate around a particular rate-distortion (R-D) tradeoff λ, either by expert crafting in conventional image compression, or by directly optimizing parameters of a deep neural network.

B. Covariate shift
The covariate shift problem precisely occurs when the underlying distributions of training and test data differ, i.e. X tr ∼ p Xtr and X ts ∼ p Xts with p Xtr = p Xts . This leads to sub-optimal performance because the model is evaluated on a data distribution different from the one it was optimized for. While covariate shift is often found in machine learning (e.g. training with synthetic data and evaluating on real), in our case this problem is a consequence of lossy compression and it increases severely as the rate decreases. The drop in performance is related to the degree of covariate shift, which could be seen as the divergence between distributions d (p Xtr , p Xts ). In the conventional machine learning setting without compression there is no covariate shift, since X tr ∼ p X and X ts ∼ p X , nor when both training and test set are compressed with the same method and at the same rate, since X tr ∼ pX and X ts ∼ pX . 2 We only consider lossy compression, since in lossless compressionx = x.
However, covariate shift exists in the other two configurations, namely CO and OC (see Table I).
The degradation due to lossy compression can be observed clearly in Fig. 1b, when comparing the original captured image and the image after compression. This also gives an idea of the difference between the original domain and the domain induced by compression. It has images with lesser details which also suffer from blurring and coding artifacts. More examples are shown in Fig. 2 for the two compression methods (MSH and BPG), with the images compressed at a similar rate. It can be seen that degradations are consistent yet with some differences (e.g. blocky artifacts for BPG, more blurred in MSH).

C. Semantic information loss
Covariate shift explains how compression impacts the downstream task when data at training and test time are compressed unequally. Another factor that impacts task performance arises from compression resulting in semantic information loss. By the semantic information present in an image we refer only to what is relevant to the downstream task. Thus, by definition, semantic information loss is task dependent. Continuing with our example from the introduction (Fig. 1b), the letters in the license plate of the car plays little to no role in establishing the presence of a car in the image. Thus, the exact letters are not relevant semantic information for the task of car detection. However, if the task is license plate recognition, the letters are an integral part of semantic information. Compression causes semantic information loss as it makes the compressed image devoid of some semantic attributes present in the original image. The loss of letters on the plate in the compressed image ( Fig. 1b) is evidence of semantic information loss (when the task is license plate detection).
Further evidence of semantic information loss can be found in Fig. 2, since the degradation often removes details and textures, blends small objects together via blur and lack of contrast, and introduces confusing artifacts, preventing us from recognizing small objects at all (e.g. individual pedestrians), and making larger objects more difficult to recognize due to the loss of discriminative details and textures (e.g. tree leaves). Only in retrospective, after observing the original undistorted crop, we can infer the small objects in the distorted image. Similarly, a semantic segmentation model will struggle to recognize them, or directly fail when the semantic information has disappeared completely (e.g. license plate number).
Let Y be a random variable that represents the semantic information in the original image, X. For instance, if the task is semantic segmentation, Y would take values from the set of semantic maps of images. Mathematically, we formulate semantic information loss, S, in the compressed images, X, using mutual information, I, as follows: S Y (X,X) = I(X, Y ) − I(X, Y ). Predictably, S Y (X,X) is non-negative, sinceX is produced from X via the map ψ and thus, we have I(X, Y ) ≥ I(X, Y ) as a consequence of the data processing inequality.   Table I). A configuration is defined by the pair (X tr , X ts ), with X i ∼ p X represented as O and X i ∼ pX represented as C. Thus, the conventional machine learning setting (i.e. without compression) corresponds to OO, and the configuration of Fig. 1a is CO, since X tr ∼ p X and X ts ∼ pX . The former does not suffer from semantic information loss nor covariate shift, while the latter does suffer from both. The CO configuration can also be generalized to other scenarios involving on-board analysis 3 where data capture and inference takes place in the device and the training in a server (e.g. autonomous cars, unmanned aerial vehicles and other robotic devices).
The configuration OC involves training performed in the server with the original images, while the device is resourcelimited (e.g. a smartphone) but can send the image to the server, and the result of the analysis (e.g. predicted class, bounding box, segmentation map) is used in the server side or sent back to the device. In this case, the test images are compressed, which implies semantic information loss, and also covariate shift. Fig. 3 illustrates the paradigmatic scenario 3 Often called on-board perception, but we prefer on-board analysis to avoid confusion later. of (mobile) cloud computing [52]- [54]. Another example of OC configuration is distributed automotive perception [21], where the sensor module compresses the captured image and transmits it through the automotive bus system to the perception module where the downstream tasks are performed.
Similarly, the configuration CC appears in the previous scenario when training images are also compressed, and at the same rate as test images. In this case, both training and test images suffer from semantic information loss, but there is no covariate shift since both are sampled from the same pX .
Compression before training. We can remove the covariate shift from a configuration OC by transforming it into CC. This can be achieved by compressing the training data at the same rate and we refer to this adaptation approach as compression before training (see Fig. 4a). However, through this process, semantic information loss is additionally introduced to the training set. Generally, the presence or absence of semantic information loss in the test images is a major factor, while it is not always the case with the training images. While some class information may be lost in a particular training image, its presence in other training images can compensate for it. As such we expect the model with configuration CC to outperform configuration OC.
Compression before inference. Similarly, we can also transform a configuration CO into CC by compressing the test images. We refer to this process as compression before inference (see Fig. 4b). While this process allows us to correct the covariate shift due to compression, it also introduces semantic information loss at test time. The introduction of semantic information loss in the test is critical and can cause the performance of configuration CC to be even worse than the configuration CO at times (as shown in Fig. 1c). Moreover, compression before inference requires installing a full compression encoder and decoder module on-board prior to the downstream task, resulting in a significant computational penalty in the deployed system.

A. Proposed approach
Motivated by the two limitations mentioned above, we propose dataset restoration as an alternative approach that alleviates covariate shift without inducing semantic information loss in the test data (in contrast to compression before inference). The key idea is to adapt the training dataset using adversarial image restoration, and use the adapted dataset as actual training data for the downstream task (see Fig. 4c). In this way, the on-board analysis module can exploit all the information available in the captured image. Another important advantage is that adaptation takes place only in the server, and the resulting model can be readily and seamlessly deployed in the car with the same hardware, therefore without requiring to install any additional hardware nor increasing the inference cost.
We now recall that a great deal of degradation is related to the loss of texture in the decoded image and the appearance of compression artifacts (these two factors are clearly apparent in Figs. 1b and 2). Our goal was to find an appropriate image restoration technique that could learn from a given set of examples and provide us a way to remove the artifacts and recover texture in the images.
Our restoration module is based on adversarial image restoration, where a generative adversarial network (GAN) [1] conditioned on the degraded image is employed to improve the image quality. A GAN is based on two networks competing in an adversarial setting. The generator takes the input image and outputs the restored image. The discriminator observes real and restored images and it is optimized to classify between real and restored images. The generator, in contrast is optimized to fool the discriminator, and indirectly improves the quality of the restored images. Through the process, the generator learns to remove compression artifacts and replace unrealistic textures by realistic ones that could be used by the discriminator to identify the restored images. The architecture of GAN is based on the one proposed in [55] (for image-toimage translation [56]), which has a generator and multiple discriminators (see Appendix B for details).
During the process of dataset restoration, we use our trained generator to restore individually every image in the training dataset for the downstream task. Examples of some of the restored images can be found in Figs. 1b and 2. While not being able to restore lost semantic information (e.g. the same individual pedestrians), the restored images look sharper, with less artifacts and blurred regions are enhanced with hallucinated textures that resemble the real images. As such, the shift with respect to the distribution of original images, on which the trained model will be evaluated, is reduced. Table I includes two new configurations OR and RO, where R refers to restored images.

B. Adversarial restoration, covariate shift and perceptual index
There is an interesting relation between the perceptual index described in [57] and the covariate shift, which explains why adversarial image restoration is the appropriate approach (compared to non-adversarial). In the CO configuration, the training images X tr are compressed and therefore follow pX , while the test images X ts follow p X . Thus, the covariate shift is equal to d(p X , pX ), where d denotes a probabilistic divergence, like the Kullback-Leibler divergence or the Jenson-Shannon divergence [3]. On the other hand, the perceptual index in [57] is defined as the divergence between real images p X and compressed (and decoded) images pX (note that there is not training/test distinction). Therefore, in the case of CO configuration, the covariate shift corresponds to the perceptual index of the test set (in both cases, the lower the better). An important conclusion from [57] is that perception and distortion are at odds with each other, and that there exists a limit beyond which perception and distortion cannot be reduced simultaneously (see Fig. 5). Thus, the effect of dataset restoration (i.e. moving from CO to RO) is to lower the covariate shift (and perceptual index) at the expense of increasing distortion, provided we are close to the perceptiondistortion limit.
We are ultimately interested in the implications on the performance of the downstream task, semantic segmentation in particular. Our analysis in the previous section reveals that the task performance is greatly dependent on covariate Attainable at r 1 and r 2 Fig. 5. Illustration of the perception-distortion tradeoff [57], [58]. Restoration process shifts the compressed images from a point of low distortion and low perceptual quality to a point of higher distortion but higher perceptual quality in the perception-distortion plane.
shift and semantic information loss. Reducing the perception index is therefore more crucial than training with images of low distortion, as a lower perception index corresponds to a lower covariate shift. Hence, we use adversarial restoration and decrease the perceptual index at the cost of increased distortion. Are all image restoration approaches helpful? We argue that only adversarial image restoration are suitable, since they explicitly minimize the perceptual index through the discriminator and consequently the covariate shift with respect to the captured images. In contrast, non-adversarial image restoration methods do not necessarily reduce the perceptual index. Typically, these methods try to further decrease the distortion and this can be counter-productive as perception and distortion are at odds near the limit.

A. Experimental settings
Datasets. We evaluate our methods on three datasets: Cityscapes [59] is a popular dataset in autonomous driving, and contains 5000 street images (2975/500/1525 for training/validation/test sets) of which training and validation have pixel-level segmentation maps annotated with 19 different concepts, including objects and "stuff". We use the annotated sets to train (training set) and evaluate semantic segmentation (validation set). It also contains another 20000 images with coarse annotation. We ignore these annotations and use a subset of 2000 images to train the deep image compression model (i.e. MSH [10]) and the image restoration methods.
INRIA Aerial Images Dataset [60] contains aerial images of diverse urban settlements with segmentation maps with two classes (building and background). The dataset consists of aerial images from 10 cities with 36 images per city. Annotations are provided for 5 of these cities and the segmentation models were trained on 4 cities and evaluated on 1 from these. The images from the other 5 cities were used for compression and restoration. Semantic Drone Dataset [61] contains 400 high resolution images captured with an autonomous drone at an altitude of 5 to 30 meters above ground, and their corresponding annotated segmentation maps (20 classes). The 400 publicly released images were resized from a resolution of 6000x4000 to 3000x2000. The segmentation models were trained on 265 images and evaluated on 70 images while the remaining 65 images were used for the compression and restoration models. Each image was further split into 12 patches each with dimension of a 1200x800. All metrics are calculated on these patched images.
Compression methods. We use two state-of-the-art image compression methods. The Better Portable Graphics (BPG) format [7] is based on a subset of the video compression standard HEVC/H.265 [8] and is the state-of-the-art in non-deep image compression. The Mean Scale Hyperprior (MSH) [10], [13] is a state-of-the-art deep image compression method, based on an autoencoder whose parameters are learned to jointly minimize rate and distortion at a particular tradeoff λ, i.e. min R + λD. MSH models were pretrained for 600k iterations on the CLIC Professional Dataset 4 with MSE loss. Appendix A contains details of the model architecture.
Segmentation. For the downstream task we use the state-ofthe-art semantic segmentation method DeepLabv3+ [62]. The model is trained using the same procedure mentioned in the paper. We use an output stride of 16 and perform single scale evaluation.
Metrics. The quality of the inferred semantic segmentation map is evaluated using the mean intersection over union (mIoU, the higher the better). For image compression we measure rate in bits per pixel (bpp, the lower the better) and the distortion in PSNR (in dB, the higher the better).

B. Cityscapes
Rate-distortion curves. We first characterize the ratedistortion performance and the tradeoff for the two compression methods in our experiments, and the impact of the proposed restoration approach on them (see Fig. 6). The curves sweep the whole range, from low to high quality images. As expected, the distortion decreases (PSNR increases) with rate. We observe that MSH performs significantly better than BPG on Cityscapes, i.e. it produces images with lower average distortion at similar rate. Interestingly, once the images are restored, images compressed with MSH have marginally higher distortion than those compressed with BPG. Segmentation performance. We evaluate the segmentation performance under six different configurations (i.e. OO, CO, RO, CC, OC, OR). The results are shown in Fig. 7.
For the Cityscapes dataset, we observe that the model with configuration CO outperforms the model with configuration CC which shows that correcting covariate shift by compression before inference can potentially result in lowering the performance. Table II shows the performance per class. We see that the mIoU of classes representing small objects 5 is significantly lower in the configuration CC when compared to CO. Since smaller objects are relatively easier to lose by compression, the observation confirms that the introduction of semantic information loss in the test set from the process of compression before inference is responsible for the decrease in performance.
The proposed dataset restoration approach is able to improve 1.4-4.8% on the configuration CO, and we achieve close to optimal performance (77% mIoU) requiring only 0.13 bits per pixel. Thus, lossy compression can result in huge storage savings during data collection when compared to lossless image compression. For the same budget required to collect the 2975 training images with MSH at 0.13 bpp, using lossless compression (PNG in our experiments, resulting in 9 bpp) we would have collected only 42 images, which is clearly insufficient to train the segmentation network. Furthermore, for the same performance, dataset restoration effectively reduces the required budget (for example, configuration CO achieves 75.64% with 0.128 bpp, while RO approximately requires around 0.07 bpp). Amount of samples for training the restoration network. In order to learn the restoration network it is necessary to collect high quality images (uncompressed or compressed losslessly, ideally). Thus, the amount of images to train the restoration network is also an important factor. Fig. 8 shows the result for configuration CO (no restoration) and RO with a restoration network train with different amounts of training samples. It suggests that restoring images compressed with MSH can be learned with fewer images (or at least the performance saturates) than in the case of BPG.
Adversarial vs non-adversarial restoration. In order to show the connection between perceptual index and adversarial image restoration, we compute the perceptual index of the original, compressed and restored images using the blind quality assessment method HOSA [63] (as described in [57]). We use Residual Dense Networks [29] as a representative method of non-adversarial image restoration. RDN is trained  While the previous experiment shows that adversarial restoration does improve the perceptual quality of images, we are ultimately interested in the segmentation task. We use the model trained with the original images to extract features from a shallow layer. Then we extract features of original images, compressed images and restored images. Since we only use one model, all features are aligned in the channel dimension. We want to measure the level of alignment in activation, so we plot the average activation (over the validation set of Cityscapes) value for each channel of original, compressed or restored versus the original one (see Fig. 10). Obviously, the points corresponding to original images lie on the identity line, while many channels from compressed images are clearly not aligned. Adversarial restoration manages to bring back the features to the identity line, while non-adversarial restoration has little effect. This shows that adversarially restored images are not only perceptually closer to the real images, but also semantically more correlated.
Further, Table III compares the performance of different restoration methods in the configuration RO. Adversarial restoration results in a far better segmentation performance when compared to RDN. Efficiency. Table IV reports the inference times for different configurations. The time for encode-decode times for MSH and the segmentation time for DeepLabv3+ were measured on a Quadro RTX6000 GPU, while the encode-decode times for BPG was measured on a Intel Xeon(R) E5-1620 v4 CPU. Inference using dataset restoration (configuration RO) is faster than compression before inference (configuration CC) by 26% and 66%, when the compression method used is MSH and BPG respectively.

C. INRIA Aerial Images Dataset
Segmentation performance. Fig. 11 depicts the segmentation results obtained on the INRIA Aerial Images Dataset (AID) for the same six configurations mentioned in Section B. We observe that the model with configuration CO performs better than CC when the compression method used is BPG. However, the same cannot be said for MSH. We hypothesize that the smoothing artifacts caused when MSH is used destroy the discriminative features and a segmentation model capable of taking advantage of these features in the original image, cannot be learnt. This is especially critical with the AID since there are only a few features that discriminate a building from the background.
The proposed approach of RO performs consistently better than both these configurations with gains up to 3.9 % mIoU. Fig. 12 shows a portion of an image from the dataset along with the segmentation maps predicted by various models.
Interestingly, contrary to the results on the Cityscapes dataset, the performance of configuration OR does not improve over OC. This suggests that the process of restoration causes further damage (over compression) to the discriminative features used by the model trained on the original images for its prediction.

D. Semantic Drone Dataset
Segmentation performance. The segmentation results obtained on the Semantic Drones Dataset (SDD) are shown in Fig. 13. We observe that the models with configuration CC outperform CO consistently for both MSH and BPG. We attribute this result to the lack of semantic information loss in this dataset. A typical object from the SDD covers a significant portion of the image and compression does not significantly affect its recognizability. When we assume little to no semantic information loss, the effects of covariate shift are dominant and as such, the configuration CC performs better. This result shows that the performance of various configurations are dependent on the properties of the dataset.
The proposed approach of RO performs similarly to configuration CC and lies within ±1.5% mIoU of the configuration CC. The lack of significant semantic information loss in SDD affects the effectiveness of dataset restoration.

VI. CONCLUSIONS
The rapid development in sensor quality and increasing data collection rate makes lossy compression necessary to reduce transmission and storage costs. By means of dataset restoration, we enable the incorporation of lossy compression for on-board analysis, greatly mitigating the drop in performance. Dataset restoration is a principled approach, based on our analysis of the various scenarios involving learning and inference with compressed images. Our analysis framework involving covariate shift and semantic information loss can be further extended to other degradations like blur, noise, color and illumination changes, etc.

APPENDIX B ADVERSARIAL IMAGE RESTORATION
We use the FineNet from Akbari et. al [65] (adapted from [55]) with slight changes for our image restoration module.
Following the same notation in [65], the generator architecture is written as c 64 , d 128 , d 256 , d 512 , 9 × r 512 , u 256 , u 128 , u 64 , o 3 where • c k : Conv: 7 7 x k, Instance Normalization, ReLU • d k : Conv: 3 x 3 x k / ↓ 2, Instance Normalization, ReLU • r k : Conv: 3 x 3 x k, Reflection padding, Instance Normalization, ReLU • u k : Conv: 3 x 3 x k / ↑ 2, Instance Normalization, ReLU • o 3 : Conv: 7 x 7 x 3, Instance Normalization, Tanh We use two discriminators, as in [65], operating at two different scales. Akbari et al. rescale the image to half the  Let VGG denote a VGG-Net trained on the ImageNet dataset and M j denote the size of the output of the j th layer of VGG. The output of each of the 5 convolution blocks are considered for the VGG feature distillation loss, which is given by Similarly, the features of the discriminators are also distilled for stable GAN training.
The discriminators are trained using the standard GAN loss.
We use a batchsize of 1 and train the GAN for around 135k iterations. Adam optimizer with β 1 = 0.1 and β 2 = 0.9 is employed. Initially, the learning rate is set to 0.0002 and is reduced by a factor of 10 after 80k iterations.

APPENDIX C RESIDUAL DENSE NETWORK
We use the RDN architecture from [29]. We ask the reader to refer to the paper for the architecture. The following hyperparameters are used: Global layers = 16, Local layers = 6, Growth rate = 32.
We train the CAR model with the objective of maximising MS-SSIM or PSNR. The models are trained using 256x256 patches. A mini-batchsize of 1 is used and the model is trained for around 200k iterations. Adam optimizer is used with initial learning rate of 0.001 which is reduced by a factor 10 at 80k and 150k iterations.