Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.
The authors propose self-supervised learning of audio and video features, by means of a curriculum learning setup. In particular, a deep neural network is trained with a contrastive loss function; generating as large feature distances as possible when the video and audio fragment are out of sync, while generating as small distances as possible when they are in sync. The proposed self-supervised learning scheme gives good results for downstream processing, e.g., improving over pure supervised training from scratch.  In general, I think this work is highly interesting, since it may significantly reduce the need for manually labelled data sets, and hence may be of significant impact on video & audio processing algorithms.  I also think the work is original, although this is not very easy to determine exactly. For instance, the authors mention a very similar paper only in line 258, ref [34]. There may be a clear reason for this: [34] has been published on the Arxiv only in April 2018, so I guess the most likely scenario is that the authors found out about this parallel work very late. I would advise the authors in a revised version though, to mention this work higher up in the paper, perhaps moving the whole related work section there. Still, as the authors claim, the difference with [34] seems to lie in harder examples that are provided in a curriculum learning scheme and hence better results.  The authors make an important point that self-supervised learning in pre-training may lead to improved results (further approaching fully supervised learning results), when a bigger training set is given. Now this seems to be supported by results in Table 3, where a higher number of auxiliary samples leads to a higher performance. Still, in this case, the number of samples is not the only difference, it is also the type of dataset. It would be interesting to also see results for different numbers of examples from the same set.   There is a result, which I believe to be interesting, but which I think strictly speaking requires more evidence. In Table 3 the results of the "AVTS features" are mentioned. However, it would be good to compare it with another approach that directly applies an SVM to a hidden layer. How good does the SVM work if the activations are taken of a randomly initialized network? This could quantify how much the self-supervised learning improves the features for this task.  In summary, a very nice paper that in my eyes should be accepted. Some additional smaller remarks can be found below that aim to further improve the clarity.  -----  1. Line 138 : "but the objective was also difficult to optimize." Can you specify   what you mean with this? Did learning not converge in that case? 2. Line 141: you say that it led to better results in every quantitative aspec, also   the "feature quality". How was this quality determined, i.e., to what results are   you then referring? The SVM ones? 3. The main thing not clear to me for the neural network, is how the initial   convolution is done with the N stacked video frames. Is the 3D convolution over all   stacked frames together? Then again, the filter is described with 3x7x7, which would   suggest that it only goes 3 deep. It is probably described in [19], but I think it   would be good to clearly explain it also in this article. 4. Some of the results discussed in 202-207 do not seem to be in table 1. 5. Figure 3 is interesting, although it seems to mostly just take optic flow into   account. Still, it does not seem to be referred to, nor discussed in the text.  6. Autoencoders in my eyes also use self-supervised learning, as the supervised   targets are self-provided.   7. "in synch" -> "in sync" 8. "One may be argued" -> "One may argue"