This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.
This paper presents a novel method to extract cross-modal text-visual embeddings on the HowTo 100M corpus. The core idea is to extend previous work on clip-level embeddings (e.g. the max-margin ranking loss proposed for HowTo 100M) to a transformer architecture which takes into account the entire context of a video, which should lead to better learned representations and improved performance in downstream tasks. In addition, the max-margin loss is replaced by noise contrastive estimation. 
The paper is well written and explains the main problem well, however I do have a few questions:
- I do not understand the sentence "However, for images and videos, the inputs are real-valued vectors." (Section 3.2) - Transformers are being used for speech recognition or speech translation - the input features are not the problem. The outputs are assumed to be discrete (in the original formulation)
- Why not directly compare your approach to the approach presented in (Miech, 2019c) - it would be interesting to see a direct comparison, but as far as I can tell, there is no overlap in tasks?
- What is the influence of adding punctuation to the ASR output, how good is it, and how good is the underlying ASR? Why did you not use the original text annotations provided by HowTo 100M, but run the audio through Google ASR (again?) It would be good to know how good the ASR is, and if adding in punctuation post-hoc works well, and how this influences your use with a pre-trained BERT model. My guess is that the BERT model will be happy as long as it sees a "." at the end?
- Also, would it be possible to compare the results of your work with some of the work in (Miech, 2019c) - it almost seems that your work avoids comparing your results to this previous work.