  We introduce a novel self-supervised contrastive learning method to learn
representations from unlabelled videos. Existing approaches ignore the
specifics of input distortions, e.g., by learning invariance to temporal
transformations. Instead, we argue that video representation should preserve
video dynamics and reflect temporal manipulations of the input. Therefore, we
exploit novel constraints to build representations that are equivariant to
temporal transformations and better capture video dynamics. In our method,
relative temporal transformations between augmented clips of a video are
encoded in a vector and contrasted with other transformation vectors. To
support temporal equivariance learning, we additionally propose the
self-supervised classification of two clips of a video into 1. overlapping 2.
ordered, or 3. unordered. Our experiments show that time-equivariant
representations achieve state-of-the-art results in video retrieval and action
recognition benchmarks on UCF101, HMDB51, and Diving48.

Computer Vision and Pattern Recognition