Audio Visual Predictive Coding for Self Supervised Visual Representation Learning
Self supervised learning has emerged as a candidate approach to learn semantic visual features from unlabeled video data. In self supervised learning, intrinsic correspondences between data points are used to define a proxy task that forces the model to learn semantic representations. Most existing proxy tasks applied to video data exploit only either intra modal (e.g. temporal) or cross modal (e.g. audio visual) correspondences separately. In theory, jointly learning both these correspondences may result in richer visual features; but, as we show in this work, doing so is non trivial in practice. To address this problem, we introduce 'Audio Visual Permutative Predictive Coding' (AV PPC) a multi task learning framework designed to fully leverage the temporal and cross modal correspondences as natural supervision signals. In AV PPC, the model is trained to simultaneously learn multiple intra and cross modal predictive coding sub tasks. By using visual speech recognition (lip reading) as the downstream evaluation task, we show that our proposed proxy task can learn higher quality visual features than existing proxy tasks. We also show that AV PPC visual features are highly data efficient. Without further finetuning, AV PPC visual encoder achieves 80.30% spoken word classification rate on the LRW dataset, performing on par with directly supervised visual encoders that are learned from large amounts of labeled data.
2020 25th International Conference on Pattern Recognition (ICPR)