  Visual Speech Recognition (VSR) differs from the common perception tasks as
it requires deeper reasoning over the video sequence, even by human experts.
Despite the recent advances in VSR, current approaches rely on labeled data to
fully train or finetune their models predicting the target speech. This hinders
their ability to generalize well beyond the training set and leads to
performance degeneration under out-of-distribution challenging scenarios.
Unlike previous works that involve auxiliary losses or complex training
procedures and architectures, we propose a simple approach, named Lip2Vec that
is based on learning a prior model. Given a robust visual speech encoder, this
network maps the encoded latent representations of the lip sequence to their
corresponding latents from the audio pair, which are sufficiently invariant for
effective text decoding. The generated audio representation is then decoded to
text using an off-the-shelf Audio Speech Recognition (ASR) model. The proposed
model compares favorably with fully-supervised learning methods on the LRS3
dataset achieving 26 WER. Unlike SoTA approaches, our model keeps a reasonable
performance on the VoxCeleb test set. We believe that reprogramming the VSR as
an ASR task narrows the performance gap between the two and paves the way for
more flexible formulations of lip reading.

Sound;Computation and Language;Audio and Speech Processing