Training Transformer-based models demands a large amount of data, while obtaining aligned and labelled data in multimodality is rather cost-demanding, especially for audio-visual speech recognition (AVSR). Thus it makes a lot of sense to make use of unlabelled unimodal data. On the other side, although the effectiveness of large-scale self-supervised learning is well established in both audio and visual modalities, how to integrate those pre-trained models into a multimodal scenario remains underexplored. In this work, we successfully leverage unimodal self-supervised learning to promote the multimodal AVSR. In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual data into characters through a combination of CTC and seq2seq decoding. We show that both components inherited from unimodal self-supervised learning cooperate well, resulting in that the multimodal framework yields competitive results through fine-tuning. Our model is experimentally validated on both word-level and sentence-level tasks. Especially, even without an external language model, our proposed model raises the state-of-the-art performances on the widely accepted Lip Reading Sentences 2 (LRS2) dataset by a large margin, with a relative improvement of 30%. * 1 Introduction Audio-Visual Speech Recognition (AVSR) is a speech recognition task that leverages both an audio input of human voice and an aligned visual input of lip motions. It has been one of the successful application fields that involve multiple modalities  Corresponding author. in recent years. Due to the limited amount of labeled, multimodal aligned data and the difficulty of recognition from the visual inputs (i.e., lip reading), it is a challenging task to tackle. Existing AVSR models tend to use extra data to increase the performance of the system, in a form of inserting an extra supervised learning stage in the training process. For example, many existing methods rely on an extra sequence level classification to bootstrap its learning on visual features. Petridis et al. (2018); Zhang et al. (2019) train their visual front-end on LRW (Chung and Zisserman, 2016) before learning on the AVSR task. Afouras et al. (2018a,b) chunks the MV-LRS data (Chung and Zisserman, 2017) into pieces of words and pre-train the model through classification. VoxCeleb (Chung et al., 2018) are also used in Afouras et al. (2020) for the same purpose. Learning an effective visual front-end could still be notoriously hard, even with these extra supervised learning tasks. Sometimes curriculum learning is required to adapt the learned visual front-end into AVSR task (Afouras et al., 2018a). End-to-end learning of large-scale AVSR data hasn't been successful until recently (Ma et al., 2021). Although self-supervised learning could enable leveraging unlabelled or even unaligned data, it hasn't been adequately explored on this task. Shukla et al. (2020) is among the few attempts in this facet, in which it predicts lip motions from audio inputs. Their proposed learning schemes yield strong emotion recognition results but are relatively weak in speech recognition. Moreover, since in AVSR it is the lip shape and motions between frames rather than the objects in a single image that matters for recognizing speech contents, if pre-trained visual models tailored for tasks targeting at single frame images could work for AVSR remains unknown. In another scenario, self-supervised learning in unimodality has been well established as a paradigm to learn general repre-4491 sentations from unlabelled examples, such as in natural language processing (Brown et al., 2020; Devlin et al., 2019), speech recognition (Baevski et al., 2020), and computer vision (He et al., 2019; Chen et al., 2020a; Grill et al., 2020). In this work, we rely on a simple but effective approach, which is to utilize unlabelled unimodal data by using pre-trained models that are trained in single-modality through self-supervised learning. Specifically, we use Baevski et al. (2020) pre-trained on the large LibriLight (Kahn et al., 2020) dataset as our audio front-end. For visual front-end, we found that it is not as straight-forward for it to leverage pre-trained models, as we have to substitute the first convolutional layer in MoCo v2 (Chen et al., 2020b) by a 3-D convolutional layer and fine-tune it through LRW. In total, our approach doesn't require a curriculum learning stage, and the overall training time has been decreased. Experimental results show that our new front-ends significantly outperform previous ones by a big margin in both audio-only and visual-only settings, and a new state-of-the-art has been achieved in the final AVSR setting. To our best knowledge, this is the first work that successfully applies unimodal pre-trained models in the multimodal setting of AVSR. The earliest work on AVSR could be dated back to around two decades ago, when Dupont and Luet-tin (2000) showed hand-crafted visual feature improves HMM-based ASR systems. The first mod-ern AVSR system is proposed in Afouras et al. (2018a) where deep neural networks are used. The field has been rapidly developing since then. Most of the works are devoted into the architectural improvements, for example, Zhang et al. (2019) proposed temporal focal block and spatio-temporal fusion, and Lee et al. (2020) explored to use cross-modality attentions with Transformer. The other line of research focuses on a more diversified learning scheme to improve AVSR performance. Li et al. (2019) uses a cross-modal student-teacher training scheme. Paraskevopoulos et al. (2020) proposes a multi-task learning scheme by making the model to predict on both character and subword level. Self-supervised learning has also been explored in Shukla et al. (2020), where the cross-modality setting is utilized by predicting frames of videos from audio inputs. The end-to-end learning of AVSR systems are first seen in Tao and Busso (2020), albeit in a much simpler dataset than LRS2. More recent work (Ma et al., 2021) has made end-to-end learning on LRS2 possible by using a Conformer acoustic model and a hybrid CTC/attention decoder. Self-supervised learning has been chased in recent years since its ability to learn general representations of data through simple tasks that don't require labeling. Contrastive learning (Hadsell et al., 2006) has become the most impactful learning scheme in this field. In natural language processing, uni-or bi-directional language modelling (Brown et al., 2020; Devlin et al., 2019) have been used to significantly increase performances on various tasks. In audio speech processing, contrastive predictive coding (Baevski et al., 2020) has been proven to be powerful in speech recognition. In the visual domain, Earlier works create self-supervised tasks through image processing based methods, such as distortion (Gidaris et al., 2018),colorization (Zhang et al., 2016) and context prediction (Doersch et al., 2015). More recently, contrastive learning emerged as a paradigm of self-supervised learning, which results in a group of more expressive general visual representations, such as MoCo (He et al., 2019; Chen et al., 2020b), SimCLR (Chen et al., 2020a), BYOL (Grill et al., 2020), etc. 3 Architecture The overall architecture of our model is shown in Fig. 1. The audio-visual model is comprised of four components, the front-ends and back-ends for both modalities, the fusion module, and the decoders. Visual Front-end: Visual front-end serves as a component to capture the lip motion and reflect the lip position differences in its output representations. A naive way to apply pre-trained models in the visual front-end is to directly feed the RGB channels of each frame as input. However, since frames within a same clip in AVSR are largely similar in their contents while most pre-trained models in vision target at learning general representations reflecting the content of the whole image, this approach will result in similar outputs for all the frames, collapsing the informative lip position 4492 Conv3d MoCo Wav2Vec 2.0 C on c a t ena t e ModalityNorm seq2seqLoss Visual Frontend Audio Frontend Visual Backend TransformerEncoderLayer  6 Conv1d Audio Backend ModalityNorm Transformer EncoderLayer  6 Conv1d Transformer EncoderLayer  6 Conv1d Fusion Module Transformer DecoderLayer  6 Conv1d CTCLoss Figure 1: Overall architecture of our AVSR model. To overcome the aforementioned problem while still being able to utilize the pre-trained model, we truncate the first convolutional layer in MoCo v2 (Chen et al., 2020b), which is pre-trained on ImageNet (Deng et al., 2009), and replace it with a layer of 3-D convolution. The outputs of 3-D convolutional layer are intentionally made identical to the input of the first ResBlock in MoCo v2 (see Table 1), thus providing a compatible interface to transfer higher layers of MoCo v2 into this task. On the other hand, we also adopt the common practice to convert the RGB input image to gray-scale before feeding it into the model, as it prevents the model from learning chromatic aberration information. Audio Front-end: The audio front-end is rather straight-forward. We use wav2vec 2.0 (Schnei-der et al., 2019) pre-trained on Libri-Light (Kahn et al., 2020), like it is normally used for ASR tasks, both the 1-D convolutional layers and the stacked Transformer encoder layers are transferred into our audio front-end. The audio front-end takes as input raw audio wave of 16kHz, and produces one vector representation every 20ms. The audio feature dimensions are shown in Table 2. 3.2 Back-ends Since the visual frames are in 25 FPS and the wav2vec 2.0 outputs are around 49 Hz, one should note that there is 2x difference in the frequency of frame-wise visual and audio representations at the output of their front-ends. In the back-end, we use 1-D convolutional layers on the time dimension combined with Transformer encoder layers to provide single modality temporal modeling, as well as adjusting the features to have the same frequency. Visual Back-end: The incoming MoCo v2 output to the visual back-end has a feature dimension of 2048, at a frequency of 25 vectors per second. In the visual backend, we keep this frequency while reducing the feature size to 512. See Table 1. For positional encodings of the Transformer, we use fixed positional encoding in the form of sinusoidal functions. Audio Back-end: In the audio back-end, the incoming wav2vec 2.0 outputs have a feature size of 1024, at a frequency of 50 vectors per second. We downscale the frequency by setting the stride of 1-D convolutional layer to 2. The Transformer encoder layers have the identical size to that of the visual back-end, while using a separate set of parameters. Table 2 shows a clearer picture of audio frontand back-end dimensions. The odds are due to the larger receptive fields of wav2vec 2.0 1-D convolutional layers, which we circumvent by properly prefixing and suffixing the audio sequence and truncate the trailing audio vector. Thus a perfect 1:2 ratio of visual frames and audio front-end outputs are ensured. Features from both the audio and visual modalities are fused together in this section, forming vector representation of 1024 dimensions at a relatively low rate of 25 Hz. We use LayerNorm (Ba et al., 2016) separately on each of the modalities before concatenating them on the feature dimension. The LayerNorm is required since it avoids one modality overtaking the whole representation with larger variance. Similar 1-D convolutional layers and a subsequent Transformer encoder block of 6 layers take the fused representations as input, and encode them for the decoders. Following the setting of Petridis et al. (2018), there are two decoders trained simultaneously based on the same output in the fusion module. The first is a Transformer seq2seq decoder, a Transformer decoder with 6 layers is used, and we perform teacher forcing at character level by using ground truth characters as input during training. The second one is arguably a decoder since it yields character probabilities for each timestep and relies on the CTC loss in training. 4 extra 1-D convolutional layers with ReLU activation are used on top of the last Transformer encoder layer output. We also include LayerNorm between each of the layers. In this work, we use a so called hybrid CTC/atten-tion loss (Watanabe et al., 2017) for our training process. Let x = [ x 1 ,    , x T ] be the input frame sequence at the input of Transformer encoder in the fusion module and y = [ y 1 ,    , y L ] being the targets, where T and L denote the input and target lengths, respectively. The CTC loss assumes conditional independence between each output prediction and has a form of p CTC ( y | x )  T (cid:89) t =1 p ( y t | x ) (1) On the other hand, an autoregressive decoder gets rid of this assumption by directly estimating the posterior on the basis of the chain rule, which has a form of p CE ( y | x ) = L (cid:89) l =1 p ( y l | y <l , x ) (2) The overall objective function is computed as follows: L =  log p CTC ( y | x )+(1   ) log p CE ( y | x ) (3) where  controls the relative weight between CTC loss and seq2seq loss in the hybrid CTC/atten-tion mechanisms. The weight is needed not only when integrating the two losses into one training loss, but also fusing the two predictions during decoding, which we will revisit in the following subsections. For audio modality, the audio front-end is first pre-trained through self-supervised learning, which is done by wav2vec 2.0. Then the audio frontand back-end are trained through the audio-only (AO) setting, together with dedicated decoders. For the visual modality, the visual front-end is first pre-trained through self-supervised learning, then modified and trained through sequence classification at word level video clips in LRW data. After that, the visual front-end is inherited by the visual-only (VO) model, where visual back-end and dedicated decoders are used. Due to computational constraints, we pre-compute the audio and visual back-end outputs, and only learn the parameters in the fusion module and decoders part in this final stage. A detailed visualization of our training pipeline is depicted in Figure 2. 3.7 Decoding Decoding is performed using joint CTC/attention one-pass decoding (Watanabe et al., 2017) with beam search. We apply shallow fusion to incorporate CTC and seq2seq predictions:  y = arg max y   Y {  log p CTC ( y | x ) + (1   ) log p CE ( y | x ) } (4) where  Y denotes predictions set of target symbols, while  is the relative weight that tuned on validation set. In this section, we will first introduce the datasets and various settings we used in each component of our model. Then we will present results of audio-only, visual-only and audio-visual settings. We also present a breakdown of the relative contribution of every component through ablation study. We use the large-scale publicly AVSR dataset, the Lip Reading Sentences 2 (LRS2) (Chung et al., 2017) as our main testbed. During training, we also use the Lip Reading in the Wild (LRW) (Chung and Zisserman, 2016) as a word-level video classification task to pre-train our visual front-end. LRS2 consists of 224 hours of aligned audio and videos, with a total of 144K clips from BBC videos, the clips are at a length of sentence level. The training data contains over 2M word instances and a vocabulary of over 40K. The dataset is very challenging as there are large variations in head pose, lighting conditions, genres and the number of speakers. LRW is a word-level dataset, consisting of 157 hours of aligned audio and videos, totalling 489K video clips from BBC videos, each containing the utterance of a single word out of a vocabulary of 500. The videos have a fixed length of 29 frames, the target word occurring in the middle of the clip and surrounded by co-articulation. All of the videos are either frontal or near-frontal. In our experiment, we only use the visual modality from this dataset to train our visual front-end. We use character level prediction with an output size of 40, consisting of the 26 characters in the alphabet, the 10 digits, the apostrophe, and special tokens for [space] , [blank] and [EOS/SOS] . Since the transcriptions of the datasets do not contain other punctuations, we do not include them in the vocabulary. Our implementation is based on the Pytorch library (Paszke et al., 2019) and trained on four NVIDIA A100 GPUs with a total of 160GB memory for 1 week. The network is trained using the Adam optimizer (Kingma and Ba, 2015) with  1 = 0 . 9 ,  2 = 0 . 999 and (cid:15) = 10  8 and an initial learning rate of 10  4 . We use label smoothing with a weight set to 0.01, learning rate warm up and reduce on plateau scheduler. The relative weight in CTC loss and seq2seq loss  is set to 0.2. When decoding, we set  to 0.1. The samples in the pre-train set are cropped by randomly sampling a continuous range of 1 / 3 words of the whole utterances, in order to match the length of clips in the train set. Over-length samples are further truncated at 160 frames to reduce memory occupation. Preprocessing: We detected and tracked 68 facial landmarks using dlib (King, 2009) for each video. To remove differences related to face rotation and scale, the faces are aligned to a neural reference frame using a similarity transformation following Martnez et al. (2020). Interpolation and frame smoothing with a window width of 12 frames are used to deal with the frames that dlib fails to detect. Then a bounding box of 120  120 is used to crop the mouth ROIs. The cropped frames are further converted to gray-scale and normalized with respect to the overall mean and variance of the train set. Each raw audio waveform is normalized to zero mean and unit variance following Baevski et al. (2020). Data Augmentation: Following Ma et al. (2021), random cropping with a size of 112  112 and horizontal flipping with a probability of 0.5 are performed consistently across all frames of a given image sequence when training visual-only and audiovisual models. For each audio waveform, additive noise is performed in the time domain following Afouras et al. (2018a) during training audio-only 4495 Methods WER Visual-only LIBS (Zhao et al., 2020) 65.3 TM-CTC* (Afouras et al., 2018a) 54.7 Conv-seq2seq (Zhang et al., 2019) 51.7 TM-seq2seq* (Afouras et al., 2018a) 50.0 KD-TM (Ren et al., 2021) 49.2 LF-MMI TDNN* (Yu et al., 2020) 48.9 E2E Conformer* (Ma et al., 2021) 42.4 E2E Conformer** (Ma et al., 2021) 37.9 Our Model 43.2 Audio-only TM-CTC* (Afouras et al., 2018a) 10.1 TM-seq2seq* (Afouras et al., 2018a) 9.7 CTC/attention* (Petridis et al., 2018) 8.2 LF-MMI TDNN* (Yu et al., 2020) 6.7 E2E Conformer** (Ma et al., 2021) 3.9 Our Model 2.7 Audio-Visual TM-DCM (Lee et al., 2020) 8.6 TM-seq2seq* (Afouras et al., 2018a) 8.5 TM-CTC* (Afouras et al., 2018a) 8.2 LF-MMI TDNN* (Yu et al., 2020) 5.9 E2E Conformer** (Ma et al., 2021) 3.7 Our Model 2.6 Table 3: Audio-only, visual-only and audio-visual results of word error rate (WER) tested on LRS2. and audio-visual models. Babble noise are added to the audio stream with 5dB SNR and probability of p n = 0 . 25 . The babble noise is synthesized by mixing 20 different audio samples from LRS2. Evaluation: For all experiments, word error rate (WER) are reported which is defined as WER = ( S + D + I ) /N . The S , D and I in the formula denotes the number of substitutions, deletions and insertions respectively from the reference to the hypothesis, and N is the number of words in the inference. The babble noise added to the audio waveform during evaluation is generated using the same manner as training, while we set a different seed to avoid model fit to a specific generated noise. Decoding is performed using joint CTC/attention one-pass decoding (Watanabe et al., 2017) with Modules Ours TM-CTC E2E Conformer Audio front-end 315.0M -3.9M Visual front-end 23.5M 11.2M(freezed) 11.2M Audio back-end 20.2M 20.2M 31.8M Visual back-end 20.2M 20.2M 31.8M Fusion module 19.7M 19.7M 0.8M Decoders 26.2M 20.5K 9.5M Table 4: The parameters comparison of ours, TM-CTC (Afouras et al., 2018a) and E2E Conformer (Ma et al., 2021) models. beam width 5 (the values were determined on the held-out validation set of LRS2). We don't use an external language model in our experiments. We present results for all experiments in Table 3, reporting WERs on visual-only, audio-only and audio-visual models. Note that many of the models listed here are also using extra training data in different stages of training pipeline, such as MV-LRS (Chung and Zisserman, 2017), LRS3 (Afouras et al., 2018b), LibriSpeech (Panayotov et al., 2015) and LRW. We present the parameters of our model, TM-CTC model (Afouras et al., 2018a) and the current state-of-the-art model (Ma et al., 2021) in Table 4. Our model back-ends and fusion module configura-tions follow TM-CTC model, the hyper-parameters settings in the seq2seq decoder are the same as in the back-ends. The most significant difference is that we utilize pre-trained front-ends, resulting in a larger model size. Audio-visual Setting: In the main audio-visual setting, the pre-train and train sets in LRS2 are used as train set in the final training stage. Our proposed audio-visual model achieves a WER of 2.6% without the help of an external language model, which improves by 1.1% over the current state-of-the-art (Ma et al., 2021). This is rather a big improvement, with a relative improvement of around 30%. Audio-only Setting: The training data used for training audio-only model consists of 224 hours labelled data from LRS2, as well as the 60K hours unlabelled data from LibriLight (Kahn et al., 2020) that are indirectly used through inheriting wav2vec 2.0 parameters. Our model also achieves a WER of 2.7%, which reduces the WER of the current state-4496 of-the-art (Ma et al., 2021) by 1.2%, indicating a relative improvement of 31%. Visual-only Setting: The visual-only model uses labelled LRS2 data in its pre-train and train sets, the LRW for supervised pre-training, and indirectly using the 1.28M unlabelled images from ImageNet through MoCo v2. The visual-only model achieves a WER of 43.2%, lagging behind the current state-of-the-art E2E Conformer model (Ma et al., 2021) with 5.3%. Compared to E2E Conformer, the main difference is that a large Transformer language model is used during decoding, which itself brings a 4.5% difference compared with a normal RNN language model in their ablation studies (Ma et al., 2021). The gap between our visual-only model and the E2E Conformer model with a RNN language model is 0.8%, which resides in a quite reasonable range. Additionally, we use a 6-layers Transformer encoder for temporal modelling instead of a 12-layers conformer encoder, which resulted in a smaller back-end size. If we consider a fairer comparison by only looking at benchmarks without using an external language model, the best-reported benchmark is Ren et al. (2021), which achieved a WER of 49.2%, lagging behind our model by 6.0%. In this section, we investigate the impact of every individual building block by testing them in LRW, audio-only and visual-only settings. MoCo v2 Contribution in Visual Word Classification: Results of visual word classification on LRW are shown in Table 5. We first train a model by replacing the ResNet-18 front-end in Stafylakis and Tzimiropoulos (2017) with a ResNet-50 frontend, matching the size of MoCo v2 but with fresh weights. This results in an absolute improvement of 2.1%. Then we initialize the ResNet-50 frontend with MoCo v2 weights and a further absolute improvement of 2.3% is observed, which implies that self-supervised learning is actually functioning in better represent the lip movement. Additionally, When Using 6 layers of Transformer encoder instead of TCN as back-end, we can observe another absolute improvement of 6.0%. We also noticed that using MoCo v2 front-end could significantly reduce the training time. Performance Breakdown in Audio-only Setting: Results of audio-only model on LRS2 are shown in Table 6. Starting from Afouras et al. (2018a), Method Acc Baseline(Stafylakis and Tzimiropoulos, 2017) 74.6% + ResNet-50 front-end 76.7% + MoCo v2 front-end 79.0% + Transformer encoder back-end 85.0 % Table 5: Ablation study on visual word classification performance on LRW. we first train a model by replacing the STFT audio feature with a wav2vec 2.0 front-end pre-trained on LibriSpeech, resulting in an absolute improvement of 11.1%. Then we use another pre-trained model learned on an even larger unlabelled single modality dataset Libri-Light, and a further absolute improvement of 0.6% is observed. We further train the model with a hybrid CTC/attention decoder during the training stage, which results in another absolute improvement of 0.9%. Performance Breakdown in Visual-only Setting: Results of the visual-only model on LRS2 are shown in Table 7. Starting from Afouras et al. (2018a), we first introduce end-to-end training by using a hybrid CTC/attention decoder (the frontend is still pre-trained through LRW), resulting in an absolute improvement of 16.0%. Then we initialize the front-end with pre-trained MoCo v2 weights, a same end-to-end training manner results in a further absolute improvement of 5.8%. Robustness under Noisy Inputs: To evaluate the model's tolerance to audio noise, we tested the performance of our model under babble noise with different SNR levels. Our audio-only and audiovisual models reach WERs of 32.5% and 24.5% when the SNR level is 0dB, respectively, which 4497 Modality Model 0dB 5dB clean AO Afouras et al. (2018a) 58.0% -10.5% Our model 32.5 % 6.8% 2.7 % AV Afouras et al. (2018a) 33.5% -9.4% Our model 24.5 % 6.3% 2.6 % Table 8: Word error rate (WER) under different SNR levels. reduce the reported result in Afouras et al. (2018a) by 25.5% and 9%  . When the SNR level rises to 5dB, our audio-only and audio-visual model obtain WERs of 6.8% and 6.3%. Besides achieving significant improvement over the baseline model under babble noise environment, we further investigate the model performance under human noise environment. The human noise is extremely challenging because the noise itself contains some words, while the model cannot easily distinguish which audio signal is the one to be recognized. We synthesize the human noise by randomly crop many 1 second signals from different audio samples in the LRS2 dataset. As shown in Fig. 3, we conduct experiments varying different levels of human noise, the models are trained using babble noise augmented audio. The WER increases greatly after the SNR level drops down under 0db. It is because the model may not be able to distinguish the two overlapped spoken words at a low SNR level. And the overall performance under each SNR level is worse than babble noise, indicating that noise with specific information is harder than disorganized babble noise. Recognition under Low Resource: A significant benefit of using self-supervised pre-trained models is that only a small amount of labelled data is needed for training a model. To further investigate the models' performance in low resource environment, we use the 28 hours train set of LRS2 to train an audio-only and a visual-only model. The results are shown in Table 9. The audio-only model trained with 28 hours data achieves a WER of 3.4%, which is a little bit worse than the one trained with 224 hours data. The result indicates that for the audio-only model, the self-supervised model pre-trained on a large-scale single modality dataset can significantly reduce the demands of data. While  Ma et al. (2021) also provides a performance under noisy inputs, however, we are not able to compare with them due to a lack of necessary details to generate the same noise. the visual-only model trained with 28 hours data has a great gap with the one trained with 224 hours data, the reason can be that the visual-only model is harder to train and demands a larger amount of data. In this work, we propose to utilize self-supervised learning for AVSR by simply incorporating the pre-trained model trained in massive unlabelled single modality data. Although the visual pre-trained models are not straight-forward to be transplanted into visual front-end, we still manage to integrate pre-trained models in both modalities for the AVSR task. Experimental results are impressive, resulting in a 30% relative improvement. It's interesting to observe that self-supervised model in audio modality has an even larger improvement than that of the visual counterpart. We believe the reasons can be listed as follows:  The training data scale of audio modality is significantly larger than that of visual modality, with the Libri-Light dataset used for pretraining wav2vec 2.0 consists of 60K hours audio signals, the ImageNet dataset, on the con-4498 trary, has only 1.28M images, roughly equivalent to 14 hours silent video under 25 FPS. The MoCo v2 model is pre-trained on images to better represent frame-level contents, while there are no pre-training steps to model the temporal correlation between frames. In contrast, the wav2vec 2.0 model is pre-trained on consistent audios, thus having a better temporal modelling ability. As there has not emerged a dominating cross-modality self-supervised learning approach in the field of AVSR, in future work, we are going to explore two more directions in the self-supervised learning scenario based on this work. The first is utilizing the temporal correlations within the visual domain, while the other is the cross-modal correlations between the audio and visual modality. We hope this work could pave the way towards multimodality self-supervised learning, especially for various aspects in AVSR. This work will not pose ethical problems, the data resources we use are all from published works and do not involve privacy issues related to data collection. The data is collected from BBC and contains thousands of diverse speakers, allowing the speech recognition models to generalize to all speakers. In terms of computational experiments, we used publicly available pre-trained models, which makes the training more environmentally friendly and lowers the computational requirements to reproduce our work. This work was sponsored by the National Natural Science Foundation of China (NSFC) grant (No. 62106143), and Shanghai Pujiang Program. We would like to thank all the anonymous reviewers for their valuable and constructive comments.
abstain abstain abstain abstain method result abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method method abstain abstain objective objective other other other other other other other other other other other other other other other other other abstain abstain other abstain abstain method abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other method abstain abstain abstain abstain method abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result result abstain abstain abstain abstain result result abstain abstain result abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain method other other