+1. wav2vec 2.0 is one of the state-of-the-art models for modern ASR at the moment. You can find it on TF Hub: wav2vec2 1 and wav2vec2-960h.
However, do note that, as mentioned in the wav2vec 2.0 2 paper, in the original experiment, the authors appeared to have used raw data as inputs, instead of visual representations like (log-mel) spectrograms (which is what this experiment used during pretraining 1 instead, as mentioned further below):

“In this paper, we present a framework for self-supervised learning of representations from raw audio data.  Our approach encodes speech audio via a multi-layer convolutional neural network and then masks spans of the resulting latent speech representations [26, 56], similar to masked language modeling.”

It’s also mentioned in the wav2vec-U (unsupervised wav2vec for ASR) paper:

“Wav2vec 2.0 consists of a convolutional feature encoder f : X 7→ Z that maps a raw audio sequence X to latent speech representations z1, . . . , zT , which a Transformer g : Z 7→ C then turns into context representations c1, . . . , cT (Baevski et al., 2020b,a).”

There’s another model called Conformer: Convolution-augmented Transformer for Speech Recognition (TF Hub has a TF Lite quantized version 1). In the paper, the authors describe the experiment’s data preprocessing step as follows:

“We evaluate the proposed model on the LibriSpeech [26] dataset… We extracted 80-channel filterbanks features computed from a 25ms window with a stride of 10ms. We use SpecAugment [27, 28] with mask parameter (F = 27), and ten time masks with maximum time-mask ratio (pS = 0.05), where the maximum-size of the time mask is set to pS times the length of the utterance.”

(where “SpecAugment applies an augmentation policy directly to the audio spectrogram” - source: https://ai.googleblog.com/2019/04/specaugment-new-data-augmentation.html.)
Another ASR/Conformer-related method: Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition 1, which uses scaled up versions of the network called Conformer XL and Conformer XXL with up to 1 billion parameters:

"• Model: We use Conformers [5] (and slight variants thereof) as the model architecture.
“• Pre-training: We use wav2vec 2.0 pre-training [6] to pre-train the encoders of the networks.
" • Iterative Self-Training: We use noisy student training (NST) [7, 8] with adaptive SpecAugment [9, 10] as our iterative self-training pipeline.”


image1061×725 173 KB

And, this experiment appeared to have used log-mel spectrograms:

“…We pre-train the Conformer encoder akin to wav2vec 2.0 pre-training [6] with utterances from the “unlab-60k” subset of Libri-Light [2]. Unlike in the original work which takes raw waveforms as input, we use log-mel spectrograms.”

Another interesting post about the Conformer:

  
      

      Medium – 23 Oct 20
  

  
    

Google Brain Sets New Semi-Supervised Learning SOTA in Speech Recognition 2

  Google improved the SOTA on the LibriSpeech automatic speech recognition task, with their score of 1.4 percent/ 2.6 percent…

  
    Reading time: 4 min read