• The direct application of BERT-type training to speech input presents challenges due to the
unsegmented and unstructured nature of speech. To overcome this obstacle, a pioneering
model known as Discrete BERT [23] has been developed. This model converts continuous
speech input into a sequence of discrete codes, facilitating code representation learning.
The discrete units are obtained from a pre-trained vq-wav2vec model [25], and they serve as
both inputs and targets within a standard BERT model. The architecture of Discrete BERT,
illustrated in Figure 13 (a), incorporates a softmax normalized output layer. During training,
categorical cross-entropy loss is employed, with a masked perspective of the original speech
input utilized for predicting code representations. Remarkably, the Discrete BERT model
has exhibited impressive efficacy in self-supervised speech representation learning. Even
with a mere 10-minute fine-tuning set, it achieved a Word Error Rate (WER) of 25% on
AReviewofDeepLearningTechniquesforSpeechProcessing