Shallow Fusion is a very common technique in ASR,  is basically combine an acoustic model as wav2vec2 with a pretrained language model, you train it with the same vocabulary as the acoustic model and at the inference time you combine both output combining the logits of the two models by:
am_y = p(y|x)
lm_y = lm(y|x)
y = argmax log am_y + λlog lm_y
Using this you can improve a lot the model performance.
It’s the same idea we’re discussing here.
This paper is a good begining: https://arxiv.org/pdf/1807.10857.pdf 11