We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages a strong language model to provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on a clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to a fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, we further improve the WER on the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self-supervised approach.
Overview:
This paper is dedicated to proposing a self-supervised objective, local prior matching (LMP), for speech recognition. This approach can take advantage of vase quantities of unlabeled speech data. What' more, the objective is simple to implement and theoretically well-motivated. In the paper, based on a supervised pretrained model, it then finetunes 360 hours with unlabeled data and LPM reduces the WER consistently. They also conduct extensive ablation experiments to show the effect of their self-supervised approach.
Strength Bullets:
1. I think this self-supervised learning objective (LMP) is very novel. The motivation that the source of indirect supervision on processing unlabeled speech comes from prior knowledge about the world and the context of the speech makes sense to me. The author combines the Bayesian method to build the model which is aligned with the motivation. They also provide clear and well-organized derivations. 
2. The paper performs extensive ablation studies over all components, including beam size, mixing ratio, LPM weights, model update strategies, model initialization, length filtering and choice of language models. It provides convincing evidence of the effect of each component. 
3. It provides interesting experiments results to study the relationship between the amount of unlabelled data for self-supervision and final performance. And LPM can surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data.
Weakness Bullets:
1. The paper only evaluates their method on LibriSpeech dataset. Although this dataset is popular, one or two more datasets will be more convincing. 
2. For bayesian based methods, it is well known that it performs badly in high dimensional space. The reason is that we can not sample enough data points to obtain a good posterior estimation. Could you provide more analysis about the quality of posterior with different amount of sampled data?
3. For the experiment between the amount of unlabelled data for self-supervision and final performance, it would be better the author can provide a curve with more results.
Recommendation:
I think it is a good paper. The proposed approach is useful. This is a weak accept.