You can pretrain it using this Script transformers/run_wav2vec2_pretraining_no_trainer.py at master · huggingface/transformers · GitHub 10
but except it could be really unstable to pretrain from scratch as it’s written in the readme