Hey,
Note that we now have an official fine-tuning example that also works for HuBERT:

  
      

      github.com
  

  
    transformers/examples/pytorch/speech-recognition at master ·... 35

  master/examples/pytorch/speech-recognition

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/speech-recognition at master · huggingface/transformers

  

  
    
    
  

  


Also see examples below:
  
      

      github.com
  

  
    transformers/examples/pytorch/speech-recognition at master ·... - Examples 23

  master/examples/pytorch/speech-recognition

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/speech-recognition at master · huggingface/transformers