Ok, a couple more updates:

When I ran with 2GPUs and had an error, training_args.parallel_mode was equal to ParallelMode.DISTRIBUTED, so I’m not sure what’s going wrong there
I finished running training on common_voice for 5 epochs on a single GPU, my final WER on the evaluation set remained around 0.39. In the end, training took about 19 hours.

As a next step, I may try to load in a portion of the OpenSTT dataset and add this to the training set to see if it improves performance. Any other ideas?