deepspeed doesn’t work with autocast, it has its own way of dealing with mixed precision, if you look in the trainer.py it’s carefully bypassed.
If after removing autocast the problem persistes, please let’s use an Issue for debugging problems, so it’s easy to track.
Edit: I see it was already filed: [wav2vec] deepspeed eval bug in the case of >1 gpus · Issue #11446 · huggingface/transformers · GitHub 19
Thank you.