@TOPRAN, are you the same person as the one who posted this? [wav2vec] deepspeed eval bug in the case of >1 gpus · Issue #11446 · huggingface/transformers · GitHub 25 - as it appears very similar - See my followup in that issue which partially addresses this problem - let’s continue the discussion in the issue.