I revisited this with the latest hf and tried it with fp16 and it seems to work now.
Also had a similar issue with roberta-large models on xnli and paws. Tried with fp16 and fp32 and every time, one of them worked.