Hello,
I had the same issue with mT5 (both small and base) on BoolQ dataset (~9.5k train samples) and found out something that may be useful to you.
No matter what settings I used, how long I trained, and whether I oversampled the minority class on training set, all predictions on validation set were the same. Interestingly, this only occurred when using boolean QA data with mT5. Other tasks such as SQuAD, or switching to T5, worked just fine.
So, I looked into the differences of the pre-training stage between T5 and mT5. One thing to note is that no supervised pre-training is used in mT5. Since mT5 works fine on SQuAD, I trained for one epoch on SQuAD before proceeding to train on BoolQ using the settings described in the mT5 paper. This resolved the issue for me and now the accuracy improves as expected.
In short: Train on some other tasks first.
I hope it helps you too!