Hi!
Facing a problem whilst trying to finetune the ByT5 model for the text classification task.
I’ve tried to use this notebook exploring-T5/t5_fine_tuning.ipynb at master · patil-suraj/exploring-T5 · GitHub 5 by @valhalla for ByT5 finetuning on the text classification task. When I start from ‘google/byt5-small’ I get really strange results. A model always generates negative sentiment label (‘n’ in my code)

Screenshot 2021-07-30 at 13.04.301356×458 40.4 KB

When I switch to t5-small pretrained checkpoint the results are reasonable

Screenshot 2021-07-30 at 12.19.551356×458 41.4 KB

What am I missing here? Any piece of advice would be much appreciated!