A short reply for now. More details will follow.
It seems to slightly depend on the batch size. Using gradient accumulation to augment the smaller batch sizes (of longer sequence length) corrects my issue where the model previously only predicted 1 or 0. So it might be the case that it just can’t generalize enough if the batch size is too small.
But this is still somehow connected to the learning rate and optimizer (I would assume), as with an older implementation of BERT finetuning in MXNet I could train with batch sizes of 2 and still be on the same level as with shorter sequence lengths or even better. The current finetuning code from GluonNLP (MXNet) seems to have a similar issue with the longer sequence lengths/smaller batch sizes, and gradient accumulation helped here, too. So finding the changes compared to the older code might help find the root cause.