No the train_batch_size is the batch_size per device. See notebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHub 1 or in the documentation here Trainer — transformers 4.7.0 documentation 2
This speaking train_batch_size = 256 would have resulted in 256*8 = 2048 which is not possible.
Have you modified train.py otherwise fp16:True will not be used. Since it is not parsed here notebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHub 1