Hey @aaronchavez
as explained in this issue Improve PyTorch examples for FP16 · Issue #9752 · huggingface/transformers · GitHub 54
To get the full speed-up of FP16 training, every tensor passed through the model should have all its dimensions be a multiple of 8.