Hey @OlivierCR,
can you please share all of your hyperparameters you used? tran_batch_size = 128 seems pretty high to me to work.
Are you using distributed training?
The training time reduction should be way higher more around 20-40%.
When using the example have you adjusted train.py to accept fp16 as hyperparameter or have you defined it directly in the script?