+1, this works like a charm
Also implements loss scaling, which I found to be necessary
On small models you’ll not see much of an uplift, but on a big imagenet model like resnet18 or resnet50 you should see ~2x the performance (at least on a V100).
Also make sure you have the latest cuDNN, they’re up to 7.3.1 now
Edit: Apex works like a charm