ptrblck:

That shouldn’t be the case, since autocast should take care of using FP16, where it’s safe to do so, and fallback to FP32 if necessary.


Awesome! I’ll be testing that today then.



 ptrblck:

Could you post the setup for the loss function, so that I could run it in isolation?


Yes, for normal use it should be like :
layers = {"conv_3_2": 1.0, "conv_4_2": 1.0}
Contextual_Loss(layers, crop_quarter=False, max_1d_size=64, distance_type = 'cosine', b=1.0, band_width=0.5, use_vgg = True, net = 'vgg19', calc_type = 'regular')




 ptrblck:

The initial loss scale might drop and stabilize, which is the expected behavior.


Perfect! That makes sense.
Thanks a lot for your help!