Hello Tom,
Thanks for your answer. Yes I do use AMP with gradient scaling following this recipe 2.
By the way, as the optimizer update is outside of the amp autocast context, is the small epsilon really problematic with respect to the precision?
Thanks for the suggestion to use parameter groups, it may be a good idea I will probably investigate this solution.
For now, I am doing few experiments with amp disabled and a batch size halved, but I will take you inform If I use back AMP.
thanks