I have tried to capture the scale factor on each batch (100 epochs, batch size: 64, 24 batches per epoch) and then plot it:

Right now I am not getting the warning, but still it shows up some now and then. Anyway I think that we know what is causing this issue. @ptrblck Many thanks for your pointers.