@mcarilli thanks for the writeup.

Can you rerun your existing code (with  growth_interval=10 ) and print  scaler.get_scale()  just after  scaler.update()  for the first few dozen steps to get a sense for the scale value it finds, and post the results here?

So the results for scaler.get_scale() is as follows
iter 1 - 24: goes from 65536.0 to 0.00390625
iter 24 - 256: goes from  0.00390625 to 32768.0
iter 256 - 1000: bounces around from 1024 - 16384.0 but stabilizes at around 1024 near 1000.
iter 1000 - 50000: bounces around from 256 - 2048.0 but from eye balling it, the 1024 seems to be what occurs most frequently.
It seems like my network needs a range of scale values in the beginning to start stabilizing, then bounces around once the model starts learning.