@ptrblck, thank for you answer. I have some questions about AMP, maybe eu can clarify for me.
1 - About weight initilization: let’s say my initializer produces very small weights, will them flush to zero when run inside autocast?
2 - What about model inputs: if I am working with vector of very small floats as inputs to my model, what happens to them?
Thanks