I have some additional questions that I did not yet find the answers to:
Should I be making edits in my model such that the batchnorm layers are run in FP32 mode? Or is this done automatically? (as I understood they might underflow/overflow otherwise)
How does it work when using your model in mixed precision for testing/inference, is it normal to also do this in mixed precision mode (could it negatively affect testing performance)?
Thanks!