You shouldn’t change any layers, as this will be done automatically for you.
If you encounter any instabilities, please post the layer and ping me so that we can have a look at it.
During inference you could also use mixed-precision training and it should not affect the testing performance (same as during training).