Rafi:

So my though is that the mixed precision is used just for the training phase and the model is saved all as 32 float??


Yes, the parameters and buffers would be stored in float32. If you don’t use autocast during inference, then you would execute the standard FP32 model.
Note that mixed precision utilities are now usable in PyTorch directly without apex, so I would recommend to update the code to torch.cuda.amp 11.