torch.cuda.amp.atocast will not change the dtype of the model parameters and if you explicitly want to use float16 in the parameters (without the master copies as was done in the deprecated apex.amp O2 level), they you would indeed need to call half() on the desired layers.