There is an official example for PyTorch’s amp.
If I were in your place, I shall stick to the larger model itself. They are have already learned a lot of patterns and even though they take more time per epoch, they only need a few of them to give good results. I would strongly suggest you first try freezing all blocks except the last and finetune on your dataset. That’s the best way forward. This should be much much faster to train too.
Irrespective of number of classes, the models should learn a ton of features and should be able to generalize. I would say only a small portion of the last layers would be focusing on the class level patterns.
I hope this helps.