Solved by bsridatta in post #4 
              
              
                There is an official example for PyTorch’s amp. 
If I were in your place, I shall stick to the larger model itself. They are have already learned a lot of patterns and even though they take more time per epoch, they only need a few of them to give good results. I would strongly suggest you first try…