Note: I reduced my image size by half in both dimensions:
transforms.Resize((224,336),interpolation=Image.NEAREST)

and was able to save the model. The saved model size is 2.3GB in size just at this image size!