Which GPU are you using and how much memory does it have?
I just used the torchvision.models.vgg13 model with your custom classifier and have a memory allocation of ~5.2GB after running the forward and backward pass using input = torch.randn(20, 3, 224, 224, device='cuda').