Going from 50x50 to 320x320 can’t really be described as ‘slightly larger’! For 50x50 images, the first FC has input size of 288 and output size of 7500. For 320x320 images, that layer has input size 12800 and output size 307200. So in the first case fc1 has 2160000 trainable parameters and for the larger images it has 3932160000 trainable parameters. You will likely need to change the architecture if you need to use larger images.