Your code might have some typos.
self.conv2 should accept 32 in_channels based on your forward method.
Also, are you resizing the images?
If not, I think your fc1 layer should accept 128 * 11 * 11 features. At least that’s working with input images of 96x96.
Using a batch size of 128 the training takes approx. 1.6GB so your GPU have enough RAM.
Are other processes filling up your GPU maybe?