How large are the differences between the image resolutions?
Are you using any nn.BatchNorm layers in your model?
If so, could you remove them or replace them with nn.InstanceNorm?
Since you are feeding the samples one by one, the running estimates might be quite off.