Maybe try disabling/enabling cudnn.benchmark mode? Depending on the algorithm used for the specific image size, it might require a lot of memory. Apart from that, I do not have other ideas at the moment.