How much RAM does your GPU have, what is the batch size, and what is the tensor size?
I’ve found that the larger the images/tensors, the more channels (RGB vs Grayscale) and the larger the batch-sizes, the more GPU RAM is used. Run nvidia-smi and monitor GPU RAM whilst playing around with your parameters?