I apologize, I might have not been so clear in my explanation.  The situation is this:

When I run a super-res model for a particular training batch size (say 4) with a constant validation batch size (always 1), the model slowly accumulates GPU memory through each batch of training and validation inputs until a certain stable limit (say 4GB), where it remains until the end of my training.  This limit is normally achieved by the end of my first epoch (a single run-through all training input images and validation images).
When I increase the batch size (to say 8), this memory limit increases to, say, 8GB, which is also fine for an 11GB card.  This is of course normal to consume more memory with a higher batch size.
However, when I increase the training batch size to 16, the memory limit error does not trigger during training, but actually during validation, where the batch size should not have changed at all.  This means that technically I could achieve a batch size of 16 with my GPU, but is somehow being held back during the validation pass.  I tried to flush the GPU memory using torch.cuda.empty_cache() after my training pass (right before my validation pass) but this seemingly wasn’t enough to prevent the error from occurring.
I do observe the GPU memory increasing when I send my model to the GPU, but I also observe that, after training, the GPU memory does not drop back to the value it had before training (with just the model loaded in).  This is measured using nvidia-smi, after I run the empty cache command.