Hi,
I guess you use cudnn here?
We use the fastest possible cudnn algorithm by default which can consume more memory.
Setting torch.backends.cudnn.deterministic=True will force it to use the default deterministic algorithm which should consume very few memory.
Note that for best speed performance you should use torch.backends.cudnn.benchmark=True that uses cudnn builtin benchmarking tool to choose the best algorithm for your task and gpu.