Hi albanD

No I did not explicitly use cudnn.
I did not use it in benchmark mode.
These were my memory and gpu memory usage when I ran my training with input size = 320, batch size = 9:

[2018-12-12 13:27:27,480 train.py] ram memory info before loss.backward(): svmem(total=17138393088, available=5950562304, percent=65.3, used=11187830784, free=5950562304)
[2018-12-12 13:27:27,481 train.py] max_memory_allocated before loss.backward(): 2788280832
[2018-12-12 13:27:27,481 train.py] memory_cached before loss.backward(): 2799042560
[2018-12-12 13:27:27,482 train.py] max_memory_cached before loss.backward(): 2799042560
[2018-12-12 13:27:27,482 train.py] memory_allocated before loss.backward(): 2772128768
Erros came right after this
line 70, in train
loss.backward()
File “C:\Users\jayde\Anaconda3\envs\new_env\lib\site-packages\torch\tensor.py”, line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “C:\Users\jayde\Anaconda3\envs\new_env\lib\site-packages\torch\autograd_init_.py”, line 90, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: out of memory
However when I set input size = 320, batch size = 16,  this is what I get:
[2018-12-12 13:32:11,560 train.py] ram memory info before loss.backward(): svmem(total=17138393088, available=5892562944, percent=65.6, used=11245830144, free=5892562944)
[2018-12-12 13:32:11,561 train.py] max_memory_allocated before loss.backward(): 4837138432
[2018-12-12 13:32:11,561 train.py] memory_cached before loss.backward(): 5020450816
[2018-12-12 13:32:11,562 train.py] max_memory_cached before loss.backward(): 5020450816
[2018-12-12 13:32:11,562 train.py] memory_allocated before loss.backward(): 4684070912
[2018-12-12 13:32:11,749 train.py] ram memory info after loss.backward(): svmem(total=17138393088, available=5946978304, percent=65.3, used=11191414784, free=5946978304)
[2018-12-12 13:32:11,749 train.py] max_memory_allocated after loss.backward(): 5460978176
[2018-12-12 13:32:11,749 train.py] memory_cached after loss.backward(): 6174408704
[2018-12-12 13:32:11,750 train.py] max_memory_cached after loss.backward(): 6174408704
[2018-12-12 13:32:11,750 train.py] memory_allocated after loss.backward(): 518663680
[2018-12-12 13:32:12,162 train.py] ram memory info before loss.backward(): svmem(total=17138393088, available=5980577792, percent=65.1, used=11157815296, free=5980577792)
[2018-12-12 13:32:12,162 train.py] max_memory_allocated before loss.backward(): 5460978176
[2018-12-12 13:32:12,163 train.py] memory_cached before loss.backward(): 6193283072
[2018-12-12 13:32:12,163 train.py] max_memory_cached before loss.backward(): 6193283072
[2018-12-12 13:32:12,163 train.py] memory_allocated before loss.backward(): 5199972864
[2018-12-12 13:32:12,315 train.py] ram memory info after loss.backward(): svmem(total=17138393088, available=5965262848, percent=65.2, used=111731
No memory errors were raised.
Thank you again for helping me!