Hi,
I have the same code running on two platforms:
platform1: Quadro P4000 (8119 MB RAM)
platform2: Titan V (12033 MB RAM)
I’m using mini batches of size 400 on both platforms.
While in platform1 everything works fine (consumes 7295/8119 MB , 99% volatile memory usage)
But, in platform2 it runs in CUDA: Out of Memory Error while only consuming 1129/12033 MB, 0% volatile memory usage and stops.
Traceback (most recent call last):                                             │                                                                               
  File "run_meenet1.py", line 151, in <module>                                 │                                                                               
    criterion=criterion)                                                       │                                                                               
  File "/home/ubuntu/projectSSML/meenet/modules/helpers.py", line 90, in train_│                                                                               
batchwise                                                                      │                                                                               
    loss.backward()                                                            │                                                                               
  File "/home/ubuntu/.local/lib/python3.7/site-packages/torch/tensor.py", line │                                                                               
93, in backward                                                                │                                                                               
    torch.autograd.backward(self, gradient, retain_graph, create_graph)        │                                                                               
  File "/home/ubuntu/.local/lib/python3.7/site-packages/torch/autograd/__init__│                                                                               
.py", line 90, in backward                                                     │                                                                               
    allow_unreachable=True)  # allow_unreachable flag                          │                                                                               
RuntimeError: CUDA error: out of memory           

The scripts are exactly the same.
What might be going wrong.
Thanks