The dimension of the input is (1, 4000, 200, 200, 20) in which the first one is the batch size. Here is my watchdog window for each command during debugging:
y = Model(x)      ######## Watchdog GPU Memory Usage: 1085MiB
-> Class Model(Module):
def  init (self):
…
def forward (self, input):
tensor = input  ######## Watchdog: 8279MiB
Seems that OOM happens when the GPU memory usage is 8469MiB in a layer between but the largest part of the memory is used before getting into the forward pass which I guess makes the problem.