Hello, I have the following code snippet which gives me ‘cuda out of memory error’ after several passes of the for loop using batch size (b) 50 or above. However if I use batch size less than 40, it seems run fine. My question is in case of larger batch size (>=50), why it is giving error after several passes? Is there any memory overhead accumulating inside the loop?
model = Video3DCNN('resnet34',10, sample_size=224)
model.to('cuda:0')
optimizer1 = optim.Adam(model.parameters(), lr=0.0005)
loss = nn.CrossEntropyLoss()
b = 50
ep_loss = 0.
for i in range(100):
  torch.cuda.empty_cache()
  logits = model(torch.randn(b, 3, 10, 224, 224).cuda())
  label = torch.randint(high=1000, size=(b,)).cuda()
  loss_ = loss(logits, label)
  optimizer1.zero_grad()
  print (i, loss_.item())
  print (logits.size())
  loss_.backward()
  optimizer1.step()
  ep_loss += loss_.detach().item()

Output for batch size b==50,
/home/ahosain/workspace/cslr/resnet_3d.py:152: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
m.weight = nn.init.kaiming_normal(m.weight, mode=‘fan_out’)
0 6.941838264465332
torch.Size([50, 1000])
1 7.274449348449707
torch.Size([50, 1000])
2 7.0289177894592285
torch.Size([50, 1000])
3 7.262955188751221
torch.Size([50, 1000])
Traceback (most recent call last):
File “test.py”, line 38, in 
loss_.backward()
File “/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/tensor.py”, line 166, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/ahosain/workspace/dl_env/lib/python3.6/site-packages/torch/autograd/init.py”, line 99, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 10.76 GiB total capacity; 5.23 GiB already allocated; 1.48 GiB free; 3.19 GiB cached)