I get 3.97 GB as output from the code snippet(I have modified it slightly, please check) and if I use del on loss, outputs and other parameters used/created in the training step then it reduces to 2.35 GB. Main problem is that nvidia-smi is showing consumption of complete 16GB.
Here is the modified code snippet -
total = 0
for obj in gc.get_objects():
  try:
    if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
      if len(obj.size()) > 0:
        if obj.type() == 'torch.cuda.FloatTensor':
          total += reduce(lambda x, y: x*y, obj.size()) * 32
        elif obj.type() == 'torch.cuda.LongTensor':
          total += reduce(lambda x, y: x*y, obj.size()) * 64
        elif obj.type() == 'torch.cuda.IntTensor':
          total += reduce(lambda x, y: x*y, obj.size()) * 32
        #else:
          # Few non-cuda tensors in my case from dataloader
  except Exception as e:
    pass
print("{} GB".format(total/((1024**3) * 8)))

Some more updates on the issue:
If I disable backprop part of the model then the snippet gives 0.78GB as output but the program crashes very fastly when I don’t use del logits in this case as it doesn’t use existing memory(maybe loss.backward() has some flag which allows to reclaim unused part of GPU