I did the following and now it got me totally lost:
I moved the backward, step outside the loop as you said
loss = 0
for bag in d_bags:
      loss = loss - self.loss(d_bags[bag], b_bags[bag])
loss.backward()
opt.step()

It was giving the same error.
Then I tried to remove the batchnorm layer totally. With the anomaly flag off it is giving still the same error of inplaced operations:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. (apply at ../torch/csrc/autograd/generated/Functions.cpp:3969)


Then I turned the anomaly flag on torch.autograd.set_detect_anomaly(True) and it gave a different error:
RuntimeError: Function 'MulBackward0' returned nan values in its 1th output.