This is my learning part:
for bag in d_bags:
                    loss = - self.loss(d_bags[bag], b_bags[bag])
                    loss.backward(retain_graph=True)
                    opt.step()

It successfully performs the first iteration and then fails on loss.backward. Without opt.step like this:
for bag in d_bags:
                    loss = - self.loss(d_bags[bag], b_bags[bag])
                    loss.backward(retain_graph=True)

It does not fail.