Solved by ptrblck in post #2 
              
              
                This line of code looks strange: 
g_loss.backward(retain_graph=True)

as it would keep the computation graph in G alive. Since you are also not detaching fake in: 
fake_loss = criterion(D(fake), fake_labels)

these operations: 
d_loss = (real_loss + fake_loss)/2 + norm_penalty
d_loss.backward(retain…