deepweaver:

However, when optimizing generator, the gradient for the discriminator parameters is not zeroed! It still has the gradient for optimizing the discriminator.


@ptrblck
Then if we zero_grad the discriminator before updating the generator would not have any effect? But it would be more memory efficient?
What if I have a third network which loss is based on the discriminator intermediate layers:
            # ------------------
            # Update controller
            # ------------------
            G.eval()
            D.eval()
            C.train()
            optims['C'].zero_grad()

            c = C(sample_dys)
            C_loss = D.lap1_loss(G(c), sample_dys)

            C_loss.backward()
            optims['C'].step()

where lap1 loss is:
    def lap1_loss(self, x: tr.tensor, y: tr.tensor):
        """
        Implements the laplace loss for the discriminator layers.
        Input shape: (N, C, H, W)
        :return: the scalar loss value
        """
        assert x.shape == y.shape, "The shape of inputs must be equal."
        assert len(x.shape) == 4, "Input must be 4 dimensional."

        _, x_acts = self.forward(x)
        _, y_acts = self.forward(y)

        losses = [trf.l1_loss(x_l, y_l)*2**(2*l) for l, (x_l, y_l) in enumerate(zip(x_acts, y_acts))]
        loss = tr.sum(tr.tensor(losses, requires_grad=True))

        return loss

Here we should also zero_grad() the discriminator network or the situation is the same here and it is not a problem that gradients are accumulated in the discriminator?