PyTorch keeps track of activations – you can reuse the same network for different inputs and/or losses and things work as expected. For instance, official GAN example 10 does this
One thing to keep in mind is that layer.weight.grad accumulates gradients it has seen, but this is normally what you want. You can use hooks to print what autograd sees during interleaved forward/backward passes – example colab 12