The activations are stored in the computation graph so they won’t be accumulated or replaced.
I.e. if train_data is a new input tensor, which is not attached to any computation graph (e.g. from the previous forward pass) and if you compute the loss based on the output of the second pass only, the gradients will use only these intermediate activations.
However, if you reuse a computation graph, Autograd will backpropagate through the entire graph and will use all intermediate activations. This would be the case for e.g. a recursive model where the output is fed as the new input in a loop.