That might be possible, as it seems A is used in the loss calculation and might not be using the aforementioned stale activations. In any case, you could just run the code and see, if Autograd would raise an error.