By “backward activation” I meant the data gradient. In particular the weight gradient would need the stored forward activations as described in this tutorial by Michael Nielsen.



 vishal-purohit:

Can I access and replace these intermediate activations mentioned in this sentence “which will create intermediate activations (outputs of layers) and store them” ???


That is exactly what I think is not easily possible, since you cannot directly access these tensors (at least I don’t know how it would work with native PyTorch methods).
You might be able to use a hacky way of defining custom autograd.Functions, store the forward activations in the context, and move it to another device, but I don’t know how much work this would be.