Solved by tom in post #2 
              
              
                It depends: for each operation (there might be several in a layer), autograd stores the inputs and/or outputs needed to compute the backward. 
On newer PyTorch, the Node objects (.gradfn /. gradfn.next_functions) have “non-official” ._saved… attributes where you can see what has been stored. 
Best r…