Solved by Yaroslav_Bulatov in post #3 
              
              
                There’s a trick from Ian Goodfellow to do this in a single pass: https://arxiv.org/abs/1510.01799 
This post gives PyTorch recipe – Efficient Per-Example Gradient Computations