Solved by Yaroslav_Bulatov in post #4 
              
              
                If you just have conv/linear layers, you could use this – https://github.com/cybertronai/autograd-hacks#per-example-gradients