Solved by tom in post #2 
              
              
                If you only need to gradients w.r.t. the inputs (and don’t have bad things like batch norm), summing the outputs and then calling .backward will get you gradients. For weights it is more tricky to avoid gradients being accumulated.

inp.requires_grad_() before passing it into the network will tell…