Solved by ptab in post #2 
              
              
                Set X.requires_grad = True for your input image X . Run X through your network. By calling .backward on a scalar value calculated from your network output, gradients of that scalar with respect to the input image X will be calculated and stored in X.grad.