Answers.Body: <p>In the paper <a href="https://arxiv.org/pdf/1511.07528.pdf" rel="nofollow noreferrer"><em>The Limitations of Deep Learning in Adversarial Settings</em>, Papernot et. al., 2016, IEEE</a> the chain rule is used, "To express <span class="math-container">$\nabla F(X∗)$</span> in terms of <span class="math-container">$X$</span> and constant values only."</p>
  
  <p>Earlier is stated, "Our understanding of how changes made to inputs affect a DNN’s output stems from the evaluation of the forward derivative: a matrix we introduce and define as the Jacobian of the function learned by the DNN. The forward derivative is used to construct adversarial saliency maps indicating input features to include in perturbation <span class="math-container">$\partial X$</span> in order to produce adversarial samples inducing a certain behavior from the DNN;"</p>
  
  <p>And later, "We define the forward derivative as the Jacobian matrix of the function F learned by the neural network during training. For this example, the output of F is one dimensional, the matrix is therefore reduced to a vector (below). Both components of this vector are computable using the adversary’s knowledge, and later we show how to compute this term efficiently."</p>
  
  <p><span class="math-container">$\nabla F(X) = \Big[ \frac {\partial F(X)} {\partial X_1}, \frac {\partial F(X)} {\partial X_2} \Big] \quad\quad (2)$</span></p>
  
  <p>Between equation (2) and equation (6) your questeion is answered, resulting in an equation where the chain rule has already been applied.</p>
  
  <p><span class="math-container">$\frac {\partial F_j (X)} {\partial x_i} = \Big(W_{n+1,j} \, . \, \frac {\partial H_n} {\partial x_i} \Big) \times \frac {\partial f_{n+1,j}} {\partial x_i} \;(W_{n+1,j} \, . \, H_n + b_{n+1,j}) \quad\quad(6)$</span></p>
  
  <p>It is (6) that you must implement in code, but that is done after the DNN is converged as the paper states above.</p>
  
  <p><strong>Applying the Chain Rule</strong></p>
  
  <p>If <span class="math-container">$z = f(y)$</span> and <span class="math-container">$y = g(x)$</span>, then <span class="math-container">$\frac {dz} {dx} = f'(g(x)) g'(x)$</span></p>
  
  <p>Applied twice, if <span class="math-container">$w = e(z)$</span>, <span class="math-container">$z = f(y)$</span>, and <span class="math-container">$y = g(x)$</span>, then <span class="math-container">$\frac {dw} {dx} = e'(f(g(x)) f'(g(x)) g'(x)$</span></p>
  
  <p>Applied indefinitely we multiply all the first order derivatives of intermediate values, which has the following first order predicate logic expression.</p>
  
  <p><span class="math-container">$\forall \; 0 \le n \lt N \land v_{n + 1} = f_{n + 1}(v_n) \land F_{n+1}(x) = f_{n+1}(F_n(x)) \land g(a) = \dfrac {d} {d a} \, f(a)$</span>
  <span class="math-container">$\exists \; g_n(v_0) = \prod_{\,n = 1}^{\,n &lt; N} {g_n} (F_n(v_0))$</span></p>
  
  <p>Derivatives of convolution kernel values are the values themselves because they represent attenuation of the form <span class="math-container">$y = ax + b$</span>.  This is true of first degree linear activation functions too.  The derivative normally used for ReLU (although the x = 0 case is actually undefined) is as follows:</p>
  
  <p><span class="math-container">$\dfrac {d} {dx} R(x) = {\begin{cases}x &lt; 0 &amp; 0\\x = 0 &amp; 1\\x &gt; 0 &amp; 1 \end{cases}}$</span></p>
  
  <p>The derivative of a max function used in max pool (although the u = v case is actually undefined) is as follows:</p>
  
  <p><span class="math-container">$\dfrac {d} {dt} M(u, v) = {\begin{cases}u &lt; v &amp; \frac {dv} {dt}\\u = v &amp; \frac {dv} {dt}\\u &gt; v &amp; \frac {du} {dt}\end{cases}}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-21T04:52:44.130Z
Answers.Text: Thanks for the answer. Once my model is trained i.e the DNN has converged, I want to compute equation (6) but I don't know how to do that when there are different types of layers like conv2d, linear, relu, maxpool etc. Could you please provide me with some simple example (in PyTorch) if possible? Thanks for your help once again.
Answers.UserDisplayName: Akhilesh Pandey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-26T05:04:38.630Z
Answers.Text: Thanks for the answer. It was really very helpful. However, as of now, I am able to compute the Jacobian w.r.t single X. Is it possible to extend, the same technique in the case where I am working in batches?
Answers.UserDisplayName: Akhilesh Pandey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-27T03:32:42.763Z
Answers.Text: Yes, I am referring to mini-batch.
Answers.UserDisplayName: Akhilesh Pandey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-20T18:32:39.357Z
Answers.LastActivityDate: 2018-09-21T19:59:50.607Z
Answers.LastEditDate: 2018-09-21T19:59:50.607Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I am trying to implement <a href="https://arxiv.org/abs/1511.07528" rel="nofollow noreferrer">this paper</a>. In this paper, the author uses the forward derivative to compute the Jacobian matrix <b>dF/dx</b> using chain rule where F is the probability got from the last layer and X is input image.
  My model is given below. Kindly let me know how to go about doing that?</p>
  
  <pre><code>class LeNet5(nn.Module):
  
  def __init__(self):
  
      self.derivative= None # store derivative
  
      super(LeNet5, self).__init__()
      self.conv1= nn.Conv2d(1,6,5)
      self.relu1= nn.ReLU()
      self.maxpool1= nn.MaxPool2d(2,2)
  
      self.conv2= nn.Conv2d(6,16,5)
      self.relu2= nn.ReLU()
      self.maxpool2= nn.MaxPool2d(2,2)
  
      self.conv3= nn.Conv2d(16,120,5)
      self.relu3= nn.ReLU()
  
      self.fc1= nn.Linear(120,84)
      self.relu4= nn.ReLU()
  
      self.fc2= nn.Linear(84,10)
      self.softmax= nn.Softmax(dim= -1)
  
  
  def forward(self,img, forward_derivative= False):
      output= self.conv1(img)
      output= self.relu1(output)
      output= self.maxpool1(output)
  
      output= self.conv2(output)
      output= self.relu2(output)
      output= self.maxpool2(output)
  
      output= self.conv3(output)
      output= self.relu3(output)
  
      output= output.view(-1,120)
      output= self.fc1(output)
      output= self.relu4(output)
  
      output= self.fc2(output)
      F= self.softmax(output)
  
      # want to comput the jacobian dF/dimg 
      jacobian= computeJacobian(F,img)#how to write this function
  
      return F, jacobian
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T08:54:14.267Z
Comments.Text: do you want to implement this for training purpose ? because a lot of framework already do that.
Comments.UserDisplayName: Jérémy Blain
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T08:58:02.230Z
Comments.Text: No, not for training as I can do that using backward() methods. I want it during evaluation. I need to craft adversarial examples and for that, I need to modify a few pixels in the original image based on the Jacobian matrix.
Comments.UserDisplayName: Akhilesh Pandey
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T09:00:22.160Z
Comments.Text: When I say "training"" I was talking about you, train to write something in python. Because there are existing module in python, that does exactly what you want. Why reinventing the wheel ?
Comments.UserDisplayName: Jérémy Blain
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T09:03:43.663Z
Comments.Text: I want to compute the Jacobian matrix of the output layer F w.r.t each pixel in the input image. How to get that using modules? Also, the existing module does that using backward derivative whereas I want it using forward derivative.
Comments.UserDisplayName: Akhilesh Pandey
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T09:09:39.453Z
Comments.Text: Here is a module in python which implement a lot of attacks : https://foolbox.readthedocs.io/en/latest/user/installation.html You can find code which implements the JSMA attack (JacobianAttack). Myabe you can find something useful ? also worth to see : https://raghakot.github.io/keras-vis/ a module for Keras which implement saliency map creation (I know there are also other module for other framework like Tf, pytorch, but I haven't the links). Maybe you can use it, or look at the code to find ideas.
Comments.UserDisplayName: Jérémy Blain
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-20T10:14:40.017Z
Comments.Text: Can I achieve that using normal numpy libraries like `np.gradient()` or something like that?
Comments.UserDisplayName: Akhilesh Pandey
Tags: <deep-learning><feedforward-neural-networks>
Title: Compute Jacobian matrix of Deep learning model?
UserDisplayName: Akhilesh Pandey
ThreadText: # Compute Jacobian matrix of Deep learning model?
  
  (Asked by: username_0 on 2018-09-20)
  
  I am trying to implement this paper. In this paper, the author uses the forward derivative to compute the Jacobian matrix dF/dx using chain rule where F is the probability got from the last layer and X is input image.
  My model is given below. Kindly let me know how to go about doing that?
  class LeNet5(nn.Module):
  
  def __init__(self):
  
      self.derivative= None # store derivative
  
      super(LeNet5, self).__init__()
      self.conv1= nn.Conv2d(1,6,5)
      self.relu1= nn.ReLU()
      self.maxpool1= nn.MaxPool2d(2,2)
  
      self.conv2= nn.Conv2d(6,16,5)
      self.relu2= nn.ReLU()
      self.maxpool2= nn.MaxPool2d(2,2)
  
      self.conv3= nn.Conv2d(16,120,5)
      self.relu3= nn.ReLU()
  
      self.fc1= nn.Linear(120,84)
      self.relu4= nn.ReLU()
  
      self.fc2= nn.Linear(84,10)
      self.softmax= nn.Softmax(dim= -1)
  
  
  def forward(self,img, forward_derivative= False):
      output= self.conv1(img)
      output= self.relu1(output)
      output= self.maxpool1(output)
  
      output= self.conv2(output)
      output= self.relu2(output)
      output= self.maxpool2(output)
  
      output= self.conv3(output)
      output= self.relu3(output)
  
      output= output.view(-1,120)
      output= self.fc1(output)
      output= self.relu4(output)
  
      output= self.fc2(output)
      F= self.softmax(output)
  
      # want to comput the jacobian dF/dimg 
      jacobian= computeJacobian(F,img)#how to write this function
  
      return F, jacobian
  
  **Tags:** deep-learning, feedforward-neural-networks
  > **Comments:**
  > **username_1**: do you want to implement this for training purpose ? because a lot of framework already do that.  
  > **username_0**: No, not for training as I can do that using backward() methods. I want it during evaluation. I need to craft adversarial examples and for that, I need to modify a few pixels in the original image based on the Jacobian matrix.  
  > **username_1**: When I say "training"" I was talking about you, train to write something in python. Because there are existing module in python, that does exactly what you want. Why reinventing the wheel ?  
  > **username_0**: I want to compute the Jacobian matrix of the output layer F w.r.t each pixel in the input image. How to get that using modules? Also, the existing module does that using backward derivative whereas I want it using forward derivative.  
  > **username_1**: Here is a module in python which implement a lot of attacks : https://foolbox.readthedocs.io/en/latest/user/installation.html You can find code which implements the JSMA attack (JacobianAttack). Myabe you can find something useful ? also worth to see : https://raghakot.github.io/keras-vis/ a module for Keras which implement saliency map creation (I know there are also other module for other framework like Tf, pytorch, but I haven't the links). Maybe you can use it, or look at the code to find ideas.  
  > **username_0**: Can I achieve that using normal numpy libraries like `np.gradient()` or something like that?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-09-20. Score: 2
  
  In the paper [*The Limitations of Deep Learning in Adversarial Settings*, Papernot et. al., 2016, IEEE](https://arxiv.org/pdf/1511.07528.pdf) the chain rule is used, "To express $\nabla F(X∗)$ in terms of $X$ and constant values only."
  
  Earlier is stated, "Our understanding of how changes made to inputs affect a DNN’s output stems from the evaluation of the forward derivative: a matrix we introduce and define as the Jacobian of the function learned by the DNN. The forward derivative is used to construct adversarial saliency maps indicating input features to include in perturbation $\partial X$ in order to produce adversarial samples inducing a certain behavior from the DNN;"
  
  And later, "We define the forward derivative as the Jacobian matrix of the function F learned by the neural network during training. For this example, the output of F is one dimensional, the matrix is therefore reduced to a vector (below). Both components of this vector are computable using the adversary’s knowledge, and later we show how to compute this term efficiently."
  
  $\nabla F(X) = \Big[ \frac {\partial F(X)} {\partial X\_1}, \frac {\partial F(X)} {\partial X\_2} \Big] \quad\quad (2)$
  
  Between equation (2) and equation (6) your questeion is answered, resulting in an equation where the chain rule has already been applied.
  
  $\frac {\partial F\_j (X)} {\partial x\_i} = \Big(W\_{n+1,j} \, . \, \frac {\partial H\_n} {\partial x\_i} \Big) \times \frac {\partial f\_{n+1,j}} {\partial x\_i} \;(W\_{n+1,j} \, . \, H\_n + b\_{n+1,j}) \quad\quad(6)$
  
  It is (6) that you must implement in code, but that is done after the DNN is converged as the paper states above.
  
  **Applying the Chain Rule**
  
  If $z = f(y)$ and $y = g(x)$, then $\frac {dz} {dx} = f'(g(x)) g'(x)$
  
  Applied twice, if $w = e(z)$, $z = f(y)$, and $y = g(x)$, then $\frac {dw} {dx} = e'(f(g(x)) f'(g(x)) g'(x)$
  
  Applied indefinitely we multiply all the first order derivatives of intermediate values, which has the following first order predicate logic expression.
  
  $\forall \; 0 \le n \lt N \land v\_{n + 1} = f\_{n + 1}(v\_n) \land F\_{n+1}(x) = f\_{n+1}(F\_n(x)) \land g(a) = \dfrac {d} {d a} \, f(a)$
  $\exists \; g\_n(v\_0) = \prod\_{\,n = 1}^{\,n < N} {g\_n} (F\_n(v\_0))$
  
  Derivatives of convolution kernel values are the values themselves because they represent attenuation of the form $y = ax + b$. This is true of first degree linear activation functions too. The derivative normally used for ReLU (although the x = 0 case is actually undefined) is as follows:
  
  $\dfrac {d} {dx} R(x) = {\begin{cases}x < 0 & 0\\x = 0 & 1\\x > 0 & 1 \end{cases}}$
  
  The derivative of a max function used in max pool (although the u = v case is actually undefined) is as follows:
  
  $\dfrac {d} {dt} M(u, v) = {\begin{cases}u < v & \frac {dv} {dt}\\u = v & \frac {dv} {dt}\\u > v & \frac {du} {dt}\end{cases}}$
  
  > **Comments:**
  > **username_0**: Thanks for the answer. Once my model is trained i.e the DNN has converged, I want to compute equation (6) but I don't know how to do that when there are different types of layers like conv2d, linear, relu, maxpool etc. Could you please provide me with some simple example (in PyTorch) if possible? Thanks for your help once again.  
  > **username_0**: Thanks for the answer. It was really very helpful. However, as of now, I am able to compute the Jacobian w.r.t single X. Is it possible to extend, the same technique in the case where I am working in batches?  
  > **username_0**: Yes, I am referring to mini-batch.  
  
  ---
  
  