Answers.Body: <p>I think what you mean to ask is <b>how can differentiation occur when there's no obvious neural network function to differentiate?</b></p>
  
  <p>Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation. </p>
  
  <p>First, take the output of all the neurons (except the one you're about to differentiate by) as a function:</p>
  
  <p><a href="https://i.sstatic.net/jIrA9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jIrA9.png" alt="enter image description here"></a></p>
  
  <p>The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.</p>
  
  <p>You then <b>differentiate NN by a single neuron (n) as shown:</b></p>
  
  <p><span class="math-container">$$\frac{\partial NN}{\partial n} = \lim_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$</span></p>
  
  <p>In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):</p>
  
  <p><span class="math-container">$$\frac{\partial A}{\partial n}$$</span></p>
  
  <p>So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.</p>
  
  <p>I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T17:16:58.063Z
Answers.Text: So, when we talk about the derivative of a NN w.r.t a weight, we are actually refering to the change in the output of the NN caused by the change of one neuron( to which that weight is associated). Can you explain a bit more why "In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n)"? Thanks.
Answers.UserDisplayName: qiang li
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T17:21:12.753Z
Answers.Text: @qiangli Sure- differentiating the activation function is a mathematical shortcut to computing the weight of each neuron. Instead of plugging everything into the neural net to iterate over again and again, we can activate the result and get a gradient per-neuron. This is possible through some matrix calculus and algebraic rearranging: it's all available online, but learning the rearrangement probably won't deepen your understanding.
Answers.UserDisplayName: mcRobusta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T14:13:25.160Z
Answers.LastActivityDate: 2019-09-13T22:04:39.023Z
Answers.LastEditDate: 2019-09-13T22:04:39.023Z
Answers.UserDisplayName: mcRobusta
Body: <p>Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a mathematical expression to work with, how can its derivatives be calculated?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T11:20:51.947Z
Comments.Text: What do you mean Neural Networks does not have a mathmatical expression to work with? Neural Networks is purely math in its nature?
Comments.UserDisplayName: SandMan
Tags: <neural-networks><reinforcement-learning><backpropagation><actor-critic-methods><policy-gradients>
Title: How can the derivative of a neural network be calculated, given no mathematical expression?
UserDisplayName: qiang li
ThreadText: # How can the derivative of a neural network be calculated, given no mathematical expression?
  
  (Asked by: username_0 on 2019-09-12)
  
  Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a mathematical expression to work with, how can its derivatives be calculated?
  
  **Tags:** neural-networks, reinforcement-learning, backpropagation, actor-critic-methods, policy-gradients
  
  > **Comments:**
  > **username_1**: What do you mean Neural Networks does not have a mathmatical expression to work with? Neural Networks is purely math in its nature?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-12. Score: 5 | Accepted answer
  
  I think what you mean to ask is **how can differentiation occur when there's no obvious neural network function to differentiate?**
  
  Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation.
  
  First, take the output of all the neurons (except the one you're about to differentiate by) as a function:
  
  [![enter image description here](https://i.sstatic.net/jIrA9.png)](https://i.sstatic.net/jIrA9.png)
  
  The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.
  
  You then **differentiate NN by a single neuron (n) as shown:**
  
  $$\frac{\partial NN}{\partial n} = \lim\_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$
  
  In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):
  
  $$\frac{\partial A}{\partial n}$$
  
  So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.
  
  I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.
  
  > **Comments:**
  > **username_0**: So, when we talk about the derivative of a NN w.r.t a weight, we are actually refering to the change in the output of the NN caused by the change of one neuron( to which that weight is associated). Can you explain a bit more why "In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n)"? Thanks.  
  > **username_2**: @username_0 Sure- differentiating the activation function is a mathematical shortcut to computing the weight of each neuron. Instead of plugging everything into the neural net to iterate over again and again, we can activate the result and get a gradient per-neuron. This is possible through some matrix calculus and algebraic rearranging: it's all available online, but learning the rearrangement probably won't deepen your understanding.  
  
  ---
  
  