Answers.Body: <p>The forward pass computes the actual values and sets up the computation graph, while the backward pass uses the precomputed <em>local</em> partial derivatives (the gradients with respect to each operand, for example, when you perform addition <span class="math-container">$a + b$</span>, its derivative w.r.t. <span class="math-container">$a$</span> and <span class="math-container">$b$</span> is <span class="math-container">$1$</span> which are stored) from the forward pass to apply the chain rule and compute gradients. Therefore the backward pass is not redoing the derivative calculations from scratch which allows you to compute gradients for complex computations in an efficient manner, leveraging the power of the chain rule and the precomputed derivatives.</p>
  <p>You can also further refer automatic differentiation's <a href="https://en.wikipedia.org/wiki/Automatic_differentiation#Reverse_accumulation" rel="nofollow noreferrer">reverse accumulation</a> mode which corresponds to your case for neural networks.</p>
  <blockquote>
  <p>Reverse accumulation traverses the chain rule from outside to inside, or in the case of the computational graph in Figure 3, from top to bottom... This is only half the work when compared to forward accumulation, but reverse accumulation requires the storage of the intermediate variables <span class="math-container">$w_i$</span> as well as the instructions that produced them in a data structure known as a &quot;tape&quot; or a Wengert list, which may consume significant memory if the computational graph is large.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T07:15:32.053Z
Answers.Text: So if we use something like PyTorch or TensorFlow, they implement computational graphs beneath their Matrix Multiplications somewhere? So each 'value' in a Matrix would correspond to a Node in the computational graph? I was able to create a computational graph and solve the XOR problem by just hard coding 8 parameters as nodes (2 x 2 matrix, 2 x 1 matrix and bias). The loss function is then just 'another node' added and we can call .backward from that node to compute gradients, then subtract them from parameter values.
Answers.UserDisplayName: Yuirike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T07:36:54.690Z
Answers.Text: Yes in frameworks like PyTorch and TensorFlow, computational graphs are constructed implicitly and dynamically (e.g., pytorch) under the hood during your defined operations like matrix multiplication. Your description of creating a computational graph by manually defining nodes as parameter values aligns with what these libraries automate. Calling backward() on the loss node traverses this graph in reverse, applying the chain rule to compute the gradients of each parameter node. Reverse mode AD with its graph is key for efficiently computing gradients across neural networks. Hope it clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T06:27:24.477Z
Answers.LastActivityDate: 2024-12-09T07:38:30.477Z
Answers.LastEditDate: 2024-12-09T07:38:30.477Z
Answers.UserDisplayName: cinch
Body: <p>I'm trying to understand backward pass with code by referring to a simple implementation of backward pass. The PyTorch one was too complicated for me as it has a bunch of other properties, so I tried to stick close to the theory. The class as well as <code>add</code> and <code>mul</code> was generated by ChatGPT. I altered it, implemented subtraction and division as well a sigmoid but it still seemed to work properly, so I got curious.</p>
  <pre><code>class Node:
      def __init__(self, value, parents=None, op=None):
          self.value = value               
          self.grad = 0                    
          self.parents = parents or []    
          self.op = op                      
  
      def backward(self, grad=1):
          self.grad += grad
  
          for parent, local_grad in self.parents:
              parent.backward(grad * local_grad)
  
  def add(a : Node, b : Node ) -&gt; Node:
      value = a.value + b.value
      node = Node(value, parents=[(a, 1), (b, 1)], op=&quot;add&quot;)
      return node
  
  
  def mul(a : Node, b : Node) -&gt; Node:
      value = a.value * b.value
      node = Node(value, parents=[(a, b.value), (b, a.value)], op=&quot;mul&quot;)
      return node
  </code></pre>
  <p>The actual code is longer but I want to focus on the <code>backward</code> function as well as how the gradient seems to be computed in the forward pass already.</p>
  <p>Observe how <code>add</code> and <code>mul</code> function create parents and their respective partial derivatives already. If you use this basic implementation to implement a forward pass and you call <code>backward</code>, you will apparently get the correct partial derivative for output with respect to each node.</p>
  <pre><code>def comp_graph_ex(a, b):
      node_a = Node(a)
      node_b = Node(b)
      node_1 = Node(1)
      node_c = add(node_a, node_b)
      node_d = add(node_b, node_1)
      node_e = mul(node_c, node_d)
     
      node_e.backward()
      print(f'∂e / ∂b = {node_b.grad}')
  
  if __name__ == &quot;__main__&quot;:
      comp_graph_ex(2, 1)
  </code></pre>
  <p>The graph in code corresponds to <a href="https://maucher.home.hdm-stuttgart.de/Pics/ComputeGraphBasic.png" rel="nofollow noreferrer">this one</a></p>
  <p>This will apparently compute: <span class="math-container">$\frac{\partial e}{\partial b} = \frac{\partial e}{\partial c}\cdot \frac{\partial c}{\partial b} + \frac{\partial e}{\partial d} \cdot \frac{\partial d}{\partial b} $</span> using the backward pass. (I'm not sure what this is called, I found 'product rule' but this would also work if <span class="math-container">$e=c-d$</span>, <span class="math-container">$e=\frac{c}{d}$</span>, <span class="math-container">$e=c+d$</span>.)</p>
  <p><strong>So does this mean, the backward pass isn't the one that computes derivatives but it's the forward pass?</strong></p>
  <p>Specifically, observe how each mathematical operation contains already the derivative, so when I compute <span class="math-container">$a+b$</span>, I know already that <span class="math-container">$\frac{\partial a}{\partial b}=\frac{\partial b}{\partial a}=1$</span>, hence the moment I perform that operation, I store those derivatives.</p>
  <p><strong>Does the backward pass use the precomputed values from the forward pass to employ chain rule?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-09T10:32:53.533Z
Comments.Text: Are you asking a question based on the assumption that code generated by an AI is correct?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-09T11:33:28.297Z
Comments.Text: Yes because I tried it out with different inputs and was able to get the same numbers when I solved for partial derivatives by hand on examples computational graphs. ChatGPT was trained on a bunch of data, so what it outputs is what it 'thinks' will most likely correspond to my prompt, so assuming it had a bunch of PyTorch/TensorFlow docs fed in at some point, it's not too far fetched to assume it's correct. I was just surprised to see derivatives being computed in the forward pass already, which I don't think it hallucinated.
Comments.UserDisplayName: Yuirike
Tags: <neural-networks><backpropagation><feedforward-neural-networks><forward-pass>
Title: Understanding Backward pass using a simple Computational Graph Implementation
UserDisplayName: Yuirike
ThreadText: # Understanding Backward pass using a simple Computational Graph Implementation
  
  (Asked by: username_0 on 2024-12-08)
  
  I'm trying to understand backward pass with code by referring to a simple implementation of backward pass. The PyTorch one was too complicated for me as it has a bunch of other properties, so I tried to stick close to the theory. The class as well as `add` and `mul` was generated by ChatGPT. I altered it, implemented subtraction and division as well a sigmoid but it still seemed to work properly, so I got curious.
  
  ```
  class Node:
      def __init__(self, value, parents=None, op=None):
          self.value = value               
          self.grad = 0                    
          self.parents = parents or []    
          self.op = op                      
  
      def backward(self, grad=1):
          self.grad += grad
  
          for parent, local_grad in self.parents:
              parent.backward(grad * local_grad)
  
  def add(a : Node, b : Node ) -> Node:
      value = a.value + b.value
      node = Node(value, parents=[(a, 1), (b, 1)], op="add")
      return node
  
  def mul(a : Node, b : Node) -> Node:
      value = a.value * b.value
      node = Node(value, parents=[(a, b.value), (b, a.value)], op="mul")
      return node
  
  ```
  
  The actual code is longer but I want to focus on the `backward` function as well as how the gradient seems to be computed in the forward pass already.
  
  Observe how `add` and `mul` function create parents and their respective partial derivatives already. If you use this basic implementation to implement a forward pass and you call `backward`, you will apparently get the correct partial derivative for output with respect to each node.
  
  ```
  def comp_graph_ex(a, b):
      node_a = Node(a)
      node_b = Node(b)
      node_1 = Node(1)
      node_c = add(node_a, node_b)
      node_d = add(node_b, node_1)
      node_e = mul(node_c, node_d)
     
      node_e.backward()
      print(f'∂e / ∂b = {node_b.grad}')
  
  if __name__ == "__main__":
      comp_graph_ex(2, 1)
  
  ```
  
  The graph in code corresponds to [this one](https://maucher.home.hdm-stuttgart.de/Pics/ComputeGraphBasic.png)
  
  This will apparently compute: $\frac{\partial e}{\partial b} = \frac{\partial e}{\partial c}\cdot \frac{\partial c}{\partial b} + \frac{\partial e}{\partial d} \cdot \frac{\partial d}{\partial b} $ using the backward pass. (I'm not sure what this is called, I found 'product rule' but this would also work if $e=c-d$, $e=\frac{c}{d}$, $e=c+d$.)
  
  **So does this mean, the backward pass isn't the one that computes derivatives but it's the forward pass?**
  
  Specifically, observe how each mathematical operation contains already the derivative, so when I compute $a+b$, I know already that $\frac{\partial a}{\partial b}=\frac{\partial b}{\partial a}=1$, hence the moment I perform that operation, I store those derivatives.
  
  **Does the backward pass use the precomputed values from the forward pass to employ chain rule?**
  
  **Tags:** neural-networks, backpropagation, feedforward-neural-networks, forward-pass
  
  > **Comments:**
  > **username_1**: Are you asking a question based on the assumption that code generated by an AI is correct?  
  > **username_0**: Yes because I tried it out with different inputs and was able to get the same numbers when I solved for partial derivatives by hand on examples computational graphs. ChatGPT was trained on a bunch of data, so what it outputs is what it 'thinks' will most likely correspond to my prompt, so assuming it had a bunch of PyTorch/TensorFlow docs fed in at some point, it's not too far fetched to assume it's correct. I was just surprised to see derivatives being computed in the forward pass already, which I don't think it hallucinated.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-09. Score: 1 | Accepted answer
  
  The forward pass computes the actual values and sets up the computation graph, while the backward pass uses the precomputed *local* partial derivatives (the gradients with respect to each operand, for example, when you perform addition $a + b$, its derivative w.r.t. $a$ and $b$ is $1$ which are stored) from the forward pass to apply the chain rule and compute gradients. Therefore the backward pass is not redoing the derivative calculations from scratch which allows you to compute gradients for complex computations in an efficient manner, leveraging the power of the chain rule and the precomputed derivatives.
  
  You can also further refer automatic differentiation's [reverse accumulation](https://en.wikipedia.org/wiki/Automatic_differentiation#Reverse_accumulation) mode which corresponds to your case for neural networks.
  
  > Reverse accumulation traverses the chain rule from outside to inside, or in the case of the computational graph in Figure 3, from top to bottom... This is only half the work when compared to forward accumulation, but reverse accumulation requires the storage of the intermediate variables $w\_i$ as well as the instructions that produced them in a data structure known as a "tape" or a Wengert list, which may consume significant memory if the computational graph is large.
  
  > **Comments:**
  > **username_0**: So if we use something like PyTorch or TensorFlow, they implement computational graphs beneath their Matrix Multiplications somewhere? So each 'value' in a Matrix would correspond to a Node in the computational graph? I was able to create a computational graph and solve the XOR problem by just hard coding 8 parameters as nodes (2 x 2 matrix, 2 x 1 matrix and bias). The loss function is then just 'another node' added and we can call .backward from that node to compute gradients, then subtract them from parameter values.  
  > **username_2**: Yes in frameworks like PyTorch and TensorFlow, computational graphs are constructed implicitly and dynamically (e.g., pytorch) under the hood during your defined operations like matrix multiplication. Your description of creating a computational graph by manually defining nodes as parameter values aligns with what these libraries automate. Calling backward() on the loss node traverses this graph in reverse, applying the chain rule to compute the gradients of each parameter node. Reverse mode AD with its graph is key for efficiently computing gradients across neural networks. Hope it clarifies.  
  
  ---
  
  