Answers.Body: <p>So long as your introduced <em>leaf</em> tensors in <code>forward()</code> method support autograd (e.g., via attribute <code>requires_grad=True</code>), PyTorch will automatically handle BPTT via <code>backward()</code> method because it constructs a <em>dynamic computational graph</em> which builds nodes for all relevant tensors defined only from those gradient tracked leaf tensors during the forward pass. See yesterday's related <a href="https://ai.stackexchange.com/questions/47457/understanding-backward-pass-using-a-simple-computational-graph-implementation/">post</a>.</p>
  <p>Note in case your <span class="math-container">$T$</span> is large, storing intermediate tensors for BPTT can increase memory efficiency. For example, you can use <code>torch.utils.checkpoint</code> to recompute some intermediate states during BPTT to save memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T19:54:37.693Z
Answers.LastActivityDate: 2024-12-09T19:54:37.693Z
Answers.UserDisplayName: cinch
Body: <p>I am looking to implement the algorithms described by the paper <a href="https://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf" rel="nofollow noreferrer">Learning Fast Approximations of Sparse Coding</a> by Gregor and LeCun in PyTorch.</p>
  <p>My question stems from the algorithm LISTA::bprop.Suppose I write the iterative version of LISTA::fprop : <span class="math-container">$$\text{for}\ t = 1\ ...\ T:$$</span>
  <span class="math-container">$$Z(k+1) = h_\theta(W_eX + SZ(k)); Z(0) = 0$$</span>
  in the forward() method of my nn.The backpropagation step is naturally of the BPTT form. So  my question is that will PyTorch know to conduct BPTT automatically or do I have to code the backprop in too manually?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-09T21:57:05.907Z
Comments.Text: Questions about specific software are off-topic here. See https://ai.stackexchange.com/help/on-topic
Comments.UserDisplayName: nbro
Tags: <pytorch>
Title: Does BPTT happen automatically in Torch?
UserDisplayName: insipidintegrator
ThreadText: # Does BPTT happen automatically in Torch?
  
  (Asked by: username_0 on 2024-12-09)
  
  I am looking to implement the algorithms described by the paper [Learning Fast Approximations of Sparse Coding](https://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf) by Gregor and LeCun in PyTorch.
  
  My question stems from the algorithm LISTA::bprop.Suppose I write the iterative version of LISTA::fprop : $$\text{for}\ t = 1\ ...\ T:$$
  $$Z(k+1) = h\_\theta(W\_eX + SZ(k)); Z(0) = 0$$
  in the forward() method of my nn.The backpropagation step is naturally of the BPTT form. So my question is that will PyTorch know to conduct BPTT automatically or do I have to code the backprop in too manually?
  
  **Tags:** pytorch
  > **Comments:**
  > **username_1**: Questions about specific software are off-topic here. See https://ai.stackexchange.com/help/on-topic  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-09. Score: 1
  
  So long as your introduced *leaf* tensors in `forward()` method support autograd (e.g., via attribute `requires_grad=True`), PyTorch will automatically handle BPTT via `backward()` method because it constructs a *dynamic computational graph* which builds nodes for all relevant tensors defined only from those gradient tracked leaf tensors during the forward pass. See yesterday's related [post](https://ai.stackexchange.com/questions/47457/understanding-backward-pass-using-a-simple-computational-graph-implementation/).
  
  Note in case your $T$ is large, storing intermediate tensors for BPTT can increase memory efficiency. For example, you can use `torch.utils.checkpoint` to recompute some intermediate states during BPTT to save memory.
  
  
  ---
  
  