<p>So long as your introduced <em>leaf</em> tensors in <code>forward()</code> method support autograd (e.g., via attribute <code>requires_grad=True</code>), PyTorch will automatically handle BPTT via <code>backward()</code> method because it constructs a <em>dynamic computational graph</em> which builds nodes for all relevant tensors defined only from those gradient tracked leaf tensors during the forward pass. See yesterday's related <a href="https://ai.stackexchange.com/questions/47457/understanding-backward-pass-using-a-simple-computational-graph-implementation/">post</a>.</p>
<p>Note in case your <span class="math-container">$T$</span> is large, storing intermediate tensors for BPTT can increase memory efficiency. For example, you can use <code>torch.utils.checkpoint</code> to recompute some intermediate states during BPTT to save memory.</p>

CC BY-SA 4.0
2024-12-09T19:54:37.693Z
2024-12-09T19:54:37.693Z
cinch
<p>I am looking to implement the algorithms described by the paper <a href="https://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf" rel="nofollow noreferrer">Learning Fast Approximations of Sparse Coding</a> by Gregor and LeCun in PyTorch.</p>
<p>My question stems from the algorithm LISTA::bprop.Suppose I write the iterative version of LISTA::fprop : <span class="math-container">$$\text{for}\ t = 1\ ...\ T:$$</span>
<span class="math-container">$$Z(k+1) = h_\theta(W_eX + SZ(k)); Z(0) = 0$$</span>
in the forward() method of my nn.The backpropagation step is naturally of the BPTT form. So  my question is that will PyTorch know to conduct BPTT automatically or do I have to code the backprop in too manually?</p>

CC BY-SA 4.0
2024-12-09T21:57:05.907Z
Questions about specific software are off-topic here. See https://ai.stackexchange.com/help/on-topic
nbro
<pytorch>
Does BPTT happen automatically in Torch?
insipidintegrator
# Does BPTT happen automatically in Torch?

(Asked by: username_0 on 2024-12-09)

I am looking to implement the algorithms described by the paper [Learning Fast Approximations of Sparse Coding](https://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf) by Gregor and LeCun in PyTorch.

My question stems from the algorithm LISTA::bprop.Suppose I write the iterative version of LISTA::fprop : $$\text{for}\ t = 1\ ...\ T:$$
$$Z(k+1) = h\_\theta(W\_eX + SZ(k)); Z(0) = 0$$
in the forward() method of my nn.The backpropagation step is naturally of the BPTT form. So my question is that will PyTorch know to conduct BPTT automatically or do I have to code the backprop in too manually?

**Tags:** pytorch

> **Comments:**
> **username_1**: Questions about specific software are off-topic here. See https://ai.stackexchange.com/help/on-topic  

---

## 1 answers

### Answer by username_2 on 2024-12-09. Score: 1

So long as your introduced *leaf* tensors in `forward()` method support autograd (e.g., via attribute `requires_grad=True`), PyTorch will automatically handle BPTT via `backward()` method because it constructs a *dynamic computational graph* which builds nodes for all relevant tensors defined only from those gradient tracked leaf tensors during the forward pass. See yesterday's related [post](https://ai.stackexchange.com/questions/47457/understanding-backward-pass-using-a-simple-computational-graph-implementation/).

Note in case your $T$ is large, storing intermediate tensors for BPTT can increase memory efficiency. For example, you can use `torch.utils.checkpoint` to recompute some intermediate states during BPTT to save memory.


---

