Hi @tom , it took me some time to digest what you wrote, especially what you meant by another error from backprop inside the loop.
But my question still remains. Indeed, I’m exactly asking how should I program if (forward + backward + opt) can’t be done in one step – the good reason is for application consideration. In online learning I have to generate good prediction (hence the forward pass) in each step; in the problem I consider, I’d like to add a delay for the backward and optimization parts due to application reasons.
Let me give a concrete flow of I want. Let’s assume for simplicity the delay D=2. I’ll use
In[s] for input of step s, Out[s] for output of step s, and M_i for the i-th version of the model.

The initial model is M_0
Generate Out[1] = M_0(In[1])
Generate Out[2] = M_0(In[2])
Generate Out[3] = M_0(In[3])
Backward and optimize with Out[1] and In[1] to generate M_1
Generate Out[4] = M_1(In[4])
Backward and optimize with Out[2] and In[2] to generate M_2
Generate Out[5] = M_2(In[5])
…etc

Note that in my case, deep copying another M_0 and feeding in In[1] will not necessarily produce Out[1] as the procedure contains randomness, so that Out[1] is not reproducible basically.
What is the correct way to program this then?
Thanks in advance!