Hm, I don’t think so? I call backward before the accumulation block. But because it is a generative (word-by-word) model, I might need to move the whole if do == "train" block in process to inside the for i in range(target_len): loop in step. Is that what you mean?
My thought process being that for a target_len t, we do t forward steps meaning that the returned loss contains a lot of information which may have lead to the increase in GPU memory.
I’m just not sure what the best practice is in a batched recurrent set-up and where to call backward when using accumulation.