Hi @albanD,
Appreciate your time. I use .data to cache the intermediate values without interfering with the way gradients are tracked in the first forward and backward pass. Then, I use those updated values for params2 and params3 as an initialization for the the second and third forward and backward passes. So I do need something like it. If there’s another, more up to date way, to copy the underlying data without interfering with the gradient tracking, please let me know.

In your example, if each loss is generated from a single param and is independent of the others, you can simply sum them up and do a single backward that will compute the right gradients.

Could you maybe elaborate on what you mean? Perhaps with a code example? It seems to me that each subsequent pass (pass two and three) is dependent on the values obtained from the first pass (please refer to previous comment in this response), meaning it can’t be parallelized. However, it is possible that I misunderstood what you said.