Thanks for your clear explanation!
So, could we say that we do the stateless way of forward-pass given parameters of a model, whenever we want to try different ways of optimization steps from the same starting parameter state (cloned for each individual path to go) so that they don’t “block” each other by their individual backward passes? or is there other important reasons for this way of forward pass?