Dear @ptrblck @blackberry, @albanD ,
I am in a similar situation. In the pseudo code below I try to do some multiple forward pass before computing the loss.
when grad == no. there is no gradient flowing in the second forward pass (see picture)
when grad == yes the gradient is let flow through the second dynamic graph.
I have observed differences in these two cases.
I went through many posts and cannot understand what is going on under the hood.
To my understanding when doing forward pass with gradient. 2 values per node(of the dynamic graph) are created. We have 2 backward graphs, but linked to the same model, thus
1- How are the weights updated these cases?
2- Same question as @blackberry:  are these approaches  equivalent to doing a single forward pass on the concatenated inputs and then doing loss.backward ?
Many thanks for your help!
Stefano
inputs_a
inputs_b
inputs_c
if args.GRAD == 'no':
    with torch.no_grad():
        output_a = model(input_a)
        output_b = model(input_b)
        output_c = model(input_c)
elif args.GRAD == 'yes':
    output_a = model(input_a)
    output_b = model(input_b)
    output_c = model(input_c)

Compute LOSSa,LOSSb,LOSSc
LOSS = LOSSa+LOSSb+LOSSc
optimizer.zero_grad()
loss.backward()
optimizer.step()


Blockquote

Picture 1874×475 118 KB