Capture1502×1494 310 KB
Setup - Only for scenario explanation, any technical gaps please ignore :
Let’s say for a hypothetical robotic system where all the networks shown above are designed to handle different functionalities based on what it sees, i.e., Input for all the models is the  same (camera) and outputs are :
Out1: Object tracking
Out2: Ego Path estimation and Collision probability
Out3: Face recognition
Out4: Object recognition
Out5: Object segmentation
Opt1, Opt2, Opt3 are optimizers defined for NN1, NN2, NN3.
and Opt45 common for [NN4,NN5]
The assumption is all the networks have to be trained together with losses defined as shown (Just to get a holistic understanding of backprop.)
Note: NN1 and NN2 are inter-dependent on Out1 and Out2, but they don’t explore the same loss function (L1 != L2).
How can we achieve the backward prop shown in figure (in green), what should be the sequence of loss.backward() and optimizer.step() calls for each loss and optimizer?
L1.backward()
Opt1.step()

L2.backward()
Opt2.step()

L3.backward()
Opt3.step()

L45.backward()
Opt45.step()

will this achieve it? if yes is this the best way to do?
Also In this example Does L1.backward() update the graph only for NN1 and NN2?