Joint backward and “piecewise” backward is the usually the same because the joint bits of the graph a) have the same input and b) are all linear in the gradient by the chain rule.
In @zeakey’s d(loss1)/db = dc/db * d(loss1)/dc ; d(loss2)/db = db/dc * d(loss2)/dc, because db/dc is the same, so you can add up the second factor.
Now if you do things in your backward that don’t follow the chain rule - e.g. Graves’-style gradient clipping during the backward - you might have different results between the two. Similarly, when you run into floats are not mathematicians’ real numbers things (NaN, infty, numerical precision), you will see differences.
The main reason to do things separately is probably memory limitations.
A similar insight - except that you split by batches and have (more or less - see batch norm) independent forwards - also powers data parallel.
Best regards
Thomas