IT should work as you’ve explained it.
Alternatively you could call backward on each of the losses separately and check the gradients of the shared CNN and both branched CNNs.
You’ll see that the gradients are calculated for the corresponding branch and accumulated for the shared CNN.