It seems the output should be bounding boxes for pedestrians using both images as inputs.
In that case you would only have the bbox output and a single loss function.
The gradients in the shared layers will be accumulated.
I think your approach should be right.