Thanks for the explanation!
In that case I’m not sure the fusion layer would be the best approach.
If your use case would just involve a single target, I think the fused layer approach could be a good idea (at least in my opinion  ).
Both feature extraction blocks would work separately on the two image modalities, and the fusion layers would use these features to learn the label and later the bounding boxes. Basically the “classifier” part (fused layers) would be able to select the necessary features of the feature extractors.
However, you are apparently dealing with separate labels and bounding box coordinates.
If you feed the features from the feature extractors to the fused block, you would have to separate them afterwards to use two (or four) different output layers (label + bbox for both modalities). Alternatively you could try to use a multi-label approach and use a single large output layer, which should learn to predict both labels, but I’m not sure if that’s the best way here.
Does it make sense to you or did I misunderstood something?