Sorry for not being clear enough.
What I meant is that your model architecture should generally work as you would expect.
As far as I understand your use case, you would like to predict bounding boxes using two different image modalities. I assume you only have one set of bounding boxes for each image pair, i.e. the boxes have the same location in the visible and IR image. Is that correct?
If so, all shared layers /Multispectral Pedestrian Detection layer, Multispectral Feature Fusion layer) will get gradients which will be calculated using a single loss based on both image inputs.
The Visible and Infrared Feature Extraction layers will also get valid gradients corresponding to the different image modalities.
What confuses me is the statement “each image for both datasets will have its own corresponding labels”. Could you explain it a bit? Are the bounding box locations different for the visible and IR image?
Also, let me know if I completely misunderstood your use case.