The the “fusion layer” your own implementation to concatenate the output of both images after passing them through a conv layer?
If so, could you check the statistics of these activations?
If they are largely different, you might need to normalize them before concatenating.