If each dataset has its own label tensor the concatenation seems to be wrong.
Currently you are fusing the activations of both images so that your model has only a single output layer. Could you explain your use case a bit?
Maybe processing both images separately without the fusion would be the right approach?