Is the loss reducing in the same range for both approaches or is the second approach significantly worse regarding the loss?

The loss does reduces similarily in both approaches, however, when i use the feature fusion model, the loss goes to about 30% and does not seem to reduce much further, even when using a lower learning rate for an extended number of epochs. When i use the model without feature fusion, the loss goes down to around 10% in comparison with the same amout of epochs and learning rate.

Do you change anything else besides using the mentioned layer?

Nothing else is changed other than that layer.
I just can’t wrap my head around why the outcomes of the two models are so differnt. I would expect both models to have similar results, however, as i mentioned previously, the fusion-based model doesnt even produce prdictions unless the train loader has at least 1000 samples.