Thanks for your interest.
I tried applying dropout for the concatenated input as well and separately to each input with different dropout values, it still didn’t fix the issue.
Regarding the values - they indeed were high for the simple feature, to combat this I tried lowering the values and doing some kind of normalization of the raw values. Still, the model didn’t go past 70% accuracy.
Finally I tried the method of training separate models for each input and merging them later, I connected the two models by concatenating the outputs and feeding to a linear layer.
After a short period of training the model got to 70% accuracy again and didn’t improve, I checked the values of the of the outputs of the two models and it was very noticeable that the first output (from the complex features) had low values while the values from the model of “simple features” were high, meaning the combined model clearly “fixated” on the simple features.
Any additional ideas ?
I still think the two features are not redundant and both should contribute to the accuracy.