I would apply Dropout on the whole concatenated feature vector so that both inputs are mixed together.
Pre-training the models also sounds interesting.
However, it wouldn’t really explain the current issue.
Looking forward to hearing about the value ranges!