Hi,
I have a classification problem (similar to an NLP classification problem) with several different inputs, for the sake of simplicity I’m trying to handle two inputs right now.
First input is a string of values with constant length for all samples (it’s a nucleotide sequence, which could be looked at as a sentence in some language, consider it as a “complex input”).
The second input is a short list of int values describing the sample (in the NLP analogy it could be values describing “when the sentence was written” and “what is the name of the author”, etc… This is the “simple” input).
The model - I tried different architectures but in all of them the problem still persisted. In the latest architecture I used VDCNN (very deep CNN, which performed very well for an NLP problem) for the first input. For the second input I used a simple linear layer. Then I concatenated between the outputs and feed it to a final linear layer.
The problem - In all of the different architectures the model trained very fast up to 70% accuracy (and a certain loss value) but couldn’t get past this value. Then I tried training the model only on the first input by making the second input to be a constant - the model trained slowly to 60% accuracy, the important fact is that it trained. I tried training the model only with the second input and the model trained very fast to 70% accuracy (in a similar manner that it trained on both of the inputs).
The two inputs are definitely not redundant, meaning two of the inputs should contribute additively to the accuracy of the model.
What I would expect from the model is to train fast to 70% accuracy (learning features from the second input). And then continue improving slowly while learning from the first input.
But it seems that the model learns the most “simple” features from the second input and then completely ignores the first input and doesn’t learn any features from it.
Any ideas how to combat that ?