Hi,
Sorry for the delay.
I tried normalization of the raw input itself (not batch-norm, but normalization to bringing all values to a similar scale).
The first (complex) input is one hot vector (many zeros and a few ones) but the second input is is a bunch of floats with much larger range, I tried normalizing it by simply bringing the values to be between 0 to 1 or -1 to 1.
I also applied batch normalization to the features from the sub models and in between layers in the model.
I tried to train the combined model and the submodels with the same learning rate, tried lowering the learning rate to the sub models and tried completely freezing the submodels.
My colleague suggested to pre-train a model on the complex input, then “freeze” the weights to it and add the information of the second “simple” input and train the network very slowly.
I did some additional testing of (some different) submodels and I got those numbers:
We have one class that we are interested in it’s predictions so the accuracy is for that class;
First model for the complex input gets 46% of the samples classified correctly.
Second model for the simple input gets 54% of samples correctly.
Both models gets 37% of the samples classified correctly.
Meaning - there are 9% of samples that gets classified correctly by the first model but not the second
and there are 17% of samples classified correctly by the second model but not the first.
Regarding the inputs - I tried to take the outputs from softmax after both models (didn’t work), then I tried removing the softmax and got those values:
model 1 on complex input
Variable containing:
4.5105e-01  5.4895e-01
6.3620e-01  3.6380e-01
9.7610e-01  2.3901e-02
9.2572e-01  7.4278e-02
2.9720e-01  7.0280e-01
6.0810e-01  3.9190e-01
7.6838e-01  2.3162e-01
3.7808e-01  6.2192e-01
5.3675e-01  4.6325e-01
model 2 on simple input
Variable containing:
0.4752 -0.8468
-0.9316  0.6881
-0.4031  0.2606
0.2582 -0.1270
2.4042 -2.7847
-0.8459  0.6637
-0.3613  0.3322
-0.4756  0.5486
0.6900 -0.7750
After that I tried to delete the last FC layers of both sub models and concatenate the outputs, outputs looked like this:
model 1 on complex input
Variable containing:
4.7969e-04  4.7969e-04  4.7969e-04  …   4.7969e-04  4.7969e-04  4.7969e-04
4.8490e-04  4.8490e-04  4.8490e-04  …   4.8490e-04  4.8490e-04  4.8490e-04
3.1762e-04  3.1762e-04  3.3527e-04  …   3.1762e-04  3.1762e-04  3.1762e-04
…                   ⋱                   …
4.7948e-04  4.7948e-04  4.7948e-04  …   4.7948e-04  4.7948e-04  4.7948e-04
2.6548e-04  2.6548e-04  2.6548e-04  …   2.6548e-04  2.6548e-04  2.6548e-04
3.1191e-04  3.1191e-04  3.1191e-04  …   3.1191e-04  3.1191e-04  3.1191e-04
[torch.cuda.FloatTensor of size 256x2048 (GPU 0)]
model 2 on simple input
Variable containing:
1.00000e-02 *
0.0404  0.0404  0.0404  …   0.0431  0.0404  0.0521
0.0471  0.0418  0.0418  …   0.0418  0.0428  0.0675
0.0381  0.0381  0.0849  …   0.0381  0.0381  0.0425
…             ⋱             …
0.0371  0.0649  0.0371  …   0.0371  0.0594  0.0380
0.0542  0.0415  0.0415  …   0.0666  0.0577  0.0422
0.0189  0.0255  0.0255  …   0.0189  0.0683  0.0189
[torch.cuda.FloatTensor of size 256x2048 (GPU 0)]
I thought the the linear layers I put after concatenating the two outputs from the submodels would handle the different scale of values of the two submodels.