Could you try to apply Dropout on the concatenated feature layer?
Maybe this forces the model to “look” at both features.
Also, have you checked the value ranges of both inputs to the final layer?
Maybe the range of the VDCNN is quite small compared to the second input and looks like noise?