I cannot see any obvious errors in the code and would recommend to try to overfit a small data sample (e.g. just 10 samples) and make sure your new model is able to do so.
Based on the architecture, you might need to lower the learning rate for the common layers, as their gradients might be larger in their magnitude.