Could you check the bias values of the last linear layers?
Sometimes bad hyperparameters (e.g. high learning rate) throws the parameters out so that only the bias will be predicted (often the mean value of your target).