Your model might be overfitting to the majority class and thus not learning anything useful.
Could you check your class distribution and see if the majority class occurs approx. In 84% of all cases?
Also check the prediction for unique values.
If you see that the class imbalance might be the reason for this issue, you could use a WeightedRandomSampler or pass class weights to your loss function to counter this effect.