Are you dealing with an imbalanced dataset?
If so, could you check, if your model simply outputs the majority class?
If that’s the case, you could use a weighted loss or WeightedRandomSampler to create balanced batches.
Have a look at the Wikipedia on Accuracy Paradox 19 for more information on this effect.