Are you dealing with an imbalanced dataset?
If so, you could add a weight argument to your loss function or use a WeightedRandomSampler so that your model does not overfit on a specific class.
Also, try to overfit your model on a very small dataset (e.g. just 10 samples) and make sure the loss decreases towards zero.
If that’s not the case, you might have a bug in your code or the model architecture/hyperparameters are not suitable for this use case.