Can you check if your model is not underfitting. Run your model for a small batch size and try to overfit it. If your model is not able to do so, that means the model does not have the power to generalize over the dataset.