Maybe you are not training long enough. Is your validation loss much higher than your training loss?
Maybe you do not have enough data
Maybe your dataset is very imbalanced
Maybe the problem is simply too hard and your labels are too similar
That is just a small range of learning rate. Try starting from 1e-03 and decrease until you see that your train/validation loss curve is looking promising
Use a lr scheduler