Try reducing your learning rate by different orders of magnitude. The loss may be “jumping around” a valley because the learning rate is too large. Try lr=1e-3, lr=1e-4, and so on; also try intermediate values such as lr=3e-5. Play around with the learning rate to see if you can hit a sweet spot.
If you see improvements in the loss when the learning rate is reduced, but you get tired of doing this by hand, you can also think of using learning rate schedulers of which PyTorch has many.