It improves but it is still outrageously bad. I tried with 1e-5 (lr) and 10000 epochs and the loss decreases further.
But I would be surprised if it is normal to train so much on a single sample to obtain still bad, but slightly better, results