For just 10 samples, the loss should decrease basically to zero. Your model still seems to have some trouble learning the data. I would play around with some hyperparameters (learning rate) and the model architecture as the next step and force the model to learn this tiny data perfectly before digging any further.