I tried for a slightly larger data set (50 samples), the loss still decreases and i still overfit (after around 25 epochs). But this is still only with model.train() and not with model.eval(). With model.eval() the accuracies never increases, even with dropping loss.
The loss:

The accuracies:

The testing stays same through entire training.
I will try using InstanceNorm() and see if that improves the issue.
Thanks!