Your question is equivalent to asking why do we age; the list of answers is massive and they would all have some truth to them.
It could be your: dataset, test/train split, loss type, model, batch size, lr, optimizer, scheduler… etc.
Just to make sure, try out a commonly used architecture such as a Resnet18 and an optimizer like Adam. It’ll be a good indication.
At the end of the day, your model might simply be learning slowly. There’s no problem there.