Could it be something at the test-function?

Hm, yeah, could be. Sorry, but your code is a bit convoluted and hard to follow/read  . Or it could also be some issue with the dataset. Can you check the training accuracy and test accuracy? Before debugging the test function, I maybe there’s already something weird with the training going on that needs to be fixed first.