I think, first you should start seeing whether training loss goes down / training accuracy goes up to know if network is learning.
Given that the network is learning, next, calculate the accuracy on test set (instead of seeing individual test prediction).
That should be the steps to do, IMHO.