If you would like to create the confusion matrix for your whole validation dataset, you should update it batch wise. Currently you have these small numbers, since you’ve only passed a single batch to create the conf mat.
The stats (TP, TN, FP and FN) are created based on this description 125:



How to get the sensitivity and specificity of a dataset?



TP of class0 are all class0 samples classified as class0.
TN of class0 are all non-class0 samples classified as non-class0.
FP of class0 are all non-class0 samples classified as class0.
FN of class0 are all class0 samples not classified as class0.