It’s hard to infer what the issue is with just a description of the batch sizes; could you post a code snippet showing your data loading pipeline for the test set?
Looking at popular reference implementations (e.g., kuangliu/pytorch-cifar: 95.47% on CIFAR10 with PyTorch (github.com)) might also help.