Thank you for your time and answer but I think I forgot to mention a few details. Firstly, I trained the model with shuffled training set, only the validation and test set were giving me different results with and without shuffling the validation and test loaders. Secondly, my model has more than 97% accuracy on each individual class; it’s only when I change the batch size at test time and my test accuracy keeps  on changing and the model performs poorly with a batch size of 1 at test time.