I am not sure why you would need 128 batch size input at prediction. Much of the time one would need to do prediction for a single sample (batch size of 1). Yes, some operations depend on the batch size like the BatchNorm, but during prediction, pytorch uses the running averages (computed during training) for the parameters for evaluation.