Thank you for the quick reply!
Since I got into this during inference in which no gradients are required, I found that other people 21 suggested to use with torch.no_grad(): before doing inference in the mini-batch. And it worked fine for me.