Based on the fact that the exact same code doesn’t have such a problem in pytorch 0.3.1. and I indeed used pinned memory, I think ngimel’s answer is probably right, although I haven’t tried to install 0.3.1 and revisit the problem. I will probably install 0.4 later. Thank you so much guys!