Yep,
the origin batch size is 2, and the error occurs at iter 422.
Now, I decrease batch size to 1, and the error occurs at iter 844
I set a breakpoint just before grid_sample in iter 844, at the breakpoint, GPU memory usage is 3083MB,
after execution of grid_sample(),
844it [05:01,  3.73it/s]THCudaCheck FAIL file=/pytorch/aten/src/THC/generated/../THCReduceAll.cuh line=317 error=77 : an illegal memory access was encountered
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered

and the memory usage is also 3083MB in nvidia-smi