Thanks for your advice, I run grid_flow in cpu mode with
new_heatmap = F.grid_sample(heatmap.cpu(), grid_flow.cpu() )

It will not cause any error on the same data which will cause CUDA Runtime error.