But the difference of the loss is enlarged to be as large as 0.5 after merely 10 iters.
Actually, this implementation is  a simplified version of my code. In my original code, the loss can be enlarged to more than 1 in merely 2 iters. Besides, I noticed that, when I run both autograd and cuda implementations, the cuda implementation would have have a great bias, but if I run only the cuda implementations, the loss value and gradient is quite near the output of autograd implementation. What is likely to be the problem please ?
As for the output fo cuda-memcheck, each piece of the error messages looks similar, like this:
========= Program hit cudaErrorCudartUnloading (error 4) due to "driver shutting down" on CUDA API call to cudaFree. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x391b13]
=========     Host Frame:/miniconda/envs/py36/lib/python3.6/site-packages/torch/lib/../../../.././libcublasLt.so.10 [0x2a91e6]
=========     Host Frame:/miniconda/envs/py36/lib/python3.6/site-packages/torch/lib/../../../../libcublas.so.10 (cublasDestroy_v2 + 0xed) [0xb991d]
=========     Host Frame:/miniconda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch.so (cudnnDestroy + 0x219) [0x70fbc99]
=========     Host Frame:/miniconda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch.so [0x3ec9bad]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x39ff8]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x3a045]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf7) [0x20837]
=========     Host Frame:python [0x1c3160]


Any suggestions please ?