Is your code running fine on the CPU?
This would probably give you a better error message than the illegal memory access.
Also, if you are using PyTorch 1.5.0, I would strongly recommend to update to 1.5.1, as 1.5.0 has a bug, where internal CUDA assert statements are skipped and will not raise a proper error but will run into other potentially issues (such as an illegal memory access).
If the code is running fine on the CPU and you are already using 1.5.1, rerun the code with
CUDA_LAUNCH_BLOCKING=1 python script.py args

and post the stack trace here, please.