So isolate the illegal memory access, you could run the code via:
cuda-gdb --args python script.py args
...
run
...
bt

and post the backtrace here.
Also, I would recommend to update PyTorch to the latest stable release (1.7.1) or the nightly release, as you might be running into already fixes issues.