i think adding the CUDA_LAUNCH_BLOCKING=1 and everything working is probably a false-positive, where in that case random memory is just filled with zeros. Same with CPU (the illegal memory being hit is probably all zeros).
This is most definitely an index out of bounds issue (indexing with < 0 or > size ).
Can you run this repeatedly with CUDA_LAUNCH_BLOCKING=1 and see if you can get a good stack-trace to identify the location of your issue.