CUDA operations are executed asynchronously, so the illegal memory access error would most likely happen before you are calling empty_cache(). To isolate it, you could run the script via CUDA_LAUNCH_BLOCKING=1 python script.py args and check the stacktrace, which should point to the failing operation. Also, in case you are using an older PyTorch release, I would recommend to update it to the latest stable or nightly release.