I set it before launching python (ie: in the env) and reported the setting in python just to be sure.
I do see a difference when it’s not set: the exception is raised a few instructions later. So it seems to be doing something. It consistently raised at the same point (with CUDA_LAUNCH_BLOCKING set) for several runs and with differing GPU loads.
I’m running pytorch 1.4.0 (old for compatibility with HPC infrastructure) and CUDA Version 10.2.89. These aren’t the most recent, but I put this here partly for others who may encounter this odd behaviour to help them decifer it