@Billy I’ve fixed it, but the debug info is deeply unhelpful and something is way off generally.
I had seen similar threads to the one you linked; they were where I got XLA_FLAGS suggestions & info.
Of course it’s all complicated by my OS being windows and the other discussions being linux oriented and the whole thing being at a technical level generally way over my head - I just want to code in Python.
However, the issue was resolved by providing the file (as a copy) at this path
C:\Users\Julian\anaconda3\envs\TF250_PY395_xeus\Library\bin\nvvm\libdevice\
Note that C:\Users\Julian\anaconda3\envs\TF250_PY395_xeus\Library\bin was the path given to XLA_FLAGS, but it seems it is not looking for the libdevice file there it is looking for the \nvvm\libdevice\ path This means that I can’t just set a different value in XLA_FLAGS to point to the actual location of the libdevice file because, to coin a phrase,  it’s not (just) the file it’s looking for.
This is really annoying as I will have to hand patch every environment I create… and I have yet to see whether this would also avoid libdevice issues with JAX as well
The debug info earlier:
2021-08-05 08:38:52.889213: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:73] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
2021-08-05 08:38:52.896033: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:74] Searched for CUDA in the following directories:
2021-08-05 08:38:52.899128: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:77]   C:/Users/Julian/anaconda3/envs/TF250_PY395_xeus/Library/bin
2021-08-05 08:38:52.902510: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:77]   C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2
2021-08-05 08:38:52.905815: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:77]   .

is incorrect insofar as there is no “CUDA” in the search path; and FWIW I think a different error should have been given for searching in C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2 since there is no such folder (there’s an old V10.0 folder there, but no OS install of CUDA 11)
There is a Windows environment variable CUDA_PATH that points to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0 but that is not referenced by any error messages either. The searched path  C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2 seems to be a guess, and as such a) unsuccessful b) unhelpful c) misleading
I note that the info also included the message “You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule’s DebugOptions” but I could not work out how to do that, and again, a level of technical complexity that seems inappropriate.
Rhetorical question: since TensorFlow generally is quite happy with the cuda installation in the (anaconda) environment, why is other stuff looking elsewhere in this idiosyncratic way?
So… you have some new input & 1 new question for sometime: are such libdevice issues something that will get sorted out (for the benefit of JAX too)?
Hope this has been helpful.
Cheers, Julian