Our IT person installed 0.4.1 and the problem is still present.
He also has done some further testing. Here are his findings:

Some initial tests confirm the issue:


Running with the installed 0.4.0+CUDA 9.0.176 - problem


Running with the newly installed  0.4.1+CUDA 9.0.176 - problem


Running with Nvidia-built version in a container 0.5.0a0 + CUDA 9.0.176 - problem


Locally pip install 0.4.1 - still the same problem


I couldn’t right away build it from scratch, because we use gcc 6.3.0 compiler in the toolchains and there is a known issue with CUDA and one of the C++ libraries. I have made some custom fixes to get it tow work with TensorFlow and if all else fails, I can try with pytorch as well, but I want to try few other things first.


I have tested the hardware with a lot of TensorFlow benchmarks with pretty good results, but have not ran any pytorch benchmarks yet, so I will do that next, just to confirm that things are working. I do have to say that I am a bit puzzled though.

Thanks again for following-up!