Cheers, so we’re unable to find any sort of pattern unfortunately.
We have taken PyTorch itself out of the equation, but reproducing the cuda availability issue in TensorFlow and also Blender.  I put a forum post 3 over at the Nvidia forums after this discovery)
I can confirm the machines are not in any hibernate status.  We’ve seen this error happen even when a user is logged into a Gnome session and using OpenGL/OpenCL software such as Autodesk Maya (nvidia-smi type C+G).  Rebooting the affected machine seems to fix.
It’s very weird.  Graphics work fine but CUDA intermittently drops out.