Robert_Kudyba:

2021-08-20 15:21:38.393015: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_UNKNOWN: u
nknown error



I see that CUDA has failed to initialize. Your environment is not in good shape.
We had many CUDA setup issues in the repo like:

  

      github.com/tensorflow/tensorflow
  

  
    
  
	  
  

  
    
      "failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error" unless running with sudo 14
    

    
      
        opened 
        
          
        
        Sep 18, 2019
      
      

        
          closed 
        
          
        
        Sep 23, 2019
      
        

      
        
          
          josei
        
      
    

    
        stat:awaiting tensorflower
        type:build/install
        type:support
        comp:gpu
        TF 1.14
    
  


  
    **System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04…): ClearLinux 31030
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:  -
- TensorFlow installed from (source or binary): binary
- TensorFlow version: tensorflow-gpu 1.14.0
- Python version: 3.7.4
- Installed using virtualenv? pip? conda?: pyenv's pip
- Bazel version (if compiling from source): -
- GCC/Compiler version (if compiling from source): -
- CUDA/cuDNN version: cuda_10.0.130_410.48_linux, cudnn-10.0-linux-x64-v7.6.3.30
- GPU model and memory: Geforce RTX 2060, 6GB RAM


**Describe the problem**

The error "failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error" is thrown when initializing tensorflow-gpu, falling back to CPU instead of GPU.

When running python with sudo, GPU is detected but libraries cannot be opened. A subsequent run without sudo works, enabling GPU being used. I don't understand why running with sudo is needed to enable future calls without sudo work.

The specific output is:

    $ python -c "import tensorflow as tf; tf.Session(config=tf.ConfigProto(log_device_placement=True))"
    2019-09-18 17:49:26.342297: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
    2019-09-18 17:49:26.364789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3192000000 Hz
    2019-09-18 17:49:26.365673: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b4f7008c70 executing computations on platform Host. Devices:
    2019-09-18 17:49:26.365685: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
    2019-09-18 17:49:26.387440: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
    2019-09-18 17:49:26.397897: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
    2019-09-18 17:49:26.397918: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: linux
    2019-09-18 17:49:26.397923: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: linux
    2019-09-18 17:49:26.397950: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 430.50.0
    2019-09-18 17:49:26.397965: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 430.50.0
    2019-09-18 17:49:26.397969: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 430.50.0
    2019-09-18 17:49:26.399634: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
    Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device

    $ sudo python -c "import tensorflow as tf; tf.Session(config=tf.ConfigProto(log_device_placement=True))"
    2019-09-18 17:49:33.476640: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
    2019-09-18 17:49:33.492804: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3192000000 Hz
    2019-09-18 17:49:33.493345: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b8b43539a0 executing computations on platform Host. Devices:
    2019-09-18 17:49:33.493356: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
    2019-09-18 17:49:33.494037: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
    2019-09-18 17:49:33.525519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:33.525827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
    name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.755
    pciBusID: 0000:01:00.0
    2019-09-18 17:49:33.525900: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.525942: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.525980: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.526017: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.526055: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.526092: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.526130: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
    2019-09-18 17:49:33.526135: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...
    2019-09-18 17:49:33.614070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
    2019-09-18 17:49:33.614090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
    2019-09-18 17:49:33.614095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
    2019-09-18 17:49:33.615437: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:33.615752: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b8b71df500 executing computations on platform CUDA. Devices:
    2019-09-18 17:49:33.615761: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
    2019-09-18 17:49:33.616523: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
    /job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
    Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
    /job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device

    $ python -c "import tensorflow as tf; tf.Session(config=tf.ConfigProto(log_device_placement=True))"
    2019-09-18 17:49:38.343247: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
    2019-09-18 17:49:38.359840: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3192000000 Hz
    2019-09-18 17:49:38.360790: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d522b77c70 executing computations on platform Host. Devices:
    2019-09-18 17:49:38.360803: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
    2019-09-18 17:49:38.361478: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
    2019-09-18 17:49:38.377924: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.378224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
    name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.755
    pciBusID: 0000:01:00.0
    2019-09-18 17:49:38.382955: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-09-18 17:49:38.426461: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2019-09-18 17:49:38.452107: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
    2019-09-18 17:49:38.468904: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
    2019-09-18 17:49:38.517258: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
    2019-09-18 17:49:38.545852: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
    2019-09-18 17:49:38.660617: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2019-09-18 17:49:38.660684: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.661018: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.661283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
    2019-09-18 17:49:38.661304: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-09-18 17:49:38.727136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
    2019-09-18 17:49:38.727157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
    2019-09-18 17:49:38.727164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
    2019-09-18 17:49:38.727262: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.727564: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.727848: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-09-18 17:49:38.728115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5451 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
    2019-09-18 17:49:38.729363: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d525ddbe50 executing computations on platform CUDA. Devices:
    2019-09-18 17:49:38.729373: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
    2019-09-18 17:49:38.730199: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
    /job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5
    /job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
    Device mapping:
    /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
    /job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5
    /job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device

**Any other info / logs**

CUDA and Nvidia drivers installed at /opt following [ClearLinux guide](https://docs.01.org/clearlinux/latest/tutorials/nvidia.html).

    $ echo $LD_LIBRARY_PATH
    /usr/local/cuda/lib64:

    $ nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2018 NVIDIA Corporation
    Built on Sat_Aug_25_21:08:01_CDT_2018
    Cuda compilation tools, release 10.0, V10.0.130

    $ nvidia-smi
    Wed Sep 18 17:58:36 2019       
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  GeForce RTX 2060    Off  | 00000000:01:00.0  On |                  N/A |
    |  0%   50C    P8     9W / 170W |    152MiB /  5931MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    |    0       651      G   /usr/bin/X                                    39MiB |
    |    0       789      G   /usr/bin/gnome-shell                         111MiB |
    +-----------------------------------------------------------------------------+