Yes, I delete all my install and build directories before compiling the library. I even started a new terminal to have a clean environment.
It seems that my export USE_CUDA=0 USE_CUDNN=0 have no effect. USE_CUDA is set to ON in the log summary.
--   USE_CUDA              : ON
--     Split CUDA          : OFF
--     CUDA static link    : OFF
--     USE_CUDNN           : ON
--     CUDA version        : 11.1
--     cuDNN version       : 8.2.4
--     CUDA root directory : /usr/local/cuda-11.1
--     CUDA library        : /usr/local/cuda-11.1/lib64/stubs/libcuda.so
--     cudart library      : /usr/local/cuda-11.1/lib64/libcudart.so
--     cublas library      : /usr/local/cuda-11.1/lib64/libcublas.so
--     cufft library       : /usr/local/cuda-11.1/lib64/libcufft.so
--     curand library      : /usr/local/cuda-11.1/lib64/libcurand.so
--     cuDNN library       : /usr/lib/x86_64-linux-gnu/libcudnn.so
--     nvrtc               : /usr/local/cuda-11.1/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda-11.1/include
--     NVCC executable     : /usr/local/cuda-11.1/bin/nvcc

Do I need to set any further parameters?