This issue occurs because cuda stores its cache files locally on ~/.nv , while I am using an nfs mounted home directory, somehow cuda throws an error. I fixed this error by setting CUDA_CACHE_PATH to some other local drive.