it cant be the case unless there is an issue with slurm that failed to clear previous job, i think.
the job runs on a cluster managed by slurm.
when asked for a gpu, only free gpus are allocated. once allocated, only that job is allowed to run on that gpu.
i’ll add nvidia-smi to the code to run at the beginning of the code and store it. hopefully to catch this error when it happens again.
this type of error does not happen frequently, but i got 4 jobs in a row failed because of this.
i had the same issue once. admins said that it could be a driver bug. i didnt get the follow up of their investigation.
this error happens only on clusters with slurm.
never happens on simple machine.
thanks