Is someone else using the GPU or does the server have different cards installed?
Could you explain a bit about the setup.
Might be unrelated here, but sometimes the device_ids don’t match with what nvidia_smi claims.