The out of memory error might be wrongly returned and it seems your setup is unable to use the GPU.
Were you able to run PyTorch workloads on the GPU on this system before? If so, what did you change?
Also, which GPU, NVIDIA driver, OS etc. are you using?