@ptrblck I found the issue using nvidia-smi; it was related to AWS sagemaker /jupyter notebook leaving orphaned processes on the GPU which was why there wasn’t enough memory available. Unsure why there were orphaned processes on the GPU.