The model ran for like 8 hours before the error occurred. I think I looked at the output of nvidia-smi only twice during that period and did not notice a change. I was using up this amount 10873MiB / 11178MiB. Then I came back many hours later and saw the memory error message.