Thanks for responding @ptrblck.
There are other processes using the GPU. I’m seeing this error on a predictor service that has a 6 workers that receive prediction requests, preprocess the data, and then call the pytorch model. The workers are spun up via multiprocessing so I assume that that means each one has its own copy of the model.
Then I’m guessing that after making 6 copies of the model, most of the GPU memory is consumed. Then I see this error when a particularly large example is received by one of the workers.
Does that make sense? If so I will try reducing the number of workers to free up some of the GPU memory.