That’s an interesting question. I think the answer is probably: it depends.
By default LMS tries to minimize what’s resident on the GPU; that’s controlled by the set_limit_lms tunable. The rationale for that is to favor “large model works at all” over best possible performance. Reasonable folks could prefer different defaults, so things might change in the future. Anyway, that conservative setting works in your favor for this use case.
On the other side, though, a couple things work against: Generally, all the tensors needed for a specific GPU operator have to be resident on the GPU while the operator is running. That means there’s some model-dependent minimum amount of GPU memory that will be needed.
And the current LMS decides where to move tensors (GPU or host memory) based only on the state of the single PyTorch instance. Multiple LMS-using PyTorch instances (containerized or not) wouldn’t coordinate. The LMS code doesn’t have any cross-instance distribution awareness. And it doesn’t, say, sense that GPU memory is nearly full and so try to move stuff to host memory. It’s currently just based on what tensors are needed by the activate operations and the tunable settings.
So LMS will allow you to over-subscribe, but for now you’ll have to do some experimentation to figure out how many instances will fit. And if you’re too agressive, yeah–allocation failures or insufficient resource are the likely failure modes.