@stas ,
I saw your tweet and had two questions.


Does using deepspeed inference require consolidating the files in the checkpoint dir or it performs that for you and then reshards them across available GPUs?


Does this now support model > GPU RAM or is the comment in the doc 1 still accurate:



Additionally DeepSpeed is currently developing a related product called Deepspeed-Inference which has no relationship to the ZeRO technology, but instead uses tensor parallelism to scale models that can’t fit onto a single GPU. This is a work in progress and we will provide the integration once that product is complete.

In general I am trying to figure out if there exists a sharded checkpoint  format that allows either (a) inference or (b) resuming training on a different world size.