Yes, Deepspeed’s workflow is designed to be very fast and so yes ideally you want to be saving each node’s data locally and have the system deal with syncronization after the training moved on with its work.
For example at the current 104B GPT training at JeanZay/BigScience we save 1.4TB checkpoints which takes about 60secs. This is with 256 gpus and 64 nodes, and each process only writes its data locally to an SSD drive. The system itself uses GPFS over SSD discs to synchronize, but this is transparent to the training software.
And your use-case is much simpler since you don’t have multi-node. So yes, a local NVMe storage for intermediary checkpoints would be ideal.
But again practice with a tiny training (like 5 steps) and make sure you know how to extract the final weights so that you know it works for your needs.