I’m not sure what “distributed” means in your use case.
Are you working with different models? If so, you could just save the state_dict of each model using a separate file.
Or are you working in a distributed setup, where the models are scattered and gathers using different nodes?
In that case, you could most likely want to reduce the model to the main node and just store this state_dict.