albanD:

you on


Yes, If I save checkpoint outside forward pass, checkpoint saves weights of single default device.
But, If I would like to save checkpoint inside forward, checkpoint saves weights of all GPUs (which are same) I believe it is issue in PyTorch.
If it is an issue, I would like to work on it and try to avoid duplicates weights inside the checkpoint.