In DataParallel() with 2 GPUs.
If I save checkpoint inside forward pass, checkpoint saves the weights of both GPUs (these weights are the same). However, If I keep checkpoint outside of forward pass, checkpoint saves weights of default devices.
Which place is good to keep checkpoint, (inside forward pass or outside forward pass) ?