DistributedDataParallel currently does not work with checkpoint, see this discussion: https://github.com/pytorch/pytorch/issues/24005 25
This PR (https://github.com/pytorch/pytorch/pull/35083 20) fixes the error, but as under the new delay_allreduce mode, communication and computation no longer overlap, it is expected to be slower.
DataParallel might work with checkpoint, have you tried that?