Does the re-wrapping technique works well for you? I’d imagine it’s slow to use as wrapping with dataparallel each time copies ALL its weights to other GPUs, rather than copying just the ones needed when pruning the weights.