paepcke:

    # Copy and save model copies before and
                # after back prop:
                before.append(copy.deepcopy(ddp_model))
                loss_fn(outputs, labels).backward()
                after.append(copy.deepcopy(ddp_model))



Hi, it seems like this is the essential portion of your code that saves params before/after backward. Although, simply calling backward() is not enough to modify the model parameters, you also need to call the optimizer.step() which will actually apply the averaged grads to the parameters.