Hi! This is possible. DDP relies on torch.autograd.backward to accumulate the gradients into the grad tensor of the model parameters. There is a functional alternative in torch.autograd.grad that doesn’t accumulate at all. If you’re interested in the local gradients, instead of running loss.backward(), you can run torch.autograd.grad(loss, model.parameters()) and get back a list of gradient tensors, one for every model parameter. This doesn’t accumulate them into the grad tensor of the model parameter, so it doesn’t kick off DDP. If you want to run DDP afterwards anyway, make sure to pass the retain_graph=True kwarg to torch.autograd.grad. I haven’t tried any of this out, but in theory it should work.