I think what is happening is that you don’t detach the input of the main branch before you pass it to the aux branch. Then when you call backward for the loss from the aux branch, it will generate gradients for your main branch. At that point, you already generated gradients for the main branch, and DDP throws an error. The assertion error is bad and we should remove it (or at least update our expectations here), because the error message that will be shown next is:

  
      github.com
  
  
    pytorch/pytorch/blob/3593981976f9649c3e358c44c137b0c8b5a92980/torch/csrc/distributed/c10d/reducer.cpp#L287-L306 1

// Receiving a call to `mark_variable_ready` twice for the same variable
// is only possible if the variable was initially deemed unused, and was
// marked ready from the `prepare_for_backward` function, only to become
// part of the autograd graph at a later point in time.
TORCH_INTERNAL_ASSERT(has_marked_unused_parameters_);
TORCH_CHECK(
    false,
    "Expected to mark a variable ready only once. ",
    "",
    "This error is caused by use of a module parameter outside the ",
    "`forward` function. The return value of the `forward` function ",
    "is inspected by the distributed data parallel wrapper to figure ",
    "out if any of the module's parameters went unused. If this is the ",
    "case, it knows they won't receive gradients in a backward pass. ",
    "If any of those parameters are then used outside `forward`, this ",
    "error condition is triggered. ",
    "",
    "You can disable unused parameter detection by passing the keyword "
    "argument `find_unused_parameters=False` to ",
    "`torch.nn.parallel.DistributedDataParallel`.");



  
  
    
    
  
  


But even this error message assumes this case only happens if you use DDP with find_unused_parameters=True, which is not the case for you. You simply call backward twice and generate gradients more than once for at least one model parameter.