Hey @zzzf,
Does this problem persistent, if you change the flow of fw(1)->fw(2)->fw(3)->bw(1)->bw(2)->bw(3) to fw([1, 2, 3]) -> bw([1, 2, 3])?
BTW, which version of PyTorch are you using? If it’s <=v1.6, I would expect it throws an error here:

  
      github.com
  
  
    pytorch/pytorch/blob/b31f58de6fa8bbda5353b3c77d9be4914399724d/torch/csrc/distributed/c10d/reducer.cpp#L724 2

// want to start performing reductions on `torch.autograd.backward()`.
void Reducer::prepare_for_backward(
    const std::vector<torch::autograd::Variable>& outputs) {
  std::lock_guard<std::mutex> lock(mutex_);
  std::unordered_set<torch::autograd::Node*> seen;
  std::vector<torch::autograd::Node*> queue;

  // Check that any prior reduction has finished.
  // The variable `require_finalize_` is true until all gradients
  // have been computed and reduction of all buckets has been kicked off.
  if (require_finalize_) {
    TORCH_CHECK(
        false,
        "Expected to have finished reduction in the prior iteration before ",
        "starting a new one. ",
        "",
        "This error indicates that your module has parameters that were ",
        "not used in producing loss. ",
        "",
        "You can enable unused parameter detection by (1) passing the keyword "
        "argument `find_unused_parameters=True` to ",