Thank you for such an elaborate response. I have learned something today.
I have the last question. I am not good at c++ but I don’t see the behavior you describe in code

One issue that arise here is that if you use optimized libraries like cudnn, they provide a single backward function for both input and weight.


  
      github.com
  
  
    pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp#L1178

  Tensor grad_input, grad_weight;
  if (input.numel() == 0) {
    if (output_mask[0]) {
      grad_input = at::empty_like(input, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
    }
    if (output_mask[1]) {
      grad_weight = at::zeros_like(weight, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
    }
  } else {
    if (output_mask[0]) {
      grad_input = at::cudnn_convolution_backward_input(input.sizes(), grad_output, weight, padding, stride, dilation, groups, benchmark, deterministic, allow_tf32);
    }
    if (output_mask[1]) {
      grad_weight = at::cudnn_convolution_backward_weight(weight.sizes(), grad_output, input, padding, stride, dilation, groups, benchmark, deterministic, allow_tf32);
    }
  }

  return std::tuple<Tensor,Tensor>{grad_input, grad_weight};
}

Tensor cudnn_convolution_transpose(



  
  
    
    
  
  


Did I search for it in the wrong place?