Thank you for the rapid reply. Since .backward() can accumulate over the new gradient, I wonder if the param.grad can be allocated before calling .backward(), or before executing the execution_engine in the torch/autograd/init.py or  in torch/csrc/autograd 1/engine.cpp (func: THPEngine_run_backward)