its convenient to use the output because sigmoid’s derivative is sigmoid(x)(1 - sigmoid(x))

yes the reason we can’t use in-place op is because you are modifying the saved variable in-place. If you do an out-of-place op here, yes the previous b is still saved in the computation graph, except now you are creating a new tensor object, and pointing b to the new tensor object. The saved variable shares storage with the old tensor object, so it is fine.