Thanks @albanD.
BTW I have been trying to figure out how exactly pytorch constructs the backward graph?
Is it appends to a tree data structure while going through the forward pass?
or does each tensor holds meta data about how they were originated (I guessing this could be grad_func) and from which tensors and backtracks until reaching a source node?