Basically the idea is that the autograd engine needs to know every operations that you performed to be able to use their backward equivalent to during the backward pass.
The basic operation is an autograd.Function. for which both forward and backward are defined. They are quite hidden to the end user. For example, for the torch.checkpoint method, it is actually using a single Function here 3.
Assuming your Tensors require grads, when you apply a Function to a Tensor, it will record this and the output Tensor will have grad_fn field that says which Function was last applied to that Tensor. Similarly, the Function will look at it’s inputs and find out what were the previous functions created it own input. By doing this, you obtain a directed acyclic graph of Functions.
The cpp version works the exact same way where each output of a Function will link to that Function’s backward.
You can explore this graph this way:
import torch

a = torch.rand(10, 10, requires_grad=True)

out = a * 2
print(out.grad_fn) # MulBackward

loss = out.sum()
print(loss.grad_fn) # SumBackward
print(loss.grad_fn.next_functions) # ((MulBackward, 0),)
# Returns which functions corresponds to which input:
# Here only one input corresponds to MulBackward
# The 0 means that it was output 0 from this Function
# (mulitplication had a single output)

more_things = loss / out
print(more_things.grad_fn) # DivBackward
print(more_things.grad_fn.next_functions) # ((SumBackward, 0), (MulBackward, 0))
# Returns which functions corresponds to which input:
# First input loss corresponds to SumBackward
# Second input out corresponds to Mul Backward

As you can see, the autograd engine only kicks in at the Function level, and so for your original question, having more convoluted python logic will not change it’s behaviour (as long as you still perform the same operations on your Tensors).