Hi,
Quite a few things happen before the start of the backward pass (discovering the graph and what needs to be computed in there). I don’t know what is the scale of the benchmark here, but for very small ops, this is possible.