Yes.
The overhead of creating the graph is simply creating one cpp object (the Node), wrapping some Tensors (the one saved for the backward, creating the links to the previous Nodes and linking the output to the newly created Node.
Backprop need to traverse this whole graph to know which operations to perform (fairly expensive) and then computing the gradients themselves is roughly as expensive as the forward pass.