So there are two cases regarding the autograd graph:

If your operations to compute the loss are torch functions, the autograd engine can compute the backward just like in Python.
If, on the other hand, you implement custom computation kernels (as Peter does in the tutorial), you need to provide a backward. The old-style thing (shown in the tutorial) to do was to export both forward and backward and use an autograd.Function. These days, you would probably create a autograd::Function in C++, similar to what we do in TorchVision 22.

Another thing to keep in mind is that if you want JIT compatibility, you need to register operators (as in the custom op tutorial 16). Unless you need funny data types to be passed to from Python, there isn’t really a downside to using these.
Best regards
Thomas