Backward is doing “reverse mode auto-differentiation” in mathematical terms.
You can follow this excellent tutorial by Andrej Karpathy 90 to get an intuition of what is going on.
And this blog post 22 from Baidu’s Silicon Valley AI Labs has very good figures as well.

image.jpg2012×762 262 KB

Finally, here is my own implementation of a scalar autograd 6, it’s just 3 files, less than 500 lines and it’s in Nim which has a syntax similar to Python but with types. It should be pretty readable. The backward proc 2 is just 30 lines.
The main difference with PyTorch implementation is that for this autograd I choose to return closures (i.e. function object) instead of saving state for the backward pass in an object with forward and backward method. spaCy backend “Thinc” is also implemented the same way. 1
To go a bit in the details with “sin” and “cos” example:
template bp_negate_sin[T](value: T): BackProp[T] =
  (gradient: T) => - gradient * value.sin()

template bp_cos[T](value: T): BackProp[T] =
  (gradient: T) => gradient * value.cos()

proc cos*[T](v: Variable[T]): Variable[T] =
  return Variable[T](
           tape: v.tape,
           value: v.value.cos(),
           index: v.tape.push_unary(v.index, bp_negate_sin(v.value))
           )

proc sin*[T](v: Variable[T]): Variable[T] =
  return Variable[T](
           tape: v.tape,
           value: v.value.sin(),
           index: v.tape.push_unary(v.index, bp_cos(v.value))
           )

I have a tape that records all operations during the forward pass. A value field that has the current value and an index of the operation in the tape. backward then just unwind the tape and call bp_cos (backprop_cos) when it finds its index.
Let’s take a binary operation now like -
proc bp_identity[T](gradient: T): T = gradient
proc bp_negate[T](gradient: T): T = -gradient

proc `-`*[T](lhs: Variable[T], rhs: Variable[T]): Variable[T] =
  return Variable[T](
           tape: lhs.tape,
           value: lhs.value - rhs.value,
           index: lhs.tape.push_binary(
             lhs.index, bp_identity[T],
             rhs.index, bp_negate[T]
             )
           )

In the tape I keep track of both the left-hand side and right-hand side operand which are respectively identity and negate.