Hi Miturian!
First some contextual comments:
Pytorch’s autograd operates on tensor computations that produce
a scalar.  (Autograd can manage things slightly more general than
just a scalar result, but let’s leave that aside for this discussion.)
Leaf tensors are those that are inputs to the overall computation,
and can have their requires_grad property set to True or False.
Autograd computes the gradient of the final scalar result with respect
to (the elements of) the requires_grad = True leaf tensors.
Autograd neither knows nor cares that your computation might be
implemented as a pytorch Module that we conceptually understand
as a model, nor that one of its requires_grad = False leaf tensors
is, in the typical use case, the input to your model and represents a
batch of samples.  It neither knows nor cares that the weights (and
other Parameterss) of your model are its requires_grad = True
leaf tensors with respect to which it will calculate gradients.
This structure of applying a model with weights to a batch of samples
and then computing a scalar loss function for the output of the model
is higher-level structure that is distinct from autograd’s lower-level
functionality of computing gradients of tensor operations that may
or may not fit that higher-level structure.
(Autograd also neither knows nor cares that the gradients it computes
will typically be used by an optimizer to train the weights of a model.)
Having gotten all of that out of the way …



 miturian:

How does .backward() turn that into an average gradient?


At a mathematical level, the average of the gradients is the gradient
of the average.  That is, taking the gradient is a linear operation.
But the more computationally mechanistic answer is that the average
is computed by summing over a number of items and then multiplying
that sum by one over the number of items.  Autograd tracks the
“computation graph” of these operations and applies the chain rule to
them numerically.  The gradient (in this case, the simple derivative) of
constant times x (with respect to x) is just the constant.  Upstream of
that in the computation graph is that the gradient of sum of x_i (with
respect to the x_i) is just one for each of the x_i.


If my network was simply the operation “1/x”, with 10 different values of x in my minibatch,


This example doesn’t really have the structure of a the model with
weights applied to a batch of input samples.  (In your case, it sounds
like you want the gradient with respect to the “input minibatch.”  That’s
fine, but its not the typical use case.)
So let me phrase it a bit differently:
I have a 1d tensor, T, of shape [10] that consists of ten different values
of x (that I will consider a minibatch).  I have a tensor operation that
computes the (scalar) average of 1 / x for that set of ten xs.  We won’t
call this a “model” because it doesn’t really fit the typical model use case.
This scalar-valued tensor operation has a single leaf tensor, T, and it
has requires_grad = True.  Let’s use autograd to compute the gradient
of the scalar result with respect to T:
>>> import torch
>>> torch.__version__
'1.9.0'
>>> T = torch.arange (10.) + 1.
>>> T.requires_grad = True
>>> T
tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.], requires_grad=True)
>>> scalar_result = (1 / T).mean()
>>> scalar_result
tensor(0.2929, grad_fn=<MeanBackward0>)
>>> scalar_result.backward()
>>> T.grad
tensor([-0.1000, -0.0250, -0.0111, -0.0063, -0.0040, -0.0028, -0.0020, -0.0016,
        -0.0012, -0.0010])

The “magic” happens because the pytorch developers have implemented
a companion .backward() function for each (differentiable) pytorch
tensor operation that computes its gradient.  Autograd tracks how
a series of such tensor operations are composed together in its
“computation graph” and uses the chain rule to link together the
individual .backward() functions to compute the gradient of the
final scalar result with respect to the requires_grad = True leaf
tensors.
Best.
K. Frank