Thanks @JuanFMontesinos, ah, I had such a hard time making this test, you helped.
Here is what I have when using the function w.mean.
w = torch.randn(2, 5, requires_grad=True)
print(w)
r = w.mean()
r.backward(retain_graph=True)
print(w.grad)
# tensor([[ 0.6635, -0.3868,  2.0399,  0.3369, -0.5601],
#         [ 0.0737,  0.7711, -1.0142,  0.8241,  0.7964]], requires_grad=True)
# tensor([[0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000]])

I am afraid I cannot understand this result. Do we have in here the gradients accumulation?