This happens because you run the convmatrix2d only once for the 3 iterations of your loop. Which means that this part of the autograd graph is shared by all 3 iterations of the loop.
But if you don’t state retain_graph=True, the graph is freed during backward. And here, since you have a shared part in the graph, you don’t want it to be freed.
Also if you update the value of convtest.weight (via an optimizer for example), you want to recall convmatrix2d to make sure you reconstruct the matrix based on the new content of convtest.weight!