Looks like you’re right! For output_toeplitz() defined as above:
convtest = nn.Conv2d(1, 1, kernel_size=3, padding=1)
output_toeplitz = convmatrix2d(convtest.weight, [1, 5, 5])

for _ in range(3):
    t = torch.matmul(output_toeplitz,torch.randn(25,1))
    print("grad after matmul with output_toeplitz:")
    print(convtest.weight.grad) # None in 1st loop, zeros(3,3) afterwards
    t.sum().backward(retain_graph = True)
    print("grad after .backward():")
    print(convtest.weight.grad) # Nonzero updated gradient
    convtest.zero_grad()

A quick side question: why doesn’t this code work without retain_graph = True in backward()? I get RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed.