Anything in the ctx field is considered as any computation buffer. So this buffer will be freed when the graph is not needed anymore. In the context of gradcheck, we run multiple backwards with retain_graph=True such that the buffers are kept and we can call backward multiple times on the same graph.
I don’t understand why do you want to delete them yourself?