For the buffer part. Every op will store anything that it will need for backward. For example the op a * b will need to store both a and b to be able to backward. These buffers are freed just after the backward call. To reduce memory usage by not keeping them around when they’re not needed. If you plan on calling this backward function twice, then you should tell the graph not to free them by using retain_graph so that the second call to backward still has the buffers.
In your case I agree that this should not happend with a single call to backward().
Your code is quite big so it may take a bit of time for us to check it. If you have a smaller version it would be welcome