I had to use a register_hook() on a parameter in a custom loss function to modify gradients when backpropagating. However I did not use hook.remove() at first and I noticed training slows down as the model iterates through batches and updates (backprop updates).
My question: what is the mechanism of a register_hook()? What does it pile up and where does it pile up? How does hook.remove() fixes this and what does it exactly free up?