Martin,
it’s possible that these references to Variables are alive, but not in Python. These buffers can be of Functions who did save_for_backward of inputs which they need for gradient, and some Variable somewhere is alive in your code that is holding a reference to the graph that has all these buffer references alive.