It looks like your hook is saving a reference to the CUDA tensor in the list. You might want to get rid of that since memory is accumulating.
You also invoking another round trip to the GPU when you call output = output.to(device) which you then call back to the cpu. you can also get rid of that line.