Sorry for late appreciation.
I totally re-wrote my code and changed my algorithm, so I can reduce 11 Days -> 4 Days training time with my hooks.
But oscillating GPU Utilization, such as:
almost 100% GPU util(training process: like backpropagation)
-> 4~10% GPU util(my hooks; even I wrote all operations to be executed in GPU)
-> 100%…
(repeating)
This speed-difference between optimized training process and my custom task is still bottleneck.
I hope pytorch to have more powerful optimizing semantics on hooks, cause it’s very useful tool for researchers.
Thanks again for your answers.