Hi,
Hooks are very slow for multiple reasons: They are python code and cannot run concurrently (due to python limitations). They require the value of the Tensor usually and so will for the GPU to sync all computation (wait for everything to run, execute, then start queuing more work). Since multi-gpu splits a batch between the gpus, the whole batch runs as slow as the slowest part of the batch. In your case, that would be the GPU that has the hooks.
A possible solution could be to only collect these statistics every 10 batch?