and that hooked tensor will correspond to the specified layer (accessed by indexing or naming)?
So, by printing DataParallel model like above list(net.named_modules()), I will know indices of all layers including activations, which will be named with the following pattern (in this example, could be named differently elsewhere):
(2): ReLU(inplace)
(6): ReLU(inplace)
...
(27): ReLU(inplace)

Then by registering the hooks with right indices, using your method,  I can compute stats for activated layers.
Apologies for layman lingo, but I am not as technical as you are ;).
And big thanks for being patient with my questions