marcin:

and that hooked tensor will correspond to the specified layer (accessed by indexing or naming)?


Yes, the tensor in your hook function will correspond to the specified layer (accessed by indexing or naming).



 marcin:

So, by printing DataParallel model like above list(net.named_modules()) , I will know indices of all layers including activations


Yes, if the activations are created as modules. The alternative way would be to use the functional API for the activation functions, e.g. as done in DenseNet 2.
If you encounter such a model, you might want to override the forward method and store the activations manually or replace them with the corresponding modules.



 marcin:

Then by registering the hooks with right indices, using your method, I can compute stats for activated layers.


Yes, your code looks fine to achieve this.



 marcin:

Apologies for layman lingo, but I am not as technical as you are ;).
And big thanks for being patient with my questions 


Haha, no need to apologize. Your post are perfectly clear!