Thank you for the reply!
Here is my very simple diagram of what I want to achieve:
split540×800 44.6 KB
I understand that identical subnets will forever be the same, so I’m going to add some noise to them to make them drift away from each other over time.
Also new weights don’t have to be 50% of original, they only need to sum to original.
But again this is a different story.
My question is is it possible somehow automatically (probably hacky) to detect indices in the next layer corresponding to the given dead/alive neurons in the current layer?
For a simple feed forward net like on the diagram it’s trivial, but in my model after relu I split/concatenate/sum tensors so it’s rather problematic to keep that knowledge “externally”.
I know Pytorch builds calculation graph as operations get applied to the tensors, is it possible to extract such kind of info from the graph?
Roman