Decomposing language models into understandable components
I'm just curious, how polysemantic is the human brain with each neuron? Cause it feels to me, what you really want, and what the human brain might have, is a high-information (feature based / conceptual based / macro pattern based) monosemantic neural network, and where there is polysemantic neurons, they share similar or the same information in the feature it is a part of (leading to space efficiency? as well as computational efficiency). Whereas in transofmrer models like this, it's as if you're superimposing a million human brains on top of the same network, and then averaging out somehow all the features in the training set into unique neurons (leading naturally to a much larger "brain"). And also they mention in the paper that monosemantic neurons in the network don't work well, but my intuition would be because they are way too "high precision" and they aren't encoding enough information at the feature-level. Features are imo low dimensional, and then a monosemantic high dimensional neuron would the encode way too little information or something. But this is based on my lack of knowledge of the human brain so maybe there are way more similarities than I'm aware of...