Brains and algorithms partially converge in natural language processing
<i>&gt; we analyze the brain activity of 102 healthy adults, recorded with both fMRI and source-localized magneto-encephalography (MEG). During these two 1 h-long sessions the subjects read isolated Dutch sentences composed of 9–15 words. After quantifying the signal-to-noise ratio of the brain responses, we train a variety of deep learning algorithms, extract their responses to the very same sentences and compare their ability to linearly map onto the fMRI and MEG brain recordings. Finally, we assess how the training, the architecture, and the word-prediction performance independently explains the brain-similarity of these algorithms and localize this convergence in both space and time.</i><p><i>&gt; We find that (1) a variety of deep learning algorithms linearly map onto the brain areas associated with reading, (2) the best brain-mapping are obtained from the middle layers of deep language models and, critically, we show that (3) whether an algorithm maps onto the brain primarily depends on its ability to predict words context</i><p>This is really cool research! Imagine wearing a VR like headset that records fMRI&#x2F;MEG signals and then instantly transcribes the words you’re thinking of into your text editor. Neuralink may have some competition eventually if these findings generalize and can be used in a wearable fashion?<p>On another note, I believe masked language modeling was originally proposed in the BERT paper (Devlin et al 2019), as a way to learn contextual word embeddings using a transformer architecture (ie autoencoder with denoising reconstruction objective).<p>This paper seems to suggest the brain may behave somewhat like a denoising autoencoder. If these correlations can be leveraged, the implications could be staggering.