summaries: This paper is about two things: understanding learned NLP representations by finding and understanding similarities between *different* representations; and comparing these representations to fMRI data. The paper takes a new approach to both problems, and finds a dimension in the space of language representations that seems to correspond to something like "semantic depth of analysis." Furthermore, the comparison with fMRI data shows that many representations are correlated with brain activations. Using a slightly modified (but relatively straightforward version) of an analysis that’s recently been called ‘neural taskonomy’, this work attempts to use the relationships (read: transfer learning affinity) among different natural language representations to assess the organization of linguistic representation in human fMRI data via regularized linear encoding models.  The authors first contribute a modified version of a previously used encoder-decoder framework to create a 'task' (linguistic representation) affinity matrix that demonstrates conspicuous, interpretable structure when reduced via MDS to fewer projections. They then use the structure discernible in this matrix to ascertain whether similar structure (specifically, a hierarchy of representation from word embeddings to later, deep transformer layers) is evident in the brain.  The submission proposes to study similarities between language representations in different language models. First, a similarity measure between language representations is derived. Then, the matrix of similarities is reduced to two dimensions using MDS. The authors then discuss qualitatively this low-dimensional space of language representations. Then, linear encoding models of fMRI recordings are used to project the first MDS dimension on the cortical surface of a subject listening to narratives. Finally, the matrix of similarities is used to predict the performances of linear encoding models.  * The authors propose "representation embeddings" (vectors describing transfer performance from task i to task j) as a low-dimensional space describing the content of language representations. * They analyze the derived representation embedding space and claim that the main principal component tracks "depth" (line 192) of linguistic processing. * They conduct a brain encoding task and evaluate 100 models on this task. They claim that a model's brain encoding performance tracks with its behavior in this low-dimensional representation embedding space, and that this low-dimensional behavior also maps intuitively onto the spatial representation of language in the brain.
abstractText: How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP (natural language processing) tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain’s natural language processing hierarchy. This suggests that the embedding captures some part of the brain’s natural language representation structure.
authors.name: Richard Antonello
authors.name: Javier Turek
references.authors: Alan Akbik Tanja Bergmann Duncan Blythe Kashif Rasul Stefan Schweter Roland Vollgraf
references.title: Flair: An easy-to-use framework for state-of-the-art nlp
references.venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations),
references.authors: Guillaume Alain Yoshua Bengio
references.title: Understanding intermediate layers using linear classifier probes
references.venue: arXiv preprint arXiv:1610.01644,
references.authors: Edward Awh John Jonides
references.title: Overlapping mechanisms of attention and spatial working memory
references.venue: Trends in Cognitive Sciences,
references.authors: Jeffrey R Binder Rutvik H Desai William W Graves Lisa L Conant
references.title: Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies
references.venue: Cerebral cortex,
references.authors: Charlotte Caucheteux Jean-Rémi King
references.title: Language processing in brains and deep neural networks: computational convergence and its limits
references.venue: bioRxiv,
references.authors: Zihang Dai Zhilin Yang Yiming Yang William W Cohen Jaime Carbonell Quoc V Le Ruslan Salakhutdinov
references.title: Transformer-xl: Attentive language models beyond a fixed-length context
references.authors: Wendy A de Heer Alexander G Huth Thomas L Griffiths Jack L Gallant Frédéric E Theunissen
references.title: The hierarchical cortical organization of human speech processing
references.venue: Journal of Neuroscience,
references.authors: Scott Deerwester Susan T Dumais George W Furnas Thomas K Landauer Richard Harshman
references.title: Indexing by latent semantic analysis
references.venue: Journal of the American society for information science,
references.authors: Christophe Destrieux Bruce Fischl Anders Dale Eric Halgren
references.title: Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.authors: Daniel B. Ehrlich Jasmine T. Stone David Brandfonbrener Alexander Atanasov John D. Murray
references.title: Psychrnn: An accessible and flexible python package for training recurrent neural network models on cognitive tasks. eNeuro
references.venue: doi: 10.1523/ENEURO.0427-20.2020. URL https://www.eneuro.org/content/8/1/ENEURO
references.authors: Kawin Ethayarajh
references.title: How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings
references.venue: arXiv preprint arXiv:1909.00512,
references.authors: Evelina Fedorenko John Duncan Nancy Kanwisher
references.title: Language-selective and domaingeneral regions lie side by side within broca’s area
references.venue: Current Biology,
references.authors: Evelina Fedorenko Idan Asher Blank Matthew Siegelman Zachary Mineroff
references.title: Lack of selectivity for syntax relative to word meanings throughout the language
references.venue: network. Cognition,
references.authors: Gregory Hickok David Poeppel
references.title: The cortical organization of speech processing
references.venue: Nature reviews neuroscience,
references.authors: Alexander G Huth Wendy A De Heer Thomas L Griffiths Frédéric E Theunissen Jack L Gallant
references.title: Natural speech reveals the semantic maps that tile human cerebral
references.venue: cortex. Nature,
references.authors: Shailee Jain Alexander Huth
references.title: Incorporating context into language encoding models for fmri
references.venue: Advances in Neural Information Processing Systems,
references.authors: Shailee Jain Vy Vo Shivangi Mahto Amanda LeBel Javier S Turek Alexander Huth
references.title: Interpretable multi-timescale models for predicting fmri responses to continuous natural speech
references.venue: Advances in Neural Information Processing Systems,
references.authors: Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey Hinton
references.title: Similarity of neural network representations revisited
references.venue: In International Conference on Machine Learning,
references.authors: Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut
references.title: Albert: A lite bert for self-supervised learning of language representations
references.authors: Yulia Lerner Christopher J Honey Lauren J Silbert Uri Hasson
references.title: Topographic mapping of a hierarchy of temporal receptive windows using a narrated story
references.venue: Journal of Neuroscience,
references.authors: Shinji Nishimoto An T Vu Thomas Naselaris Yuval Benjamini Bin Yu Jack L Gallant
references.title: Reconstructing visual experiences from brain activity evoked by natural movies
references.venue: Current biology,
references.authors: Shinji Nishimoto Alexander G Huth Natalia Y Bilenko Jack L Gallant
references.title: Eye movementinvariant representations in the human visual system
references.venue: Journal of vision,
references.authors: Jeffrey Pennington Richard Socher Christopher Manning
references.title: Glove: Global vectors for word representation
references.venue: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP),
references.authors: Tiago Pimentel Josef Valvoda Rowan Hall Maudslay Ran Zmigrod Adina Williams Ryan Cotterell
references.title: Information-theoretic probing for linguistic structure
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever
references.title: Language models are unsupervised multitask learners
references.venue: OpenAI Blog,
references.authors: R.W. Saaty
references.title: The analytic hierarchy process—what it is and how it is used
references.venue: Mathematical Modelling,
references.authors: Helmut Schmid
references.title: Part-of-speech tagging with neural networks
references.venue: arXiv preprint cmp-lg/9410018,
references.authors: Martin Schrimpf Idan Blank Greta Tuckute Carina Kauf Eghbal A. Hosseini Nancy Kanwisher Joshua Tenenbaum Evelina Fedorenko
references.title: The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing
references.venue: bioRxiv,
references.authors: Youwei Song Jiahai Wang Zhiwei Liang Zhiyue Liu Tao Jiang
references.title: Utilizing bert intermediate layers for aspect based sentiment analysis and natural language inference
references.authors: Jörg Tiedemann Santhosh Thottingal
references.title: OPUS-MT — Building open translation services for the World
references.venue: In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT),
references.authors: Mariya Toneva Leila Wehbe
references.title: Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
references.venue: Advances in Neural Information Processing Systems,
references.authors: Warren S Torgerson
references.title: Multidimensional scaling: I. theory and method
references.authors: Ben Tsuda Kay M. Tye Hava T. Siegelmann Terrence J. Sejnowski
references.title: A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Tu Vu Tong Wang Tsendsuren Munkhdalai Alessandro Sordoni Adam Trischler Andrew Mattarella-Micke Subhransu Maji Mohit Iyyer
references.title: Exploring and predicting transferability across NLP tasks
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882–7926,
references.authors: Vincent Walsh
references.title: A theory of magnitude: common cortical metrics of time, space and quantity
references.venue: Trends in Cognitive Sciences,
references.authors: Aria Wang Michael Tarr Leila Wehbe
references.title: Neural taskonomy: Inferring the similarity of task-derived representations from brain activity
references.venue: Advances in Neural Information Processing Systems,
references.authors: Leila Wehbe Brian Murphy Partha Talukdar Alona Fyshe Aaditya Ramdas Tom Mitchell
references.title: Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses
references.venue: PloS one,
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault R’emi Louf Morgan Funtowicz Jamie Brew
references.title: Huggingface’s transformers: State-of-the-art natural language processing
references.authors: Guangyu Robert Yang Madhura R. Joglekar H. Francis Song William T. Newsome Xiao- Jing Wang
references.title: Task representations in neural networks trained to perform many cognitive tasks
references.venue: Nature Neuroscience,
references.authors: An Gie Yong Sean Pearce
references.title: A beginner’s guide to factor analysis: Focusing on exploratory factor analysis
references.venue: Tutorials in quantitative methods for psychology,
references.authors: Amir R. Zamir Alexander Sax William Shen Leonidas J. Guibas Jitendra Malik Silvio Savarese
references.title: Taskonomy: Disentangling task transfer learning
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
sections.heading: 1 Introduction
sections.text: There are a multitude of common techniques for analytically representing the information contained in natural language. At the word level, language is often represented by word embeddings, which capture some aspects of word meaning using word co-occurrence statistics [8, 25]. Language representations that highlight specific linguistic properties, such as parts-of-speech [29] or sentence chunks [1], or that utilize well-known NLP models such as the intermediate layers of pretrained language models [6, 10, 21, 27], are also frequently studied [12, 31]. In fields such as linguistics, natural language processing, and cognitive neuroscience, qualitative adjectives are often used to describe these language representations – e.g. “low-level” or “high-level” and “syntactic” or “semantic”. The use of these words belies an unstated hypothesis about the nature of the space of language representations – namely that this space is fundamentally low-dimensional, and therefore that the information from the representations in this space can be efficiently described using a few categorical descriptors. In this work, we attempt to directly map the low-dimensional space of language representations by generating “representation embeddings” using a method inspired by the work of Zamir et al. [44]. This method uses the transfer properties between representations to map their relationships.
  The work described here has two main contributions. First, we used the representation embeddings to demonstrate the existence of low-dimensional structure within the space of language representations.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  We then used this structure to explore the relationships between – and gain deeper insight about – frequently used language representations. How do the intermediate layers of prominent language models relate to one another? How do the abstractions used by these layers evolve from low-level word embeddings of a context to a representation of the predicted next word for that context? Do different language models follow similar representation patterns? What are the differences between how unidirectional and bidirectional language models represent information? These are examples of the types of questions we can explore utilizing our representation embedding space. Second, we showed that this low-dimensional structure is reflected in brain responses predicted by these representation embeddings. In particular, we show that mapping the principal dimension of the representation embeddings onto the brain recovers, broadly, known language processing hierarchies. We also show that the representation embeddings can be used to predict which representations map well to each area in the brain.
sections.heading: 2 Related Work
sections.text: Our work closely follows the methods developed by Zamir et al. [44]. Those authors generated a taxonomy of vision tasks and analyzed their relationships in that space. This was achieved by applying transfer learning across vision tasks, and showed that this can reduce data labeling requirements. We used some of their methods, such as transfer modeling and normalization using an analytic hierarchy process (AHP) [28], for our analysis on language tasks. Following the same lines but for NLP, the work from Vu et al. [37] explores transferability across NLP tasks. Those authors focus on the application of transfer learning to show that the amount of labeled data, source and target tasks, and domain similarity are all relevant factors for the quality of transferability. Kornblith et al. [20] investigated and compared techniques for measuring the similarity of neural network representations, such as canonical correlation analysis (CCA) and centered kernel alignment (CKA).
  Our analysis of the relationship of between the language representation space and the brain leans heavily on the concept of “encoding models”. Recently, encoding models have been widely applied as predictive models of brain responses to natural language [17, 40]. Jain and Huth [18] applied encoding models to representations of an LSTM language model that incorporates context into neural representations. These representations are explored for different timescales of language processing in Jain et al. [19]. Toneva and Wehbe [33] explored the encoding model performance of different language model layers to improve neural network performance. Caucheteux and King [5] observed differences in the convergence of hidden state representations from ANNs trained on various visual and language processing tasks to brain-like representations. Schrimpf et al. [30] systematically examined the performance of encoding models across a variety of representations and neural response datasets and observed consistent high performance of Transformer-based model hidden states at predicting neural responses to language. Wang et al. [39] constructed encoding models on a set of 21 vision tasks from Zamir et al. [44] with the objective of localizing tasks to specific brain regions. Those authors built a task graph of brain regions showing similar representation for highly transferable tasks.
sections.heading: 3 Methods Overview
sections.text: Our method aims to use the transfer properties between different language representations to generate quantitative descriptions of their relationships. These descriptions can then be used as an embedding space without labelling any individual representation with predefined notions of what that representation is or how it relates to other representations. This is analogous to how word embeddings use co-occurrence statistics to generate quantitative descriptions of each word without labelling any individual word with our predefined notions of what that word means. By using transfer properties to generate the embedding for each representation, we implicitly assume that language representations with similar transfer properties are themselves similar.
  We define a representation t as a function over a language input that extracts a vector of meaningful features for each word. Some representations can be computed independently for each word, such as word embeddings [1, 10, 25], while others depend on both a word and its context, such as part of speech labels [1], or the hidden state in some intermediate layer of a language model [5, 6, 27, 30]. We define a stimulus s as a string of words drawn from some natural language source, and S the set of all stimuli from that source. To generate a representation embedding over a set of representations T that are derivable from the same stimuli S we will construct time-dependent mappings between
  representations. For a representation t ∈ T , and for all times j in our stimulus s ∈ S , the vector t(sj) is defined. Next, for each pair of representations t1, t2 ∈ T we will attempt to map t1(sj) to t2(sj). This provides a transfer mapping ft1→t2(·) that can be learned for each pair of representations,
  ft1→t2(t1(sj)) ≈ t2(sj) ∀j. However, learning these transfer mappings directly would make comparison across representations difficult, as many of the representations have different dimensionalities. Instead, we adapted the encoder-decoder framework proposed in Zamir et al. [44], which ensures that every transfer mapping has the same input dimensionality. This method proceeds in three parts (Figure 1):
  First, we train a linear encoder for each representation that compresses the information in the input down into a latent space. The latent space should only contain information necessary to predict the given representation’s features from our input. This is accomplished by training an encoder-decoder from the input to the given representation via a latent space, then discarding the decoder.
  Second, we train a new decoder for each pair of representations that uses the previously generated latent space for one representation to predict the other representation.
  Third, we evaluate the performance of each decoder targeting a given representation relative to every other decoder targeting that same representation. This provides a measure of how much of the information in one representation is present in every other representation. Just as words that have similar co-occurrence statistics—when measured against a large enough group of other words—are thought to have similar meanings, we argue that pairs of representations that have similar transfer properties—when measured against a large enough group of other representations—have similar information content.
sections.heading: 3.1 Generating Representation Encoders
sections.text: We define1 U(S) as the universal input feature space for our stimuli S . In practice, this space should represent the input with high fidelity, so could be selected as a one-hot encoding of each input token, or word embeddings for each input token. We use GloVe word embeddings for U(S). Now let T be a set of representations over S. For each representation t ∈ T , we generate an encoder Et(·) such that the encoder extracts only information in U(S) that is needed to predict t(S). We do this by using a bottlenecked linear neural network that maps every u ∈ U(S) to an intermediate low-dimensional latent space Lt = Et(U(S)) and then maps it to the given representation space,
  t(s) ≈ f(Et(u)) ∀s ∈ S ∧ u = U(s), where f(·) is mapping from Lt to t(S). We used a small latent space of 20 dimensions to encourage the encoder to extract only the information in U(S) that is relevant to compute t(S). Experimentation showed that variations in the latent space size do not meaningfully change encoder performance.
  1We assume that elements j in a string sj are properly treated, such that we can exclude the positional indexes for all s. Thus, we define U(S) = {U(s)|s ∈ S} as the result of applying a function U(·) to all elements s ∈ S .
  Once we have learned these mappings, we assign Et to be our representation encoder for each representation t ∈ T . Regardless of the dimensionality of the representation, each latent space has a fixed dimensionality, which enables a fair comparison between representations.
sections.heading: 3.2 Generating Representation Decoders
sections.text: The encoders for each representation generate a latent space Lt that extracts the information in U(S) relevant to computing t(S), while compressing away irrelevant information. For every pair of representations (t1, t2) ∈ T , we next generate a decoder Dt1→t2 such that Dt1→t2(Lt1) = Dt1→t2(Et1(U(S))) approximates t2(S). This yields a total of n2 decoders, where n = |T | is the total number of representations. All networks were trained with batches of size 1024 and standard stochastic gradient descent with a learning rate of 10−4 for the initial encoders and 2× 10−5 for the decoders. Hyperparameters were chosen via coordinate descent.
sections.heading: 3.3 Generating the Representation Embedding Matrix
sections.text: Finally, we compare the performance of the n decoders for each representation in order to generate the representation embedding space. To ensure that comparisons are made on equal footing, we only compare decoders with the same output representation (e.g. Dt1→t2 is compared with Dt3→t2 , Dt4→t2 , etc.). This is critical, because evaluating representations with varying dimensionalities would unfairly bias the results. This and related challenges were explored in detail in Zamir et al. [44].
  We used the decoders to generate a pairwise tournament matrix Wt for each representation t by “fighting” all pairs of decoders that output to representation t using a held-out test set Stest of sentences. Element (i, j) in Wt contains the ratio of samples in the test set for which Dti→t has lower MSE than Dtj→t, i.e., Wt(i,j) = Es∈Stest [Dti→t(s)<Dtj→t(s)] Es∈Stest [Dti→t(s)>Dtj→t(s)] . For example, if the decoder DA→C has lower mean squared error than decoder DB→C for 75% of the data in Stest, we assign the ratio of 0.75/0.25 = 3 to entry (A,B) in the tournament matrix WC for representation C.
  We then use this set of pairwise comparisons to approximate a total order over the quality of all encoders for each decoded representation. This was done using the Analytic Hierarchy Process (AHP) [28], a technique commonly used in operations research to convert many pairwise comparisons into an estimated total order. AHP establishes that the elements of the principal eigenvector of Wt constitutes a good candidate for a total order over the encoders for t. This eigenvector is proportional to the time that an infinite length random walk on the weighted bidirected graph induced by Wt will spend at any given representation. Thus, if the encoder for one representation is better than the others, the weight on that representation will be higher as compared to the weights of other encoders. This eigenvector is then normalized to sum to 1. This procedure yields a length n vector for each of the n representations. We set a value of 0.1 into each vector at the position corresponding to the target representation, and then stack the resulting length n vectors together into the representation embedding matrix, R.
sections.heading: 4 Results
sections.heading: 4.1 Language Representation Embeddings
sections.text: We applied our method to a set of 100 language representations taken from precomputed word embeddings or pretrained neural networks for NLP tasks operating on English language inputs.2 We extracted representations from numerous different tasks, which included three word embedding spaces (GloVe, BERT-E, FLAIR) [25, 10, 1], three unidirectional language models (GPT-2 Small, GPT-2 Medium, Transformer-XL) [27, 6, 41], two masked bidirectional language models (BERT, ALBERT) [10, 21], four common interpretable language tagging tasks (named entity recognition, partof-speech identification, sentence chunking, frame semantic parsing) [1], and two machine translation models (English→Mandarin, English→ German) [32]. We also included the GloVe embedding for the next word in the sequence, which constitutes a low-dimensional representation of the ideal output of a language model. For multilayered networks (language and translation models), we extracted all intermediate layers as separate representations. Full descriptions of all representations, as well as details on how each feature space is extracted, can be found in Appendix A.
  2Data and code for this paper are available at https://github.com/HuthLab/rep_structure.
  G lo Ve BE
  RT -E
  FL AI R PO S CH U N K N ER FR AM E N W E GPT-2 S GPT-2 M Trans-XL BERT ALBERT Eng Zh Eng De
  En g
  D e
  En g
  Zh AL
  BE RT
  BE RT
  Tr an
  sXL
  G PT
  -2 M
  G PT
  -2 S
  NWE FRAME
  NER CHUNK
  POS FLAIR
  BERT-E GloVe
  Encoded Feature Space
  De co
  de d
  Fe at
  ur e
  Sp ac
  e
  0.00 0.02 0.04 0.06 0.08 0.10
  GloVe
  BERT-E
  FLAIR
  Transformer-XL ALBERT
  BERT
  GPT-2 Small
  GPT-2 Medium
  POS
  CHUNK
  NERFRAME
  GloVe Next Word
  Englis h Ge
  rman
  English Mandarin
  0 4 8 12 16 Number of Factors
  10
  20
  % V
  ar . E
  xp la
  in ed
  Figure 2: Language Representation Embeddings with Low Dimensionality: (Left): The representation embedding matrix R shows how well a given linguistic feature space (encoder, columns) transfers to another feature space (decoder, rows). For better visualization, rows and columns corresponding to different layers from the same network have been scaled down in this plot. A full-scale matrix is in the supplementary material. (Right): Applying multi-dimensional scaling to the representation embedding matrix reveals low-dimensional structure in the linguistic feature spaces. It is dominated by a left-to-right progression from the input word embedding, to syntactic and semantic tagging tasks near the middle layers of language models, to the next word embedding. Multidimensional scaling was weighted such that each full model had equal weight, ensuring that language models were not more influential on account of having more layers. The dominant main diagonal was set to 0.1 to preserve the effects of off-diagonal values. The scree plot in the lower left shows that these first two dimensions explain substantially more variance (22% and 10%) than other dimensions, demonstrating that the structure in this space is low-dimensional.
  Applying our method to these representations, we generated a representation embedding space (Figure 2). Generating this space took roughly one week of compute time on a cluster of 53 GPUs (a combination of Titan X, Titan V, and Quadro RTXs) and 64 CPU servers of different characteristics (Intel3 Broadwell, Skylake, and Cascade Lake). The representations were trained on a text corpus of stories from The Moth Radio Hour [36] which totalled approximately 54,000 words. The held-out test set for the tournament matrix included a story of 1,839 words. Each row shows the relative performance of each of the 99 other representations in decoding a given representation. Each column shows how well a given encoded representation can decode to the other representations. Note that there is a distinct asymmetry: if a given representation (e.g. part-of-speech tagging) is never the best encoder for any other representation, then its column will contain very low values. However, other representations may decode to this one quite well, so the corresponding part-of-speech row can contain higher values.
sections.heading: 4.1.1 Multidimensional Scaling Analysis
sections.text: To better visualize the relationships between the representations, we performed multidimensional scaling (MDS) [34] on the rows of the representation embedding matrix. We then plotted the representations according to their locations on the first two MDS dimensions (Figure 2, right). This showed that the layers of each of the unidirectional language models form similar trajectories through the space, beginning near the word embeddings (green) and ending relatively close to the next-word embedding (black). This trajectory is monotonic on the first MDS dimension, but values on the second MDS dimension rise and then fall, with the highest values assigned to the middle layers (e.g. layer 5 of 12 in GPT-2 small). Language tagging tasks that require semantic information,
  3Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
  such as framing and named entity recognition (blue), are close in space to the middle layers of the unidirectional language models. Language tagging tasks that required syntactic information, such as part-of-speech identification and sentence chunking (red), are similarly clustered.
  We also observe several other interesting relationships. Like language models, the machine translation representations also begin near the word embeddings. However, translation to Mandarin (Eng→Zh) appears to align more closely to the unidirectional language models than translation from English to German. This may be because English and Mandarin are less closely related to one another, and could thus require deeper syntactic and semantic processing for successful translation.
  We also observe that the bidirectional language model representations begin farther away from the input word embedding, closer to the middle of the unidirectional language models. However, they also end up at a similar point, closer to the next-word embedding. This may be a result of the procedure to extract a feature space from the intermediate layers of the bidirectional language models, which differed meaningfully from extracting the intermediate layers of the unidirectional language models. Since these networks are trained on a masked language modeling task, we extracted the hidden representation for the token immediately preceding the mask token.
  To study the dimensionality of the space of language representations, we examined the variance explained by low-dimensional approximations of the embedding matrix. The scree plot in Figure 2 shows the result of applying exploratory factor analysis (EFA) [43] to the representation embedding matrix. We see that the first few dimensions explain much more variance in the matrix than the other dimensions. The first dimension alone constitutes 22% of the total variance. This demonstrates that the representation embedding matrix—and thus, we infer, the space of language representations—has low-dimensional structure.
sections.heading: 4.2 Representation Embeddings Predict fMRI Encoding Model Performance
sections.text: Next, we hypothesized that these language representation embeddings are a useful representation of how natural language is processed in the human brain. Since it is the only system truly capable of producing and understanding complex language, the human brain is an excellent litmus test to determine whether this low-dimensional embedding captures the underlying structure of linguistic information. To test this hypothesis, we used the representation embeddings generated above to predict how well each representation is able to predict fMRI data collected from the human brain as the subject is listening to natural language stories.
sections.heading: 4.2.1 fMRI Data
sections.text: We used functional magnetic resonance imaging (fMRI) data collected from 5 human subjects as they listened to English language podcast stories over Sensimetrics S14 headphones. Subjects were not asked to make any responses, but simply to listen attentively to the stories. For encoding model training, each subject listened to at approximately 5 hours of unique stories across 5 scanning sessions, yielding a total of 9,189 datapoints for each voxel across the whole brain. For model testing, the subjects listened to the same test story once in each session (i.e. 5 times). These responses were then averaged across repetitions. Training and test stimuli are listed in Appendix B.1. Functional signal-to-noise ratios in each voxel were computed using the mean-explainable variance method from Nishimoto et al. [24] on the repeated test data. Only voxels within 8 mm of the mid-cortical surface were analyzed, yielding roughly 90,000 voxels per subject.
  MRI data were collected on a 3T Siemens Skyra scanner at the University of Texas at Austin Biomedical Imaging Center using a 64-channel Siemens volume coil. Functional scans were collected using a gradient echo EPI sequence with repetition time (TR) = 2.00 s, echo time (TE) = 30.8 ms, flip angle = 71°, multi-band factor (simultaneous multi-slice) = 2, voxel size = 2.6mm x 2.6mm x 2.6mm (slice thickness = 2.6mm), matrix size = 84x84, and field of view = 220 mm. Anatomical data were collected using a T1-weighted multi-echo MP-RAGE sequence with voxel size = 1mm x 1mm x 1mm following the Freesurfer [15] morphometry protocol.
  Experiments were approved by the University of Texas at Austin IRB. All subjects gave written informed consent. Subjects were compensated for their time at a rate of $25 per hour, or $262 for the entire experiment. Compensation for the 5 subjects totaled $1260.
  Group Average per Anatomical ROI
sections.heading: 4.2.2 fMRI Encoding Models
sections.text: Encoding models predict a measured brain response B, e.g. blood-oxygen-level dependent (BOLD) responses recorded with fMRI, based on the stimulus observed by the subject. Due to limitations on dataset size, encoding models are typically structured as linearized regression, where each regression predictor is some feature of the stimulus. Encoding models can provide insight into where and how information is represented in the brain. For instance, an encoding model that has a feature for whether a given auditory stimulus is a person’s name can be used to determine which regions of the brain are activated by hearing names.
  Here we constructed voxelwise encoding models using ridge regression for each of the 100 language representations t analyzed above. Let g(ti) indicate a linearized ridge regression model that uses a temporally transformed version of the representation ti as predictors. The temporal transformation accounts for the lag in the hemodynamic response function [23, 17]. We use time delays of 2, 4, 6, and 8 seconds of the representation to generate this temporal transformation. For each subject x, voxel v, and representation ti, we fit a separate encoding model to predict the BOLD response B̂, i.e. B̂(x,v,ti) = g(x,v,ti)(ti). An optimal ridge parameter α was estimated for each g(x,v,ti) by using 50-fold Monte Carlo cross-validation, with a held-out validation set of 20% of the datapoints from the training dataset. We then measured encoding model performance ρ by computing the correlation between the true and predicted BOLD responses for a separate test dataset consisting of one story. That is, ρ(x,v,ti) measures the capacity of representation ti to explain the responses of voxel v in subject x.
sections.heading: 4.2.3 Mapping the First Dimension of the Representation Embedding Space to the Brain
sections.text: If a brain area was specialized for processing a specific type of information, then we would expect representations that capture that information to be good predictors of that brain area. Each voxel in the brain can thus be thought of as another language representation, and we can infer where that representation would lie in the MDS space by projecting it onto those dimensions.
  The main MDS dimension (left to right in Figure 2) is especially interesting as it seems to capture an intuitive notion of a language representation hierarchy. Representations with low values along the main dimension include word embeddings as well as the earliest layers of most of the language models and machine translation models. Representations with high values along the main dimension include the deeper layers of these models, as well as the majority of the interpretable syntactic and semantic representations. We tested whether this MDS dimension could capture patterns of hierarchical processing observed in cortex. First we defined p(x,v) = zscore([ρ(x,v,t1) . . . ρ(x,v,tn)]) as the 100-dimensional vector that denotes the encoding model performances for each subject x and voxel v, z-scored over all representations T . We then projected each p(x,v) onto the first MDS dimension by taking their dot product. This gave us a value that quantifies which part of the first MDS dimension best predicts the activity of a given voxel.
  Figure 3 shows shows the projection of each voxel for one subject (lower center) and averaged across subjects in each anatomical ROI (upper center) onto the first MDS dimension. Blue voxels and regions are better predicted by representations that are low on the first MDS dimension, whereas red voxels and regions are better predicted by representations that are high on the first MDS dimension.
  While the literature has not settled on a single hierarchical view of language processing in human cortex, one point of general agreement among existing theories is that there is a set of “lower” language areas, including Wernicke’s area/auditory cortex (AC), Broca’s area (BA), and the premotor speech area sPMv [13, 16]. Across our five subjects, we see negative projections on the first PC (corresponding to earlier LM layers) in AC, but both positive and negative projections in BA and sPMv. This matches other results using narrative stories and encoding models [7] where it was found that, of these three core language areas, AC is the best explained by lower-level features (phonemes and sound spectrum).
  Outside of the core language areas, the literature is more divided on representational hierarchy. It is broadly agreed upon that these other areas, including much of the temporal, parietal, and prefrontal cortex, constitute the “semantic system” [4] in which language meaning is derived and represented. Our data show that most of these regions have positive projections on the first PC, corresponding to later LM layers. More fine-grained analyses have suggested that some areas, such as the angular gyrus (AG) and precuneus (PrCu), contain the highest-level representations [19, 22]. This matches our group-level results, which show that the AG and PrCu have the most positive projections on the first PC of any brain areas.
sections.heading: 4.2.4 Predicting Encoding Model Performance with Representation Embeddings
sections.text: If the representation embeddings reflect how the brain represents linguistic information, then it should be possible to match each representation embedding rk to its corresponding performance vector ρ(x,•,tk), which describes how well that representation can predict responses in each cortical voxel. We test this via a leave-two-out experiment, where we train a learner using 98 (rk, ρ(x,•,tk)) pairs and then use this learner to predict which of the remaining two representation embeddings match the remaining two performance vectors. If the correlation between the predicted performance vector for representation A matches the ground truth performance vector for representation A better than the ground truth performance vector for representation B (and similarly B matches B better than it matches A), then we have succeeded in correctly discriminating a performance vector from its corresponding representation embedding.
  For each subject x and pair of language representations (ti, tj), we trained a linear regression ĥ(x,i,j) to map a representation embedding rk to the corresponding encoding model performance built from tk over all voxels.
  ĥ(x,ti,tj)(rk) ≈ corr(g(x,•,k)(tk), B(x,•)) = ρ(x,•,tk),
  rk is defined as the 196-element vector that concatenates the row and column that correspond to the representation ti, leaving out the four elements corresponding to the held out pair: Rii, Rij, Rji, and
  Rjj. The model ĥ(x,ti,tj) was then used to predict the held out encoding model performance across all voxels, ρ(x, •, ti) and ρ(x, •, tj). For each subject, we then computed discriminability scores for all pairs of representations to generate a matrix Mx. This quantifies how much better the predicted encoding model performance for ti matched with the true encoding model performance for ti than the true encoding model performance for tj (and vice versa):
  Mx(i,j) =(corr(ĥ(x,ti,tj)(ri), ρ(x,•,ti)) + (corr(ĥ(x,ti,tj)(rj), ρ(x,•,tj))))
  −(corr(ĥ(x,ti,tj)(ri), ρ(x,•,tj)) + (corr(ĥ(x,ti,tj)(rj), ρ(x,•,ti))))
  Figure 4 shows the discriminability scores for all representation pairs averaged across subjects, as well as the average encoding model performance for each individual representation. The vast majority of pairs have high discriminability scores (red), indicating that the representation embeddings broadly capture the structure of the brain’s language representations.
  However, a few pairs have negative scores (blue), indicating below-chance discrimination performance. This could happen by chance, or it could happen if the task affinities are very different from the brain representations. We believe the negative scores for GloVe NWE are caused by the latter. This representation is highly dissimilar from the others, hence its far-removed location in the MDS space. However, the temporal imprecision of the BOLD responses recorded by fMRI (on the order of 1 second, or 2-5 words) renders the GloVe NWE and normal GloVe embeddings nearly identical for predicting brain data. This disconnect is likely the cause of the negative scores.
  We also computed how often the discriminability score was higher than 0 in the majority of subjects (at least 3 out of 5). We see (Figure 4, upper right) that encoding model performances are well predicted by the representation embeddings, with representation embeddings getting matched to their corresponding encoding model performances over 90% of the time for all representations, and 100% of the time for the majority of representations.
sections.heading: 5 Future Work, Limitations, and Conclusions
sections.text: By measuring transferability between different language representations, we were able to visualize the relationships between 100 representations extracted from common NLP tasks. This uncovered a low-dimensional structure that illustrates how different tasks are related to one another, and how representations evolve over depth in different language models. Finally, we show evidence that these representation embeddings capture the structure of language representations in the human brain.
  There are further possible iterations of this method that may shed additional light on the structure of language representations in the brain. Since this method relies on using the transfer properties between representations to generate embeddings, it is necessary that the chosen set of representations spans the rich space of language representations. If some part of the space of language representations has not been sampled, then the representation embeddings may fail to capture some aspects of the structure of that space. We tried to ameliorate this concern by sampling a large number of representations (100). However, it is likely that these chosen representations still do not capture the full diversity of the space. In future work, an even larger number of representations could be used to better map this space.
  In a similar vein, we selected GloVe embeddings as the “universal” input representation for our encoder-decoder setup. GloVe embeddings are likely not “universal” in the same way that a one-hot encoding of the story might be, but practical concerns made this approximation necessary. Use of other universal spaces could also be enlightening, such as directly using fMRI data to map to a latent representation space—i.e. replace U(S) in Figure 1 with fMRI BOLD responses B recorded from the same natural language experiment. Of course, the possibilities are not restricted to language representations, as previous work had focused on vision instead [39, 44].
  We also used linear as opposed to nonlinear networks for our encoder-decoders. This limits the expressiveness of the transfer networks, and thus may under-estimate the relatedness between tasks. However, using only linear networks also increases interpretability by ensuring that only simplyrelated representations can transfer to one another, as also noted by others [2, 26]. Future work may explore the use of regularized nonlinear models for the encoder-decoder networks.
  This method does require extracting representations from a neural network trained to accomplish some task. For many tasks used in linguistics, neuroscience, and cognitive psychology, there is no existing pre-trained network. However, training artificial neural networks on a task that is used to measure cognition in humans and animals is an emerging method that can shed light on how these tasks are accomplished [11, 35, 42], and provide standardized benchmarks for comparing between different computational cognitive models. Furthermore, we can use our proposed method to understand what representations are needed to accomplish different tasks, and to visualize the relationships between those representations. For example, the debate on whether lexico-semantic and syntactic representations are actually separable [14] centers on evidence from different tasks that claim to measure syntactic representations versus semantic ones. The claims about the tasks themselves are difficult to test, but methods like those presented in this work can directly quantify how much information is shared between representations needed to accomplish those tasks, either in artificial neural networks or in neuroimaging data. Our proposed method may even provide new evidence in other domains where overlap has been observed between neural representations elicited by ostensibly different cognitive tasks [3, 38].
  In sum, we believe that our work can provide a template for investigating relationships between linguistic and cognitive representations in many different domains.
sections.heading: Acknowledgements
sections.text: We would like to thank Amanda LeBel for collecting the fMRI data and reconstructing cortical surfaces, Lauren Wagner for annotating the experimental stimuli, and the anonymous reviewers for their insights and suggestions. Funding for this work was provided by the Burroughs-Wellcome Fund
  Career Award at the Scientific Interface (CASI), Intel Corporation, the Whitehall Foundation, and the Alfred P. Sloan Foundation. The authors declare no conflicts of interest.
title: Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses