Is there a principle to guide transfer learning across tasks in natural language processing (NLP)? Taxonomy (Zamir et al., 2018) finds that a structure exists among visual tasks, as a principle underlying transfer learning for them. In this paper, we propose a cognitively inspired framework, CogTaskonomy, to learn taxonomy for NLP tasks. The framework consists of Cognitive Representation Analytics (CRA) and Cognitive-Neural Mapping (CNM). The former employs Representational Similarity Analysis, which is commonly used in computational neuroscience to find a correlation between brain-activity measurement and computational modeling, to estimate task similarity with task-specific sentence representations. The latter learns to detect task relations by projecting neural representations from NLP models to cognitive signals (i.e., fMRI voxels). Experiments on 12 NLP tasks, where BERT/TinyBERT are used as the underlying models for transfer learning, demonstrate that the proposed CogTaskonomy is able to guide transfer learning, achieving performance competitive to the Analytic Hierarchy Process (Saaty, 1987) used in visual Taskonomy (Zamir et al., 2018) but without requiring exhaustive pairwise O ( m 2 ) task transferring. Analyses further discover that CNM is capable of learning model-agnostic task taxonomy. The source code is available at https://github.com/ tjunlp-lab/CogTaskonomy.git . Transfer learning (TL) has attracted extensive research interests in natural language processing with a wide range of forms, e.g., TL from pretrained language models (PLM) to downstream tasks (De-vlin et al., 2018; Radford et al., 2018), from a task with rich labeled data to a task with low resource (Chu and Wang, 2018; Yu et al., 2021), from high-resource languages to low-resource languages (Gu et al., 2018; Ko et al., 2021), etc. 1 A high-level concept or question on cross-task transfer learning is how these involved tasks are related to each other. Is sentiment analysis related to paraphrasing? Is textual entailment more related to question answering than named entity recognition? All these sub-questions resolve themselves into whether a structure exists among NLP tasks. Such task taxonomy is of notable values to transfer learning in NLP in that it has the potential to guide TL and reduce redundancies across tasks (Zamir et al., 2018). In this paper, partially inspired by the task taxonomy in visual tasks (Zamir et al., 2018), we study the hierarchical task structure for NLP tasks. But significantly different from the visual Taskonomy (Zamir et al., 2018), we construct NLP taskonomy from a cognitively inspired perspective. Cognitively inspired NLP is the intersection of NLP and cognitive neuroscience that aims at uncovering cognitive processes in the brain, including cognition in language comprehension. With the increasing availability of cognitively annotated data, on the one hand, cognitive processing signals (e.g., eye-tracking, EEG, fMRI) have been explored to enhance neural models for a wide range of NLP tasks (Barrett and Sgaard, 2015; Bingel et al., 2016; Hollenstein and Zhang, 2019; Hollenstein et al., 2019a). On the other hand, representations learned in NLP models are used to predict brain activation patterns recorded in cognitive processing data (Mitchell et al., 2008; Pereira et al., 2018; Hale et al., 2018; Hollenstein et al., 2019b). These studies on the bidirectional association between the two areas demonstrate that information underlying cognitive processing data is closely related to tasks and representations in NLP. Hence we want to know whether it is feasible to isolate task repre-1 In this paper, we focus on cross-task transfer learning in the same language. sentations from cognitive processing data and use them to learn task taxonomy in NLP. To examine this hypothesis, we propose CogTaskonomy, a Cog nitively Inspired Task Tax onomy framework, as illustrated in Figure 1, to learn a task structure for NLP tasks. CogTaskonomy consists of two main cognitively inspired components: Cognitive Representation Analytics (CRA) and Cognitive-Neural Mapping (CNM). CRA extracts task representations from NLP models and employs Representational Similarity Analysis (RSA) (Kriegeskorte et al., 2008), which is commonly used to measure the correlation between brain activity and computational model, to estimate NLP task similarity. CNM trains fully connected neural networks to build the mapping from sentence representations of pretrained models fine-tuned on specific tasks to fMRI signals recorded when human subjects read those sentences. It then uses mapping correlation coefficients as task representations to compute task similarity. Both methods require sentence representations to compute task representations. We use pretrained language models fine-tuned on specific tasks, particularly BERT (Devlin et al., 2018) and TinyBERT (Jiao et al., 2020), to obtain sentence representations. We compare the proposed CogTaskonomy against the Analytic Hierarchy Process (AHP) used in Taskonomy (Zamir et al., 2018). We guide TL across tasks with the learned task structure and evaluate the effectiveness of these methods by estimating TL performance from various source to target tasks. Contributions Our main contributions include:  We propose CogTaskonomy, a cognitively inspired framework to measure task similarity and to build the task taxonomy in NLP. This is the first attempt to study NLP task structures with cognitive processing data. We present two cognitively inspired methods, CRA and CNM, and compare them against AHP. Different from AHP, the two methods do not require O ( m 2 ) exhaustive pairwise transfer learning for task similarity estimation. We build a taxonomy tree for 12 NLP tasks, including sentiment analysis, question answering, natural language inference, semantic textual similarity, passage ranking, etc., to guide transfer learning across them. TL experiments and analyses validate the effectiveness of the proposed CogTaskonomy and find that CNM is able to learn stable task relations that are general to different underlying models. Our work is related to cognitively inspired NLP and a variety of learning formalisms that involve knowledge transfer across different tasks. We briefly review these topics within the scope of NLP and the constraint of space. Using NLP Representations for Brain Activity Prediction Since the pioneering work (Mitchell et al., 2008), connecting statistical NLP representations with cognition has attracted widespread at-905 tention. Chang et al. (2009) explore adjective-noun composition in fMRI based on co-occurrence statistics. Huth et al. (2016) use distributed word representations to map fMRI data to activated brain regions, revealing a semantic map of how words are distributed in the human cerebral cortex. A great deal of research (Murphy et al., 2012; Anderson et al., 2016; Sgaard, 2016; Bulat et al., 2017) has been devoted to word decoding. Pereira et al. (2018) extend brain decoding to sentence stimuli, suggesting that neural network language models can be used to interpret sentences in a long-term context. Ren and Xiong (2022) investigate the relationship between linguistic features and cognitive processing signals by developing a unified attentional network to bridge them. Augmenting NLP Models with Cognitive Processing Signals Recent years have witnessed that many efforts have been devoted to exploring cognitive processing signals (e.g., eye-tracking, EEG, fMRI) in neural NLP models. Muttenthaler et al. (2020) use cognitive data to regularize attention weights in NLP models. Hollenstein et al. (2019a) evaluate word embeddings using cognitive data. Toneva and Wehbe (2019) utilize fMRI scans to interpret and improve BERT. Many other works use cognitive processing signals to improve NLP models (Barrett and Sgaard, 2015; Bingel et al., 2016; Gauthier and Levy, 2019; Hollenstein and Zhang, 2019; Ren and Xiong, 2021), just to name a few. A very important trend in recent NLP is that models, algorithms, and solutions are not developed for only a single task, but for multiple tasks or across tasks (Devlin et al., 2018; Radford et al., 2018; McCann et al., 2018; Worsham and Kalita, 2020). Learning methods that are capable of handling a set of tasks simultaneously or sequentially, e.g., multi-task learning, transfer learning, meta learning, have attracted growing research interests in NLP. Beyond learning methods, yet another important dimension to this research trend is task relation learning, which is the topic of this work. 2 Multi-task Learning is to jointly train all tasks of interests with task linkages, e.g., in the form of regularization or sharing parameters across tasks 2 Task taxonomy learned by our methods could be applicable to other learning formalisms beyond transfer learning. We leave this to our future work. (Collobert et al., 2011). It is important in multitask learning to find related tasks for target tasks as auxiliary tasks (Ruder, 2017). Transfer Learning targets at transferring knowledge from a source task to a target task. According to the task and domain difference in the source and target, TL is divided into transductive TL (same task, different domain, a.k.a. domain adaptation), inductive TL (same domain, different task) and unsupervised TL (both different) (Eaton and des-Jardins, 2011; Ghifary et al., 2014; Wang et al., 2019; Yuan and Wen, 2021). If the source and target are dissimilar, negative transfer may hurt TL (Niu et al., 2020). Meta Learning aims to gain experience over a set of related tasks for improving the learning algorithm itself (Hospedales et al., 2020). Existing meta learning methods implicitly assume that tasks are similar to each other, but it is often unclear how to quantify task similarities and their roles in learning (Venkitaraman and Wahlberg, 2020). Lifelong Learning is to learn continuously and accumulate knowledge along a sequence of tasks and uses it for future learning (Chen and Liu, 2018). The system is tuned to be able to select the most related prior knowledge to bias the learning towards a new task favourably (Silver et al., 2013). As task relatedness is important for cross-task learning formalisms mentioned in Section 2.2, efforts have also been made to learn task relations. Craw-shaw (2020) groups previous methods on task relationship learning into three categories. The first is task grouping or clustering, which divides a set of tasks into clusters so that tasks in the same cluster can be jointly trained (Bingel and Sgaard, 2017; Standley et al., 2019). The second is learning transfer relationships, which analyzes whether transfer between tasks is beneficial to learning, regardless of whether tasks are related or not (Zamir et al., 2018; Dwivedi and Roig, 2019; Song et al., 2019). The third is task embedding, which learns a specific representation space for tasks (James et al., 2018; Lan et al., 2019). Our research can be considered as a mix of these categories. CNM learns cognition-based task representations while both CNM and CRA learn task relations aiming at transfer learning. Additionally, significantly different from previous studies, we 906 learn task structures from a cognitive perspective 3 , which is expected to estimate task relatedness in a cognitively tuned space. As will be demonstrated below, our cognitively motivated methods incur a low computation cost and exhibit generalization across underlying models to some extend. Figure 1 illustrates the basic framework of CogTaskonomy. First, we obtain task-specific sentence representations of text stimuli from cognitive data by feeding them into fine-tuned or distilled pretrained language models on 12 downstream tasks (Section 3.1). Subsequently, task-specific representations are fed into two cognitively inspired components, cognitive representation analytics (Section 3.2) and cognitive-neural mapping (Section 3.3), for estimating task similarity and inducing task taxonomy. Fine-tuning a pretrained language model for an end task is a widely used strategy for quickly and efficiently building a model for that task with limited labeled data. Zhou and Srikumar (2021) find that fine-tuning reconfigures underlying semantic space to adjust pretrained representations to downstream tasks. In view of this, we take sentence-level textual stimuli of cognitive data as input data for a specific fine-tuned model to obtain representations that contain information specific to that task. 4 Additionally, Cheng et al. (2020) suggest that knowledge distillation (KD) helps models to be more focused on task-relevant concepts. Therefore, without loss of generality, we use BERT and TinyBERT (performing KD) to obtain task-specific sentence representations. BERT Following Devlin et al. (2018), we prepend a special classification token [CLS] to each input sentence in order to extract the contextualized representation of the corresponding sentence. Merchant et al. (2020) find that fine-tuning primarily affects top layers of BERT. Hence, we take the hidden state of the prepended token of each sequence in the last layer as the sentence representation. 3 Dwivedi and Roig (2019) also use RSA to learn task taxonomy, in some way similar to our CRA. But they learn relations for visual tasks and use different correlation functions from our CRA. 4 Sentence-level textual stimuli of cognitive data refer to natural textual stimuli, i.e., sentences presented to subjects for collecting cognitive processing signals. TinyBERT TinyBERT (Jiao et al., 2020) performs knowledge distillation at both the pretraining and fine-tuning stage. By leveraging KD, TinyBERT learns to transfer knowledge encoded in the large teacher BERT (Devlin et al., 2018) to itself. As a result, TinyBERT can capture both general and task-specific knowledge. Similarly, we use the hidden state of [CLS] token in the last layer as the contextualized representation for a given sentence. With task-specific representations learned by feeding text stimuli of cognitive data into a fine-tuned model, we can estimate pairwise task similarity for any two tasks in a given task list T = { t 1 , t 2 , ..., t m } . The first cognitively inspired method is the cognitive representation analytics that adapts a common method in computational neuroscience to our scenario. We first briefly introduce the common method, representational similarity analysis, and then elaborate the adaptation. Representational Similarity Analysis is widely applied in cognitive neuroscience, which can not only realize cross-modal cognitive data comparison but also quantitatively relate brain activity measurements to computational models. It first calculates a representation dissimilarity matrix (RDM) of different modal data, and then estimates the correlation between RDMs. In this way, it successfully captures cross-modal data relationships (Kriegeskorte et al., 2008). RSA can be also applied for the comparison between computational models and cognitive data. The RDM of a computational model is obtained by comparing the dissimilarity of data representations obtained from the computational model in pairs. It is then compared with the RDM of brain activity measurements. 5 We take all sentence representations R i generated by a task-specific model PLMFT i (a pretrained language model (either BERT or TinyBERT) fine-tuned on the i th task) as the base to simulate cognitive representations required by RSA. For each pair of sentence representations ( R ij , R ij  ) for the j th and j  th sentence of the i th task, we compute a dissimilarity score in three metrics ( e ): Euclidean distance ( euclidean ), Canberra distance ( canberra ) and Pearson correlation coefficient (  ). Among 5 In our CRA, only RDMs from computational models are used. This is because we don't have cognitive data that are curated for specific NLP tasks. In our preliminary experiments, we have created pseudo cognitive data for different NLP tasks by predicting cognitive signals with a mapping model similar to that used in CNM. But it performs poorly. them, the first two distance metrics can naturally represent dissimilarity (Dis), while the last  needs to be converted to 1   to indicate dissimilarity, as follows: Dis ijj  = (cid:40) e ( R ij , R ij  ) e is not  1  e ( R ij , R ij  ) e is  (1) RDM for the i th task consists of the dissimilarity scores of all sentence pairs. We formulate it as follows: RDM i = [ Dis i 12 , Dis i 13 , . . . , Dis i 1 n , . . . Dis ijj  , . . . , Dis i ( n  1) n ] , j  = j  (2) where n is the number of sentences. RDMs computed in this way are then used for estimating similarity between NLP tasks. The pairwise similarity Sim ii  of the i th and i  th task is computed as follows: Sim ii  = Similarity ( RDM  i  RDM i  ) (3) Similarity (  ) is a similarity function, which can be Spearman rank correlation ( r s ),  and cosine ( cos , by default). In summary, we calculate the similarity between each RDM pair and finally obtain a similarity matrix for a set of tasks, as shown in Figure 2. 3.3 Cognitive-Neural Mapping The idea behind cognitive-neural mapping is to project sentence representations of NLP models fine-tuned in a specific task to cognitive signals (i.e., fMRI voxels in this paper) recorded when humans read those sentences with a neural network. The connections between the specific task and cognitive signals learned in this way could be transformed into cognitively inspired task representations for further task similarity estimation. The mapping can be considered as a way to isolate brain activity related to the specific task from fMRI cognitive signals. Particularly, for the i th task and s th subject, we use a fully connected 3-layer feed-forward neural network to project sentence representation R ij specific to this task to fMRI y isj of the s th subject reading the j th sentence as follows: y isj = W i 2 ( ReLU ( W i 1 ( R ij )) (4) To optimize the mapping model, we use the mean squared error (MSE) as loss function. 5-fold cross-validation is performed for each mapping model. Before training, grid search is conducted, and the optimal number of hidden layer units in the mapping network is obtained by three times of cross-validation on the verification set accounting for 20% training data. Each mapping is run 5 times. We average models over all subjects and 5 runs and then evaluate mapping model performance in all voxels. Particularly, we compute the cognitively inspired task representation CogR i for the i th task, which consists of the correlation coefficients on all voxels between predicted values and ground-truth values, defined as follows : CogR i =[ c ( y i 0 , y 0 ) , . . . , c ( y ik , y k ) , . . . , c ( y iv , y v )] , 0  k  v (5) where  y ik is a vector of all predicted values for the k th voxel from all input sentences by the mapping model tuned for the i th task, y k is a vector of the ground-truth values for the k th voxel from all sentence-level signals of text stimuli in fMRI data, v is the number of voxels used, and c (  ) is a function for comparing two input vectors. We instantiate c in two functions: the coefficient of determination ( R 2 ) and  . 6 We then use cosine similarity to calculate pairwise task similarity as follows: Sim ii  = cos( CogR  i  CogR i  ) (6) 4 Experiments We conducted experiments with widely-used NLP benchmark datasets and cognitive data to evaluate the effectiveness of CogTaskonomy. The brain fMRI dataset in our experiments is from Pereira et al. (2018), which is recorded on a whole-body 3-Tesla Siemens Trio scanner with a 32-channel head coil by showing 627 natural language sentences to 5 adult subjects. 7 Since voxels were 6 R 2 is a statistical measure that examines how much a model is able to predict or explain an outcome, usually defined as the square of the correlation between predicted values and actual values. According to the results in Appendix A.1, we set R 2 as c in CNM by default. 7 This dataset is publicly available at https://osf. randomly selected, Z-Score standardization was carried out for voxels obtained from different stimuli at each location on the basis of the original data set to avoid the influence of outliers. Subjects are asked to read each encyclopedic statement carefully, while the fMRI scanner records brain signals at this point. As a result, each fMRI scan covers multiple words at a time, subject to continuous stimulation. Each fMRI recording contains a number of voxels. We flattened 3d fMRI images into 1d vectors. v voxels were randomly selected, yielding matrices I s  R 627  v for each subject s . We selected 8 NLP tasks from the GLUE benchmark (Wang et al., 2018), including CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, STS-B. These tasks are considered important for generalizable natural language understanding, exhibiting diversity in domains, dataset sizes, and difficulties (Wang et al., 2018). To cover the spectrum of NLP tasks as much as possible, we also included Extractive Question Answering (QA), Relation Extraction (RE), Named Entity Recognition (NER), and Passage Reranking (PR). The datasets of these four tasks are SQuAD 2.0 (Rajpurkar et al., 2018), Semeval-2010 task 8 (Hendrickx et al., 2010), CoNLL 2003 (Sang and Meulder, 2003), MS MARCO (Nguyen et al., 2016; Craswell et al., 2020), respectively. We mainly used two methods as our baselines, including Direct Similarity Estimation (DSE), Analytic Hierarchy Process (AHP) (Zamir et al., 2018). Detailed experimental settings are shown in Appendix A.2. Direct Similarity Estimation (DSE) A straightforward way to estimate pairwise task similarity is to calculate sentence-level similarities based on task-specific sentence representations and then average them. Concretely, let R ij be the task-specific representation for the j th sentence in the i th task. The task similarity Sim ii  for a task pair ( i, i  ) is computed as follows: R ij = PLMFT i ( x j ) (7) Sim ii  = (cid:80) j Similarity ( R  ij  R i  j ) n (8) ipated in experiments 2 and 3 were chosen in this paper. where PLMFT i is the pretrained language model fine-tuned on the i th task. PLM can be instantiated as TinyBERT or BERT. Analytic Hierarchy Process (AHP) The main idea is to construct a matrix W t for each target task t , where the element at ( i , i  ) in the matrix shows how many times the i th source task is better than i  th source task in terms of the transferability to the target task on a held-out set. The principal eigenvector of W t is then taken as the task representation for the corresponding task, and all task representations are stacked up to obtain an affinity matrix. 8 The affinity matrix is then viewed as the task similarity matrix. Task Transferring To assess the similarity between tasks, all models fine-tuned on non-target tasks will be used as source models, and continue to be fine-tuned in the same way to transfer on the target task. In task transferring, all parameters of source models are fine-tuned (i.e., not fixed). We used the same learning rate and a number of training steps for all task transferring. This allows a fair comparison between different source tasks. Oracle Task Ranking The final similarity ranking of source tasks to a given target task is based on the results obtained from the task transferring experiments. Generally speaking, the better the source-to-target transfer performance is, the more similar the two tasks are, since the essence of TL is to apply knowledge learned in the source task to the target task. Based on this concept, we rank tasks in terms of transfer learning performance, for more details please see Appendix A.3. Task Ranking Score Based on similarity results computed by each task estimation method, we can obtain the most similar task for each target task. We then check the ranking position of the most similar task in the oracle task ranking. We average ranking positions of all target tasks as the final task ranking score for the corresponding task estimation method. Note that we exclude the transfer to the target task itself in computing task ranking scores. 9 8 For more details about AHP, please refer to (Zamir et al., 2018). Since the test sets of our NLP tasks are not publicly available, we obtain AHP results based on the validation set of each task except the NER task of which the test set is publicly available. In all experiments, the hyper-parameters are the same for all tasks. 9 Generally, the lower the task ranking score, the better the task similarity estimation method. Task ranking scores (using the ranking of task transferring as the oracle ranking) of different task similarity estimation methods are shown in Table 1. From these results, we have the following observations: observations:  Both CRA and CNM are better than random ranking and DSE, suggesting that cognitively inspired task similarity estimation is able to capture relations of NLP tasks. When TinyBERT is used, DSE is even worse than random ranking. This suggests that simply using task-specific sentence representations cannot well detect task relations and distinguish different tasks. TinyBERT performs better than BERT across three task estimation methods (i.e., CRA, CNM and AHP) although the number of parameters in the former is only half of that in the latter. We conjecture that TinyBERT uses knowledge distillation, making sentence representations more relevant to individual tasks and hence resulting in better task similarity estimation. We can also combine CRA and CNM (CRA+ CNM) by averaging task similarity scores estimated by them. Such combination is better than both methods alone. Although AHP is better than our methods, it directly uses the results of transfer learning to measure similarities between different tasks, which is very time-consuming. if we have m tasks, we have similarity method would yield a task ranking score of 1 on each target task. A random method would yield a ranking score of 0 . 5(1 + 11) = 6 in our experiments theoretically. We have also conducted random sampling 5000 times on TinyBERT and BERT, and obtained mean task ranking scores of 6.05 and 6.04 respectively. Hence, we take the 6 as the task ranking score for random ranking. to perform O ( m 2 ) transfer learning to obtain the task similarity matrix across all task pairs. In contrast, our methods do not require any costly transfer learning between tasks. It is hence easier to perform and able to guide transfer learning across tasks. We further evaluated the actual transfer learning performance of each target task from the most similar source task according to different task similarity estimation methods. Results are shown in Appendix A.4, which further validate the effectiveness of our methods and show that CRA+CNM is very close to that of AHP. In later experiments and analyses, we will show more advantages of our methods over AHP. CRA adopts RSA to transform the dissimilarity of task-specific sentence representations into the similarity of tasks. We have different options for dissimilarity measurement (e.g., euclidean , canberra ) in sentences and for similarity measurement (e.g., cos ,  ) in tasks. Hence we want to know the im-pact of the combinations of different measurements in sentence dissimilarity and task similarity on final performance. Results are provided in Table 2. Again, we have several interesting observations. First, with different combinations of these measurements, our CRA significantly outperforms random ranking in almost all cases. This suggests that RSA is able to be adapted to NLP task structure detection. Second, in comparison to the combination of  and r s in the original RSA (Kriegeskorte et al., 2008), in our case, the combination of  and cos is better than other combinations in the majority of cases. Third, TinyBERT is more robust to these 910 0 5000 10000 15000 20000 25000 30000 The Number of Voxels 4.5 5.0 5.5 6.0 6.5 T a s k R a nk i n g S c o re Random BERT TinyBERT Figure 3: Task ranking scores of CNM with TinyBERT and BERT predicting different numbers of voxels. different combinations than BERT. Since CNM bridges pretrained language models on the input side and voxels in fMRI images on the output side, we further evaluated CNM by varying the selection of PLMs (either BERT or TinyBERT) and the numbers of voxels. Results are displayed in Figure 3. It is interesting to find that with a small number of cognitive signals (voxels), TinyBERT for CNM can achieve a good task ranking score. By contrast, without sufficient cognitive signals, BERT for CNM fails in task similarity estimation, obtaining a task ranking score worse than random ranking. This is consistent with our previous finding in the main results that TinyBERT (with KD) captures more task-relevant knowledge than BERT for task relation detection. We conducted experiments to take a deep look into the feed-forward neural mapping model in CNM. The number of voxels was set to 30K. Pretrained Language Models We compared the prediction performance (measured by MSE between predicted results and ground-truth voxels) across different tasks using BERT vs. TinyBERT as the pretrained language model to obtain task-specific sentence representations. Results are shown in Figure 4(a). We can clearly see that both BERT and TinyBERT are better than the random baseline across all tasks. And TinyBERT is better than BERT on all tasks, which resonates with the main results shown in Section 4.5. Subjects We analyzed prediction performance across different subjects, as shown in Figure 4(b). Although the prediction performance varies across different tasks, the shapes of the prediction perfor-(a) MSEs (averaged over 5 subjects and 30K voxels) for different tasks with BERT vs. TinyBERT being used as the pretrained language model. mance curve over 12 tasks for different subjects are similar to each other, indicating that similar brain activities are activated for these tasks across different subjects. Models underlying our cross-task transfer learning are different pretrained language models, which is a widely acknowledged practice for transfer learning in NLP. We therefore want to investigate how general our task similarity estimation methods (e.g., CNM, CRA, AHP) are to the underlying models. This is important as we want to find a task taxonomy method that is not sensitive to underlying models. That is, the learned task taxonomy can be used to guide transfer learning for any model. For this, we first computed the Pearson correlation coefficient (  ) and the Spearman rank correlation ( r s ) between task similarities obtained with TinyBERT and those with BERT using the same similarity estimation method. The correlation coefficients 911 Method TB  B B  TB k 3 4 5 3 4 5 CRA 0.39 0.40 0.42 0.61 0.54 0.58 CNM 0.64 0.58 0.62 0.75 0.79 0.73 AHP 0.53 0.52 0.52 0.69 0.65 0.67 Table 3: Probabilities that transferability learned with TinyBERT (TB) can be used for BERT (B) or vice versa. between BERT-based and TinyBERT-based task similarity matrices obtained by the CRA, CNM and AHP are (  = 0.23, r s = 0.11), (  = 0.85, r s = 0.76) and (  = 0.36, r s = 0.34) respectively. Both AHP and CRA show pool correlations between task similarity matrices using BERT and TinyBERT. On the contrary, CNM is very robust to the variations of underlying models. We speculate that both CRA and AHP capture task relations specific to underlying models while CNM could remove such bias by building the task taxonomy based on the cognitive data. In other words, CNM is able to detect model-agnostic task relations, yet another desirable advantage over AHP with exhaustive computation cost. To further examine this hypothesis, we used the task ranking estimated with another underlying PLM x to guide transfer learning with an underlying PLM y . In our work, this would be using TinyBERT to guide BERT (TB  B) or vice versa. For each target task, we used the top k source tasks according to the task ranking with the guiding PLM x for transfer learning with the PLM y . The results were compared to the actual performance of transfer learning to the target task from the top 6 source tasks according to the task ranking with the PLM y itself. The probability of the top k source tasks occurring in the real top 6 tasks shows how much transferability learned with the PLM x can be used for the PLM y . Results are shown in Table 3, which again suggests the superiority of CNM over AHP. We further analyzed the generality of CNM to different subjects of cognitive data used in CNM, which can be found in Appendix A.5. The experimental results show that the CNM is also robust to different subjects. We visualize all pairwise task similarities for 12 tasks learned by CNM (averaged over 5 subjects) as a heatmap, shown in Figure 5(a). It is clear to see from the heatmap that 6 GLUE tasks (i.e., CoLA, QNLI, RTE, MNLI, SST-2, and MRPC) form a cluster. These tasks are all related to sentence understanding. We further perform hierarchical clustering over the 12 tasks according to their similarities to create a taxonomy tree, which is illustrated in Figure 5(b). In this paper, we have presented a cognitively inspired framework, termed CogTaxonomy, to learn relation and structure for NLP tasks. Experiments demonstrate that the task taxonomy detected by CogTaxonomy can be used to guide transfer learning across 12 different NLP tasks. Both CRA and CNM, the two essential components of CogTaxonomy, do not require exhaustive transfer learning across all source-target task pairs. The former is robust to different combinations of dissimilar-ity/similarity measurements. The latter resorts to cognitive signals to learn model-agnostic task relations. The present research was supported by Zhejiang Lab (No. 2022KH0AB01) and the Natural Science Foundation of Tianjin (No. 19JCZDJC31400). We would like to thank the anonymous reviewers for their insightful comments.
abstain abstain objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain objective abstain abstain abstain abstain abstain result abstain method objective objective method objective method objective abstain abstain other other other other other other other other other other other other other other abstain method abstain other other other other other other other other other other other other other other method other abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain other method abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain other other