Summarizing biomedical discovery from genomics data using natural languages is an essential step in biomedical research but is mostly done manually. Here, we introduce Textomics, a novel dataset of genomics data description, which contains 22,273 pairs of genomics data matrices and their summaries. Each summary is written by the researchers who generated the data and associated with a scientific paper. Based on this dataset, we study two novel tasks: generating textual summary from a genomics data matrix and vice versa. Inspired by the successful applications of k nearest neighbors in modeling genomics data, we propose a k NN-Vec2Text model to address these tasks and observe substantial improvement on our dataset. We further illustrate how Textomics can be used to advance other applications, including evaluating scientific paper embeddings and generating masked templates for scientific paper understanding. Textomics serves as the first benchmark for generating textual summaries for genomics data and we envision it will be broadly applied to other biomedical and natural language processing applications. 1 1 Introduction Modern genomics research has become increasingly automated through being roughly divided into three sequential steps: next-generation sequencing technology produces a massive amount of genomics data, which are in turn processed by bioin-formatics tools to identify key variants and genes, and, ultimately, analyzed by biologists to summarize the discovery (Goodwin et al., 2016; Kanehisa and Bork, 2003). In contrast to the first two steps that have been automated by new technologies and  Equal Contribution 1 The link to access our code: https://github.com/ amos814/Textomics software, the last step of summarizing discovery is still largely performed manually, substantially slowing down the progress of scientific discovery (Hwang et al., 2018). A plausible solution is to automatically summarize the discovery from genomics data using neural text generation, which has been successfully applied to radiology report generation (Wang et al., 2021; Yuan et al., 2019) and clinical notes generation (Melamud and Shiv-ade, 2019; Lee, 2018; Miura et al., 2021). In this paper, we study this novel task of generating sentences to summarize a genomics data matrix. Several excisting approaches demonstrate encouraging results in generating short phrases to describe functions of a set of genes (Wang et al., 2018; Zhang et al., 2020; Kramer et al., 2014). However, our task is fundamentally different from these: the input of our task is a matrix that contains tens of thousands of genes, which could be noisier than a set of selected genes; the outputs of our task are sentences instead of short phrases or controlled vocabularies. To study this task, we curate a novel dataset, Textomics, by integrating data from PMC, PubMed, and Gene Expression Omnibus (GEO) (Edgar et al., 2002) ( Figure 1). GEO is the default database repository for researchers to upload their genomics data matrices, such as gene expression matrices and mutation matrices. Each genomics data matrix in GEO is a sample by feature matrices, where samples are from often humans or mice that are sequenced together to study a specific biological problem, and features are genes or variants. Each matrix is also associated with a few sentences that are written by researchers to summarize this data matrix. After pre-processing, we obtain 22,273 matrix summary pairs, spanning 9 sequencing technology platforms. Each matrix has on average 2,475 samples and 22,796 features. Each summary has on average 46 words. We further propose a novel approach to automatically generate a summary from a genomics data matrix, which is highly noisy and high-dimensional. k nearest neighbor ( k NN) approaches have obtained great success in genomics data by capturing the hidden modules within it (Levine et al., 2015; Baran et al., 2019). The key idea of our method is to find k nearest summaries according to the genomics data similarity and then exploit the attention mechanism to convert these k nearest summaries to a new summary. Our method obtained substantial improvement in comparison to baseline approaches. We further illustrated how we can generate a genomics data matrix from a given summary, offering the possibility to simulate genomics data from textual description. We then introduced how Textomics can be used as a novel benchmark for measuring scientific paper similarity and evaluating scientific paper understanding. To the best of our knowledge, Textomics and k NN-Vec2Text together build up the first large-scale benchmark for genomics data summary generation, and can be broadly applied to a variety of natural language processing tasks. Our paper is written as follows: We first introduce the Textomics dataset (section 2) and describe the Text2Vec and Vec2Text tasks (section 3). We then propose a baseline model and k NN-Vec2Text model for Vec2Text task (section 4.1) and the model for Text2Vec task. We then evaluate our method (section 5) and provide two applications (section 6) based on Textomics dataset. We then discussed the related works and the potential direction of future works (section 7 and 8). We collected genomics data matrices from Gene Expression Omnibus (GEO) (Edgar et al., 2002). The feature of each data matrix represents the expression level of a gene or other genomic measurements of a variant (typically real numbers). The sample of each matrix is an experimental subject, such as an experimental animal or a patient. Each data matrix is associated with an expert-written summary, describing this data matrix. We obtained in total 164,667 matrix-summary pairs, spanning 12,219 sequencing platforms. Samples in different platforms have different features. However, data matrices belonging to the same sequencing platform are from the same species and share the same set of features, thus can be used together for model training. To further alleviate the missing feature problem, we kept the top-20000 features with a lower missing rate and filtered out the rest. We further selected 9 platforms with the average lowest rate of missing value and the largest amount of matrix-summary pairs to guarantee the quality and the scale of the dataset. After all, we imputed the resulted data matrices using averaging imputation across different features. Data matrices belonging to the same platform have distinct samples (e.g., patient samples collected from two hospitals). To make them com-4879 parable and provide fixed-size features for machine learning models, we empirically used a five-number summary to represent each data matrix. In particular, we calculated the smallest, the first quartile, the median, the third quartile, and the largest value of each feature across samples in a specific data matrix. We then concatenated these values of all features, resulting in a 100k-dimensional feature vector for each data matrix. Compared with other statistics such as mean, median, and mode of the features, the five number statistics maintain the patterns hidden in the raw matrices better. This vector will be finally used as the input to the machine learning model. All genomics data summaries we collected were written by the biologists who generate the corresponding genomics data matrices. Therefore, these summaries can properly reflect biologists' descriptions of their datasets. Since the summary is the first piece of information that one can learn about the dataset, authors often tend to clearly characterize their dataset in the summary. However, directly leveraging raw data of these summaries is questionable. On the syntactic level, the lengths of summary for each sample are different and comments are often used in genomics descriptions. In order to align our data and leverage the advanced Transformer model that requires fix-length sentences as well as simplifies the structure of the summary, we empirically removed the text in the brackets and truncated the summaries length to 64 words (the percentage of summaries with a length greater than 64 is 41%). On the semantic level, there could be non-informative summaries such as a simple sentence Please see our data below' and some outliers that are substantially different from other summaries. In order to increase the quality of these genomics data summaries, we manually inspected and removed the non-informative summary and excluded the outliers based on the pairwise BLEU (Papineni et al., 2002) scores through a progres-sive automated procedure. Specifically, for every summary, we treated it as the query text and calculated the pairwise BLEU-1 scores with all other summaries, filtered out those median that is lower than 0.09, and then re-applied the procedure with a higher threshold of 0.13. Finally, each of the 9 platforms contains 471 matrix-summary pairs on average, presenting a desirable number of training samples to develop data summary generation models. We summarized the statistics of these 9 platforms in Supplementary Table S1. Some of the data matrices are associated with a scientific paper, which describes how the authors generated and used the data. Therefore, the data matrix and the summary can be used to help embed these papers. We additionally retrieved these papers from PubMed and PMC databases according to the paper titles enclosed in GEO. We obtained the full text for those 7,691 freely accessible ones ( Supplementary Table S1). We will introduce two applications that jointly use scientific papers and matrix-summary pairs in section 6. We aim to accelerate genomics discovery by generating a textual summary given the five-number summary-based vector of a genomics data matrix. We refer to the five-number summary-based vector as a gene feature vector for simplicity. Specifically, consider textual summary domain D and gene feature vector domain V , let D = { DD , DV } = { ( d i , v i ) } Ni =1 dist  P ( D , V ) be a dataset containing N summary-vector pairs sampled from the joint distribution of these two domains, where d i (cid:44) (cid:104) d 1 i , d 2 i , ..., d n di i (cid:105) denotes a token sequence and v i  R l v denotes the gene feature vector. Here d ji  C , C is the vocabulary. We now formally define two cross-domain gener-0.0 0.2 0.4 0.6 0.8 1.0 Gene feature vector similarity 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 S u m m a r y e m b e dd i n g s i m il a r i t y Spearman correlation=0.45 d e n s i t y Figure 2: Density plot showing the Spearman correlation between text-based similarity (y-axis) and vector-based similarity (x-axis) on sequencing platform GPL6246. ation tasks, Vec2Text and Text2Vec, based on our 4880 dataset. Given a gene feature vector v i , Vec2Text aims to generate a summary d i that could best describe this vector v i ; given a textual summary d i , Text2Vec aims to generate the gene feature vector v i that d i describes. Since we are studying a novel task on a novel dataset, we first examined the feasibility of this task. To this end, we obtained the dense representation of each textual summary using the pre-trained SPECTER model (Cohan et al., 2020) and use these representations to calculate a summary-based similarity between each pair of summaries. We also calculated a vector-based similarity based on the gene feature vector using the cosine similarity. We found that these two similarity measurements show a substantial agreement ( Figure 2, Supplementary Table S2). After fil-tering out the outliers, all 9 platforms achieved a Spearman correlation greater than 0.2, suggesting the possibility to generate textual summary from the gene feature vector and vice versa. We first introduce a baseline model that tries to encode gene feature vectors into the semantic embedding space and then decodes it to generate text. The baseline model contains a word embedding function Emb(.), a gene feature vector encoder Enc v (.) and a decoder Dec v (.). Given a gene feature vector v i , the encoder will first embed the data into a semantic representation space s (0) i = Enc v ( v i ) , and then the decoder will start from this representation for the text generation. The generation process is autoregressive. It generates j-th word  d ( j ) i and its embedding s ( j ) i as: P (  d ( j ) i | s ( <j ) i ) = Dec v ( s ( <j ) i ) , j = 1 , ..., n d i . (1) Then we sample the next word and obtain its embedding as: s ( j ) i = Emb (  d ( j ) i ) ,  d ( j ) i sample  P (  d ( j ) i | s ( <j ) i ) . (2) This model is trained using the following loss function: L baseline =  1 | DV | | DV | (cid:88) i =1 n di (cid:88) j =1 log P (  d ( j ) i | s ( <j ) i ) . The baseline model attempts to learn an encoder that projects a gene feature vector to a semantic representation. However, the substantial noise and the high-dimensionality of the gene feature vector pose great challenges to effectively learn that projection. k -nearest neighbors models have been extensively used as the solution to overcome such issues in genomics data analysis (Levine et al., 2015; Baran et al., 2019). Therefore, one plausible solution is to explicitly leverage summaries from similar gene feature vectors to improve the generation. Inspired by the encouraging performance in using k -nearest neighbors ( k NN) in seq2seq models (Khandelwal et al., 2020, 2021) and genomics data analysis (Levine et al., 2015; Baran et al., 2019), we propose to convert the Vec2Text problem to a Text2Text problem according to the k -nearest neighbor of each vector. For a given gene feature vector g , we use e i  R to denote its Euclidean distance to another gene feature vectors v i in D . We then select the summaries of k samples that have the minimum Euclidean distances as the reference summary list  t = [ d j 1 , ..., d j k ] , where j m  { 1 , 2 , ..., | D |} denotes the index of ordered summaries w.r.t the Euclidean distance, i.e, e j 1  e j 2  ...  e j | D | . In addition to alleviating the noise in genomics data using the reference summary list (Levine et al., 2015; Baran et al., 2019), our method explicitly converts the Vec2Text problem to a Text2Text problem, and can thus seamlessly incorporate many advanced pre-trained language models into our framework. The resulted problem we need to solve is a k sources to one target generation problem. One naive solution is to concatenate the k reference summaries together. However, this concatenation will make the source text much longer than the target text and how to order each summary during concatenation also remains unclear. Instead, we propose to transform this problem into k one-to-one generation problem and then use attention-based strategy to fuse them. Concretely, let n j = max { n j 1 , ..., n j k } be the maximum length among all the reference summaries. We first get the representation of each summary x j m = Emb ( d j m ) = (cid:104) x (1) j m , ..., x ( n j ) j m (cid:105) for m = 1 , ..., k . Here x ( i ) j m denotes the vector embedding of the i-th word in m-th summary. We construct fixed-length reference summaries by padding after the end of each summary with length less than n j . We then utilize self-attention module (SA) (Vaswani et al., 2017) to get the aggregated embedding of each reference with their embeddings as well as the gene feature vector distance e i . Let Q r , K r , V r be the query, key, value matrices of embedding 4881 sequence r = (cid:104) r (1) , ..., r ( l r ) (cid:105) , we have: SA ( r ) = Attention ( Q r , K r , V r ) . (4) We then calculate the attention score as following: a j m = SA ( (cid:104) x (1) j m , ..., x ( n jk ) j m (cid:105) ) , (5) sc j = SA ( (cid:104) e j 1  a j 1 , ..., e j k  a j k (cid:105) ) , (6) where sc j = [ sc j 1 , ..., sc j k ]  R k . Here we used a 2-layer self attention scheme to first acquire the aggregated feature of each summary and then calculate the attention score based on that. The fi-nal score is then calculated based on the attention scores and temperature  as: w j m = exp (   sc j m ) (cid:80) kl =1 exp (   sc j l ) . (7) Then, we aggregate embedding sequences by taking weighted averages:  x ( l ) j = k (cid:88) m =1 w j m x ( l ) j m , l = 1 , ..., n j . (8) Let P <l,x ( d ) = P  LM ( d ( l ) | d ( <l ) , x ) , 0 < l < n d be the probability distribution of d ( l ) output by the language model  LM conditioned on the sequences of the embedding vectors x and the first l  1 sequence tokens. We feed the aggregated embedding sequences into the language model to reconstruct the summary d using an autoregressive-based loss function: L k NN-Vec2Text =  (cid:88) d  DD n d (cid:88) l =1 log P <l,  x j ( d ) | DD | . (9) 4.2 Text2Vec We model the reverse problem of generating the gene feature vector v from a textual summary d as a regression problem. Our model is composed with a semantic encoder Enc d ( . ) and a readout head MLP ( . ) . Specifically, the encoder will embed the textual summary into dense representation x = Enc d ( d ) , and the readout head will map the representation to the gene feature vector  v = MLP ( x ) . Then we train this model by minimizing the rooted mean squared errors (RMSE): L v = (cid:115) 1 | DV | (cid:88) v i  DV ||  v i  v i || 22 . To evaluate the performance of k NN-Vec2Text on the task of Vec2Text, we compared it to the baseline models in 4.1. For the baseline models, we used a one layer MLP network as its encoder, and tested with different decoder structure, including canonical Transformer (decoder of T5) (Vaswani et al., 2017), GPT-2 (Radford et al., 2019), and Sent-VAE (Bowman et al., 2016). For k NN-Vec2Text, we directly used both the encoder and the decoder of T5 (Raffel et al., 2020), one of the state-of-the-art Transformer style models. we set k = 4 and  = 0 . 1 as this setting achieved the best empirical performance, though it is worth noting that our model is robust on the choices of k (from 1 to 4) and  (from 0 to 1). For all 9 platforms, we reported the average performance under 5-fold cross validation to evaluate the robustness of our method. The results of BLEU-1 score (Pa-pineni et al., 2002) are summarized in Figure 3a . We found that k NN-Vec2Text substantially outperformed other methods by a large margin. Specifically, k NN-Vec2Text obtained a 0.206 BLEU-1 score on average while none of the other three methods achieved an average BLEU-1 score greater than 0.150. The prominent performance of our method demonstrates the effectiveness of using a k -nearest-neighbor approach to convert the Vec2Text problem to a Text2Text problem. To further understand the superior performance of the k NN-Vec2Text model, we presented a case study in Table 1 . In this case study, the generated summary is highly accurate compared to the ground truth summary. By examining the summaries of the 4 nearest neighbors in the gene feature vector space, we found that the generated summary is composed of short spans from each individual neighbor, again indicating the advantage of using a k -nearest neighbor for this task. Our method leveraged an attention mechanism to unify these four neighbors, thus offering an accurate generation. We also observed consistent improvement of our method over comparison approaches on other metrics and summarized the results in Supplementary Table S3. We next used the Text2Vec task to illustrate how our dataset can be used to compare the performance of different pre-trained language models. In particular, we compared a recently proposed scientific paper embedding method SPECTER (Cohan et al., 2020), which has demonstrated prominent performance in a variety of scientific paper analysis tasks, with SciBERT (Beltagy et al., 2019), BioBERT (Lee 4882 GPL96 GPL6244 GPL10558 GPL6887 GPL198 GPL570 GPL6246 GPL1261 GPL13534 Textomics platform 0.0 0.1 0.2 0.3 BLEU1 s c o r e kNN-Vec2Text GPT-2 Sent-VAE Transformer GPL6887 GPL10558 GPL1261 GPL96 GPL198 GPL570 GPL6244 GPL6246 GPL13534 Textomics platform 10% 30% 50% 70% I m p r o v e m e n t o v e r b a s e li n e SPECTER SciBERT SentBERT BioBERT BERT a b Figure 3: Performance on Vec2Text et al., 2020) and SentBERT (Wang and Kuo, 2020) and the vanilla BERT (Devlin et al., 2019). While the other language models directly take the token sequence as the input, SPECTER model needs to take both the abstract and the title. To make a fair comparison, we concatenated the title and the summary as the input for models other than SPECTER. For all 9 platforms, we reported the average performance under 5-fold cross validation. We further implemented a simple averaging baseline approach that predicts the vector for a test summary according to the average vectors of training samples. This baseline does not utilize any textual summary and can thus help us assess the effect of using textual summary information in this task. We used RMSE to evaluate the performance of all methods. We reported the RMSE improvement of each method over the averaging baseline model in Figure 3b . We found that all methods outperform the baseline approaches by gaining at least 15% improvement, indicating the importance of considering textual summary in this task. SPECTER achieved the best overall performance among all five methods, suggesting the advantage of separately modeling the title and the abstract when embedding scientific papers. Embedding scientific papers is crucial to effectively identify emerging research topics and new knowledge from scientific literature. To this end, many machine learning models have been proposed to embed scientific papers into dense embeddings and then applied these embeddings for a variety of downstream applications (Cohan et al., 2020; Lee et al., 2020; Wang and Kuo, 2020; Beltagy et al., 2019; Devlin et al., 2019). However, there is currently limited golden standard that can measure the similarity between two papers. As a result, existing approaches use surrogate metrics such as citation relationship, keywords, and user activities to evaluate their paper embeddings (Cohan et al., 4883 GPL198 GPL13534 GPL6244 GPL570 GPL10558 GPL6246 GPL1261 GPL6887 GPL96 Textomics platform 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Sp e a r m a n c o rr e l a t i o n Abstract Introduction Method Result Conclusion GPL198 GPL13534 GPL6244 GPL570 GPL10558 GPL6246 GPL1261 GPL6887 GPL96 Textomics platform 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Sp e a r m a n c o rr e l a t i o n SPECTER SentBERT SciBERT BioBERT BERT a b Figure 4: Performance on using Textomics as the benchmark to evaluate scientific paper embeddings. Textomics can be used to measure these paper embedding approaches by examining the consistency between the embedding-based paper similarity and the embedding-based summary similarity since both the paper and the summary are written by the same authors. In particular, for a pair of summaries d i , d j  DD , let t i , t j be the text (e.g., abstracts) extracted from their corresponding scientific papers. Let Enc d be the encoder of the paper embedding method we want to evaluate. We first get their embeddings as: s d i , s d j = Enc d ( d i ) , Enc d ( d j )  R l s , (11) s t i , s t j = Enc d ( t i ) , Enc d ( t j )  R l s . (12) We then compute the pairwise Euclidean distance between all pairs of summaries and all pairs of paper text as: s d i,j = (cid:118)(cid:117)(cid:117)(cid:116) l s (cid:88) k =1 ( s ( k ) d i  s ( k ) d j ) 2  R, (13) s t i,j = (cid:118)(cid:117)(cid:117) (cid:116) l s (cid:88) k =1 ( s ( k ) t i  s ( k ) t j ) 2  R. (14) To evaluate the quality of the encoder Enc d , we can calculate the Spearman correlation between the pairwise summary similarity and the pairwise text similarity. A larger Spearman correlation means the summary / textual contents of two samples in the pair are better aligned with each other, which indicates this Enc d is more accurate in embedding scientific papers. As a proof-of-concept, we obtained the full text of 7,691 papers in our dataset from the freely accessible PubMed Central. We segmented each paper into five sections, which included abstract, introduction, method, result and conclusion. We first compared different paper embedding methods using the abstract of a paper. The five embedding methods we considered are introduced in section 5.1. Since SPECTER takes both the title and paragraph as the input we used the first sentence of the summary as a pseudo-title when encoding the summary. The results are summarized in Figure 4a . We found that SPECTER was substantially better than other methods on 8 out of the 9 platforms. SPECTER is specifically developed to embed scientific papers by processing the title and the abstract separately, whereas other pre-trained language models simply concatenated the title and the abstract. The superior performance of SPECTER suggests the importance of separately modeling paper title and abstract when embedding scientific papers. SentBERT obtained the best performance among four pre-trained language models, partially due to its prominent performance in sentence-level embedding. We further noticed that the relative performance among different methods is largely consistent with the previous work evaluated on other metrics (Cohan et al., 2020), demonstrating the high-quality of Textomics. After observing the superior performance of SPECTER, we next investigated which section of the paper can be best used to assess paper similarity. Although existing paper embedding approaches often leverage the abstract for embedding, other sections, such as introduction and results might also be informative, especially for papers describing a specific dataset or method. We thus applied SPECTER to embed five different sections of each scientific paper and used Textomics to evaluate which section can best reflect paper similarity. We observed a consistent improvement of using the abstract section 4884 in comparison to other paper sections ( Figure 4B), which is consistent with the intuition that the abstract represents a good summary of the scientific paper, again indicating the reliability of using Textomics to evaluate paper embedding methods. Creating masked sentences and then filling in these masks can examine whether the machine learning model has properly understood a scientific paper (Yang et al., 2019; Guu et al., 2020; Ghazvininejad et al., 2019; Bao et al., 2020; Salazar et al., 2020). However, one challenge in such research is how to generate masked sentences that are relevant to a given paper while also ensuring the answer is enclosed in the paper. Our dataset could be used to automatically generate such masked sentences using the summary, which is highly relevant to the paper but also not overlapped with the paper. In particular, we can mask out keywords from the summary and then use this masked summary as the question and let a machine learning model to find the answer from the non-overlapping scientific paper. Let C bio be a dictionary that contains biological keywords we want to mask out from the summary, ( d i , t i ) be a pair of textual summary and paragraph text extracted from its corresponding scientific paper. If the j -th word w i = d ( j ) i  C bio in the summary belongs to C bio , our proposed task is to predict which word in C bio is the missing word in d masked given t i . The masked summary d masked is the same as d i except its j -th word is substituted with [PAD]. For simplicity, we only mask at most one token in d i . We, therefore, form our task as a multi-class classification problem. Sim-GPL570 GPL6887 GPL6244 GPL13534 GPL6246 GPL1261 GPL10558 GPL198 GPL96 Textomics platform 0.1 0.3 0.5 0.7 A cc u r a c y CovocEupath Obi_iee ObiPlanp EcocoreXpo Argo TrakPremedonto Figure 5: Bar plot showing the accuracy of filling the masked sentences of ten biomedical categories across 9 platforms using Textomics as the benchmark. the paragraph text t i . To generate C bio , we leveraged a recently developed biological terminology dataset Graphine (Liu et al., 2021), which provides the biological phrases spanning 227 categories. We selected 10 categories that can produce the largest number of masked sentences in Textomics. We manually filtered ambiguous words and stop words. On average, each category contains 317 keywords. We used a fully connected neural network to perform the multi-class classification task. The input feature is the concatenation of the masked summary embedding and the paragraph embedding. We used SPECTER to derive these embeddings as it has obtained the best performance in our previous analysis. The results are summarized in Figure 5 . We observed improved accuracy on all ten categories, which are much better than the 0.4% accuracy by random guessing, indicating the usefulness of our benchmark in scientific paper understanding. Finally, we found that the performance of each category varied across different platforms, suggesting the possibility to further improve the performance by jointly learning from all platforms. Our task is related to existing works that take structured data as the input and then generate the unstructured text. Different input data modalities and related datasets have been considered in the literature, including text triplets in RDF graphs (Gardent et al., 2017; Ribeiro et al., 2020; Song et al., 2020; Chen et al., 2020)), text-data tables (Lebret et al., 2016; Rebuffel et al., 2022; Dusek et al., 2020; Rebuffel et al., 2020; Puduppully and Lapata, 2021; Chen et al., 2020), electronic medical records (Lee, 2018; Guan et al., 2018), radiology reports (Wang et al., 2021; Yuan et al., 2019; Miura et al., 2021), and other continuous data modalities without explicit textual structures such as image (Lin et al., 2014; Cornia et al., 2020; Ke et al., 2019; Radford et al., 2021), audio (Drossos et al., 2020; Manco et al., 2021; Wu et al., 2021; Mei et al., 2021), and video (Li et al., 2021; Ging et al., 2020; Zhou et al., 2018; Li et al., 2020). Different from these structures, our dataset takes a high dimensional genomics feature matrix as input, which doesn't exhibit structure and is thus substantially different from other modalities. Moreover, our dataset is the first dataset that aims to convert genomics feature vector to textual summary. The substantial noise and high-dimensionality of genomics data matrices 4885 further pose unique challenges in text generation. Our k NN-Vec2Text model is inspired by the re-cent success in applying k NN-based language models to machine translation (Khandelwal et al., 2021) and language models (Khandelwal et al., 2020; He et al., 2021; Ton et al., 2021). The main difference between our methods and their approaches is that while we try to leverage k NN in the genomics vector space to construct reference text, they use k NN in the text embedding space during the autoregressive generation process to help adjust the sample distribution. Some other methods can be used to generate text from vectors, such as (Bowman et al., 2016; Song et al., 2019; Miao and Blunsom, 2016; Montero et al., 2021; Zhang et al., 2019). Their inputs are latent vectors that need to be inferred from the data and do not have specific meanings, which are different from our gene feature vectors. In this paper, we have proposed a novel dataset Textomics, containing 22,273 pairs of genomics matrices and their corresponding textual summaries. We then introduce a novel task of Vec2Text based on our dataset. This task aims to generate the textual summary based on the gene feature vector. To address this task, we propose a novel method k NN-Vec2Text, which constructs the reference text using nearest neighbors in the gene feature vector space and then generates a new summary according to this reference text. We further introduce two applications that can be advanced using our dataset. One application aims at evaluating scientific paper similarity according to the similarity of its corresponding data summary, and the other application leverages our dataset to automatically generate masked sentences for scientific paper understanding. To the best of our knowledge, Textomics and k NN-Vec2Text serve as the first large-scale genomics data description benchmark, and we envision it will be broadly applied to other natural language processing and biomedical tasks. On the biomedical side, we provide the benchmark to develop new NLP tools that can generate the description for a genomics data. Since each pub-lic genomics data needs a description, such tools will substantially accelerate this process. Also, descriptions generated from Textomics could contain new knowledge. While humans write the description almost solely based on that single dataset, description generation models jointly consider thousands of datasets, enabling the transfer of knowledge from other datasets. The generated description can guide biologists to write more informative descriptions, which ultimately leads to better and larger genomics description data. When biologists start to obtain the generated description from NLP tools, they will be able to write more informative descriptions with the assistance from these NLP tools. On the NLP side, the relationship between a summary and a dataset is analogous to the relationship between an abstract and a scientific paper. A high-quality summary ideally contains all perspectives of a study, including problems, methods, and discoveries. Moreover, our work will bridge the NLP and the genomics community and motivate people to analyze genomics data using NLP methods based on the multi-modality dataset introduced in this paper. Textomics could also be used to help scientific paper analysis tasks, such as paper recommendation (Bai et al., 2019), citation text generation (Luu et al., 2020), and citation prediction (Suzen et al., 2021). Our method searches for the nearest neighbours by calculating the Euclidean distance between five-number summary vectors of the genomics feature matrices. However, this might lose useful information hidden in the original matrices. It's challenging and worth exploring end-to-end approaches that can learn embeddings from the genomics feature matrices instead of representing them as five-number summary vectors. On the Text2Vec side, one remaining challenge that could be the future direction of our work is to directly generate the whole genomics feature matrix instead of the five-number summary vector. Also, it would be interesting yet challenging to jointly learn the Text2Vec and the Vec2Text tasks, and one potential solution is to further decode the generated vector to reconstruct the embedding of summaries in Text2Vec, and leverage the resulted decoder to predict the embedding of text by using k NN method in the text embedding space. Also, it is interesting to jointly model data from multiple platforms, which might lead to beneficial results by transferring biological insights learned from different platforms.
abstain objective abstain objective objective method objective abstain other abstain objective abstain method abstain abstain abstain abstain result abstain abstain objective abstain objective result method objective objective objective objective method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain method abstain abstain other abstain other abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method method abstain abstain method abstain method abstain method method method method method abstain method abstain method method method abstain method abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method other method objective other method method other abstain objective objective abstain objective method objective objective objective abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain abstain