## BIORAG: A RAG-LLM Framework for Biological Question Reasoning

Chengrui Wang 1,2 , Qingqing Long 1,2 , Meng Xiao 1,2 , Xunxin Cai 1,2 , Chengjun Wu 1,2 , Zhen Meng 1,2 , Xuezhi Wang 1,2 , Yuanchun Zhou 1,2 *

1 Computer Network Information Center, Chinese Academy of Sciences.

2

University of the Chinese Academy of Sciences. {crwang,qqlong,shaow,xxcai,cwu,zhenm99,wxz,zyc}cnic.cn

## Abstract

The question-answering system for Life science research, which is characterized by the rapid pace of discovery, evolving insights, and complex interactions among knowledge entities, presents unique challenges in maintaining a comprehensive knowledge warehouse and accurate information retrieval. To address these issues, we introduce BIORAG , a novel Retrieval-Augmented Generation (RAG) with the Large Language Models (LLMs) framework. Our approach starts with parsing, indexing, and segmenting an extensive collection of 22 million scientific papers as the basic knowledge, followed by training a specialized embedding model tailored to this domain. Additionally, we enhance the vector retrieval process by incorporating a domain-specific knowledge hierarchy, which aids in modeling the intricate interrelationships among each query and context. For queries requiring the most current information, BIORAG deconstructs the question and employs an iterative retrieval process incorporated with the search engine for step-by-step reasoning. Rigorous experiments have demonstrated that our model outperforms fine-tuned LLM, LLM with search engines, and other scientific RAG frameworks across multiple life science question-answering tasks.

## 1 Introduction

Research and trends in the Biology have shown a continuously evolving, marked by rapid discoveries and the increasing complexity of its knowledge domains (Bertoline et al., 2023; Long et al., 2021b). In addition, the growing trend for interdisciplinary research between Biology and other fields (Lepore et al., 2023; Xiao et al., 2023; Xiao et al.), such as artificial intelligence (Holzinger et al., 2023; Long et al., 2021a), material science (Atkins et al., 2023), and environmental science (Cole et al., 2021), further amplifies the complexity of knowledge syn-

* Yuanchun Zhou is the corresponding author.

Figure 1: An illustration of the difference between three paradigms: (a) fine-tuned language model embedded domain knowledge into deep space; (b) RAG-based method retrieve supplementary information from constructed knowledge base; (c) BIORAG adaptively select knowledge source and domain-specific tools to advance the biology question-reasoning task.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01107v2-figure-e978cd53-5c3c-474c-8b65-5b95f8fb040b.jpg)

thesis. To bridge the gap and facilitate multidiscipline cooperation, automated question-reasoning systems (Auer et al., 2023) play a pivotal role in enabling experts from diverse fields to effectively navigate and integrate this burgeoning and complex body of biological knowledge (Yang et al., 2023). However, this ever-changing landscape and the complex interplay between different knowledge components present obstacles (Lee et al., 2023; Castro Nascimento and Pimentel, 2023; Lecler et al., 2023; Song et al., 2020) in creating efficient domain-specific question-reasoning systems.

The prior literature partially addresses questionreasoning in the biology domain and can be grouped into two mainstream (Nguyen et al., 2024) (as shown in Figure 1 (a-b)). Fine-tuned Language Model (Gu et al., 2021) includes models like bioBERT (Lee et al., 2020), sciBERT (Beltagy et al., 2019), and large language models tailored for specific domains, such as PMC-Llama (Wu et al., 2024) and Llava-med (Li et al., 2024). These models are trained on domain-specific corpora, thereby embedding deep domain knowledge within their architectures. However, that embedded knowledge could be incomplete and computationally expensive to update. Retrieval-Agumented Generation methods follow the information indexing and retrieval, information augmentation, and answer generation paradigm. For instance, PGRA (Guo et al., 2023) adopts a retriever to search and re-ranking the context, then generate the answer. Later research has aimed to improve these systems by either optimizing the retrieval processes using prior answers (Wang et al., 2023), enhancing model functionality through iterative feedback cycles (Liu et al., 2024), or expanding the knowledge base with search engines to incorporate the latest information (O'Donnell, 2023). Although RAG-based methods address the issue of updating information, they often oversee the intricate complexities inherent in the domain knowledge of biology.

Based on the aforementioned discussion, we summarize three challenges in building efficient biology question-reasoning systems: (C1) The scarcity of high-quality domain-specific corpora. While biological research publications are abundant, there remains a significant void in the availability of extensive, high-quality datasets to build robust information indexing models. (C2) The inherent complexity of biological knowledge systems. This complexity is compounded by the interdisciplinary nature of modern biological research. Consequently, automated question-reasoning systems must be able to understand and process multifaceted and often ambiguous biological query. (C3) The continual updating of knowledge. Biology is a dynamic field where discoveries are frequently made, and existing theories are regularly revised or replaced. This fluidity necessitates that questionreasoning systems adeptly select the knowledge source from databases or contemporary search en- gines to reflect the correct scientific understanding.

Our Perspective and Contributions: To solve the above challenges, we proposed BIORAG, a novel Retrieval-Augmented Generation framework integrated with Large Language Models for biological question-reasoning. To obtain a robust domain-specific information indexing embedding model, we start by parsing, indexing, and segmenting extensive research articles from the biology domain and constructing high-quality training cor- pora. BIORAG then addresses the complexity of biological knowledge systems by combining a prebuilt research hierarchy with an embedding model for accurate context retrieval. To cope with emerging biology knowledge, BIORAG can adaptively select knowledge sources from search engines, existing domain-specific tools, or indexed research articles. Once the framework determines that it has gathered sufficient information, it will generate the answer based on the reasoned material.

We illustrate the question-reasoning power of BIORAG on 6 popularly used biology QA datasets and compare it against 6 baseline methods. Extensive case studies show the great potential to apply this framework to general science questionreasoning scenarios.

## 2 Biological Retrieval-Augmented Generation LLM Framework

In this paper, we propose the Bio logical R etrievalA ugmented G eneration LLM Framework , namely BIORAG (as shown in Figure 2). In the following sections, we first introduce the preliminary step of constructing a high-quality local information source and training the biological domain-specific information indexing embedding model. For questions that require the most current or other domainrelated data, we introduce external information sources. Then, we demonstrate the knowledge hierarchy-based query pre-processing, retriever execution component, and how the model iteratively collects sufficient information. Finally, the large language model will generate the answer based on the information obtained. The details of customized prompts are given in Section 2.4.

## 2.1 Internal Biological Information Source

High-quality domain-specific corpora are crucial for enriching the information source and enhancing the embedding model in the context of biological question-reasoning systems. To achieve this goal, we extract research papers from the global biomedical article database maintained by the National Center for Biotechnology Information 1 (NCBI) (Schoch et al., 2020). This extensive repository aggregates over 37 million scientific citations and abstracts spanning from the 1950s to the present, encompassing a broad array of biomedical fields, including clinical medicine, molecular biology, etc. For the purposes of this study, we utilize the abstracts from

Figure 2: The architecture of our proposed BIORAG framework. The pipeline consists of five iterative components designed to enhance the process of biological question-reasoning: ① Retriever Selection aims to choose the most ideal information source; ② Query Pre-processing aims to rewrite the query and find closed topic tag from pre-defined knowledge hierarchy; ③ Retriever Execution aims to combination retrieve the correlated context from knowledge base; ④ Self-Evaluation assess the adequacy of the retrieved information and decides whether to cycle through additional retrieval tools or to move to the next phase; ⑤ Inference and Generation uses the information gathered to generate an informed and accurate answer to the biological query.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01107v2-figure-3ab0709c-76dc-4f8d-bbfe-476e963b7529.jpg)

these PubMed papers as the supporting corpus for the BIORAG framework.

Local Data Preparation: Specifically, we initially downloaded over 37 million original papers from which we subsequently filtered out 14 million entries deemed to be of low quality. The preprocessing of these texts was conducted using the Unstructured tool 2 , specifically designed to ingest and preprocess unstructured textual data effectively. Our filtration process involved the removal of gibberish using regular expression techniques, as well as the exclusion of non-semantic content such as hyperlinks, charts, tables, and other embedded tags. This meticulous process yielded a corpus of 22,371,343 high-quality, processed PubMed abstracts.

Information Indexing: To further refine the retrieval performance of abstracts tailored to specific biological questions, we developed a specialized biological embedding model within the BIORAG framework. This model employs PubMedBERT (Gu et al., 2021) as the foundational model. We enhanced this model using the CLIP (Contrastive Language-Image Pretraining) technique (Li et al., 2021; Nussbaum et al., 2024), allowing us to fine-tune the model, denoted as M emb. Based on this, we constructed a local, high-quality biological vector database (Xian et al., 2024) to support efficient and effective query processing and re- trieval operations. This database serves as a critical resource in facilitating rapid and accurate access to relevant biomedical information, significantly advancing the capabilities of our BIORAGframework in handling complex biological questions.

## 2.2 External Information Sources

External biology knowledge is crucial to biological reasoning due to the rapidly evolving nature of biological research, which continuously integrates new discoveries. To address this challenge, we introduce two external information sources.

Biological Data Hub: In BIORAG, we harness several specialized biological Hubs to ensure the accuracy of experimental data and to provide detailed biological insights. Specifically, BIORAGintegrates the following databases, each serving a unique purpose in the broader context of biological analyses: (1) Gene Database : This 3 resource provides comprehensive information on the functions, structures, and expressions of specific genes. It is invaluable for addressing queries related to gene mechanisms, gene actions, and gene expressions, facilitating a deeper understanding of gene-related phenomena. (2) dbSNP Database 4 : This database houses a vast repository of single nucleotide polymorphisms (SNPs), offering critical insights into genetic variants and their potential as-

4 https://www.ncbi.nlm.nih.gov/snp/

sociations with various diseases. It is instrumental for studies exploring the genetic basis of disease and trait inheritance. (3) Genome Database : Pro5 viding complete genome sequences, this database is essential for studying the structure, function, and evolution of genomes across different organisms. It supports comprehensive genomic analyses and comparative studies, enhancing our understanding of genomic architecture and its functional implications. (4) Protein Database 6 : This resource offers detailed information about the sequences, structures, and functions of proteins. It is crucial for exploring protein-related biological processes, understanding molecular functions, and investigating the complex interactions within the proteome.

Search Engine: To ensure access to the most current discussions and developments, BIORAG incorporates a variety of search engines, including Google, Bing, arXiv, Wikimedia, and Crossref. Each platform contributes uniquely to the aggregation of information: (1) Google and Bing: These search engines scour the web for a diverse range of content, including news articles, blogs, and forums, providing insights into public discussions and concerns related to scientific topics. This breadth of information is crucial for understanding the societal impact and general discourse surrounding scientific issues. (2) arXiv: As a repository for preprint papers, arXiv offers access to the latest research reports and scholarly articles across multiple scientific disciplines before they undergo peer review. This source is invaluable for staying abreast of the newest scientific theories and experiments. (3) Wikimedia: Known for its user-friendly content, Wikimedia offers easily digestible explanations of complex scientific concepts and principles. This resource helps simplify advanced topics for broader public understanding and educational purposes. (4) Crossref: This service acts as a comprehensive aggregator of academic citation data, providing links to peer-reviewed scholarly publications and their citation networks. Crossref is essential for accessing high-quality research outputs and understanding their impact on the academic community.

## 2.3 Self-evaluated Information Retriever

Following the construction of the internal and external information source, BIORAG is firstly tasked with comprehending the complex disci-

Based on the QUESTION , analyze the related MeSH terms to format them properly.

QUESTION

: [.....]

MeSH : [ κ 1 , κ 2 , ...]

Figure 3: Training Template for M MeSH.

## Input Question

What are the differences between innate immunity and adaptive immunity?

## Predicted MeSH by M MeSH

[Adaptive Immunity, Animals, ...]

## Generated SQL

" filtered by ":[eq("MeSH", "Adaptive Immunity") or eq("MeSH", "Animals") or ...],

" ordered by ": embedding similarity

Figure 4: An example of MeSH filtering SQLs Generation.

plinary framework of the life sciences to retrieve the most relevant information accurately. Moreover, BIORAG integrates a self-evaluation mechanism to continuously assess the adequacy and relevance of the information it has collected.

Internal Information Retrieve: To effectively navigate the inherent complexity of biological knowledge systems, BIORAG leverages an integrated approach, combining a well-defined hierarchical structure with indexed information to conduct a comprehensive internal information retrieval. The Medical Subject Headings 7 (MeSH) thesaurus is popularly used for indexing, cataloging, and searching for biomedical-related information and research papers. Specifically, we first train a model M MeSH to predict MeSH of the input questions. We then use the templates in Figure 3 for fine-tuning a Llama3-8B model to classify given questions. After that, we construct MeSH filtering SQLs (as shown in Figure 4) to generate the scalar condition retrieval. A candidate result is considered relevant to the given question because it has one consistent MeSH with the question. Then, the vector retrieval process is adopted to sort the relative results based on the cosine similarity of the sentence embedding between the input questions and the filtered results.

Self-evaluation Strategy: In order to ensure the accuracy and contemporary of the retrieved information, BIORAG incorporates a self-evaluation strategy that assesses the adequacy of data collected from the internal knowledge base. In detail, this

|                          | LLM    | LLM       | LLM       | BioLLM    | BioLLM     | SciRAG   | SciRAG   | BioRAG   |
|--------------------------|--------|-----------|-----------|-----------|------------|----------|----------|----------|
|                          | GPT3.5 | Llama3-8B | Llama-70B | PMC-Llama | BioMistral | GeneGPT  | NewBing  | BioRAG   |
| Nomenclature             |        |           |           |           |            |          |          |          |
| Gene alias               | 7      | 0         | 0         | 0         | 8          | 84       | 68       | 98       |
| Gene name conversion     | 0      | 0         | 0         | 0         | 0          | 100      | 100      | 100      |
| Genomic location         |        |           |           |           |            |          |          |          |
| Gene SNP association     | 0      | 0         | 0         | 0         | 0          | 100      | 0        | 100      |
| Gene location            | 9      | 20        | 28        | 14        | 12         | 66       | 70       | 86       |
| SNP location             | 5      | 48        | 94        | 0         | 0          | 98       | 100      | 100      |
| Functional analysis      |        |           |           |           |            |          |          |          |
| Gene disease association | 31     | 0         | 0         | 0         | 8          | 66       | 64       | 71       |
| Protein-coding genes     | 54     | 6         | 12        | 40        | 80         | 100      | 100      | 100      |

Table 1: Performance of BioRAG compared to other RAG-LLMs on the GeneTuring QA dataset.The scores represent accuracy.

|                  | LLM    | LLM       | LLM       | BioLLM    | BioLLM     | SciRAG   | SciRAG   | BioRAG   |
|------------------|--------|-----------|-----------|-----------|------------|----------|----------|----------|
|                  | GPT3.5 | Llama3-8B | Llama-70B | PMC-Llama | BioMistral | GeneGPT  | NewBing  | BioRAG   |
| MedMCQA          | 54     | 51        | 71        | 56        | 49         | 0        | 55       | 73       |
| Medical Genetics | 74     | 51        | 67        | 28        | 67         | 0        | 88       | 88       |
| College Biology  | 73     | 75        | 88        | 30        | 67         | 0        | 71       | 90       |
| College Medicine | 65     | 61        | 70        | 23        | 51         | 0        | 78       | 78       |

Table 2: Performance of BioRAG compared to other RAG-LLMs on the biological-related QA benchmarks.The scores represent accuracy. Bold and underlined results denote the highest and second-highest performance, respectively.

critical evaluation is driven by the backend large language model which aims to determine whether the information retrieved internally is sufficient to address the posed question substantively. If the internal content is insufficient, the model will loop back to pertinent external knowledge sources. Additionally, when the initial assessment indicates that the scientific questions require broader searches or retrieval of entity-specific data, the model tends to deploy external tools. This methodology supports the framework's goal of providing precise, up-todate, comprehensive answers, facilitating more informed decision-making, and advancing research and applications in the life sciences.

## 2.4 Customized Prompts Detail

To maximize the effect of the retrieved corpus and knowledge, we design customized prompts in BIORAG. The prompts in Figure. 2 is detailed defined as follows,

- · Prompt # 1 : To provide the most helpful and accurate response to the following Question: {Question} . You have been given descriptions of several RETRIEVAL METHODS: {Retrieval} . Please select the RETRIEVAL

METHODS you consider the most appropriate for addressing this question.

- · Prompt # 2 : Based on the RETRIEVAL METHODS you selected, and considering the Question and the Input Requirements of the retrieval method, please REWRITE the search query accordingly.
- · Prompt # 3 : Now, using the rewritten QUERY and the retrieval FILTER methods, perform a logical combination to execute the search effectively.
- · Prompt # 4 : Based on the RETRIEVAL RESULTS from the above steps, please evaluate whether the RESULTS support answering the original Question . If they do not support it, output " NO ". If they do support it, output " YES ".
- · Prompt # 5 : Based on the RETRIEVAL RESULTS, perform a comprehensive reasoning and provide an answer to the Question .

Furthermore, we designed instruction manuals for specialized biological tools and databases, aim at exploiting their potentialities. These instructions are shown as follows,

- · Manual #Gene : The Gene database search engine is a valuable tool for retrieving comprehensive information about genes, including gene structure, function, and related genetic events. It is particularly useful for answering detailed questions regarding gene-related research and findings. To utilize this search engine effectively, the input must be a specific gene name.
- · Manual #dbSNP : The dbSNP database search engine is an essential tool for retrieving detailed information about single nucleotide polymorphisms (SNPs) and other genetic variations. It is particularly useful for answering questions related to genetic diversity, allele frequency, and related genetic studies. To utilize this search engine effectively, the input must be a specific SNP identifier or genetic variant name.
- · Manual #Genome : The Genome database search engine is an indispensable tool for accessing comprehensive information about entire genomes, including their sequences, annotations, and functional elements. It is particularly useful for answering complex questions about genomic structures, variations, and comparative genomics. To use this search engine effectively, the input must be a specific genome name or identifier.
- · Manual #Protein : The Protein database search engine is a crucial resource for obtaining detailed information about proteins, including their sequences, structures, functions, and interactions. It is particularly useful for answering questions related to protein biology, biochemical properties, and molecular function. To use this search engine effectively, the input must be a specific protein name or identifier.
- · Manual #Web Search : The Web Search Engine is a powerful tool designed to help you find information about current events quickly and efficiently. It is especially useful for obtaining the latest news, updates, and developments on a wide range of topics. To use this search engine effectively, simply enter a relevant search query.
- · Manual #PubMed : The PubMed local vector database search engine is an advanced tool designed for retrieving biomedical literature and research articles using vector-based search techniques. It is particularly useful for answering detailed questions about medical research, clinical studies, and scientific discoveries. To utilize this search engine effectively, the input should be a specific query or topic of interest.

## 3 Results &amp; Analysis

## 3.1 Datasets

We conduct experiments on 6 popularly used biological-related QA datasets to evaluate our proposed BIORAG, i.e., GeneTuring (Hou and Ji, 2023), MedMCQA (Pal et al., 2022), Medical Genetics (Hendrycks et al., 2020), College Biology (Hendrycks et al., 2020), College Medicine (Hendrycks et al., 2020). Note that the GeneTuring dataset contains more specialized biological questions. It contains 12 tasks, and each task has 50 question-answer pairs. We use 7 GeneTuring tasks that are related to NCBI resources to evaluate the proposed BIORAG. The chosen tasks are classified into three modules and briefly described as follows,

- · Nomenclature: This is about gene names. The objectives of the gene alias task and name conversion task are finding the official gene symbols for their non-official synonyms.
- · Genomics location: The tasks are about the locations of genes, single-nucleotide polymorphism (SNP), and their relations. We include the gene location, SNP location, and gene SNP association tasks. The first two tasks ask for the chromosome locations of a gene or an SNP, and the last one asks for related genes for a given SNP.
- · Functional analysis asks for gene functions. We use the gene-disease association task where the goal is to return related genes for a given disease, and the protein-coding genes task which asks whether a gene is a proteincoding gene or not.

## 3.2 Baslines

We compare BIORAGwith various baselines, which can be classified into three categories,

Table 3: Ablation study on the GeneTuring dataset.The scores represent accuracy.

|                          | Data   | Data   | Data   | Component   | Component   | Component   | Base Model   | Base Model   |
|--------------------------|--------|--------|--------|-------------|-------------|-------------|--------------|--------------|
|                          | D1     | D2     | D3     | C1          | C2          | C3          | M1           | M2           |
| Nomenclature             |        |        |        |             |             |             |              |              |
| Gene_location            | 74     | 94     | 96     | 98          | 90          | 91          | 88           | 98           |
| SNP_location             | 50     | 100    | 100    | 100         | 100         | 100         | 92           | 100          |
| Genomic location         |        |        |        |             |             |             |              |              |
| Gene_SNP_association     | 6      | 100    | 98     | 100         | 6           | 100         | 94           | 100          |
| Gene_disease_association | 82     | 60     | 84     | 84          | 84          | 36          | 70           | 86           |
| Protein_coding_genes     | 20     | 100    | 100    | 100         | 18          | 100         | 100          | 100          |
| Functional analysis      |        |        |        |             |             |             |              |              |
| Gene_name_conversion     | 64     | 70     | 62     | 70          | 66          | 32          | 64           | 71           |
| Gene_alias               | 100    | 100    | 100    | 100         | 98          | 80          | 92           | 100          |

- · LLM (General LLMs): We select GPT-3.5Turbo (175B), Llama3-8B (8B), Llama-70B (70B) as representative baselines.
- · BioLLM (Biological LLMs): PMC-Llama (13B) (Wu et al., 2024) and BioMistral (7B) (Labrak et al., 2024) are two medical LLMs. They are pre-trained on open-source biomedical texts.
- · SciRAG (Scientific RAG-LLM framework): GeneGPT (175B) (Jin et al., 2024) is a biological RAG-LLM framework that integrates the NCBI databases, i.e., Gene, dbSNP, OIMI, and Blast. NewBing 8 (&gt;400B) is a retrievalaugmented LLM that has access to relevant web pages retrieved by Bing.

## 3.3 Experimental Settings

We take the Llama3-70B as the basic language model of BIORAG. For our embedding model M emb, we take AdamW as the optimizer and finetune 2 epochs. The number of retrieved results by biological databases, search engines, and local PubMed databases are set to 10, 10, and 4, respectively. The max iteration of self-evaluation is set to 15. If the model does not output the final answer within 15 times, BIORAG stops the iteration and outputs the current wrong answer. We use the accuracy to verify the overall performance. For the GeneTuring dataset, we only consider exact matches between model predictions and the ground truth as correct predictions for all nomenclature and genomics location tasks. For the gene-disease association task, we measure the recall as in the

8 https://www.bing.com/chat

original dataset but based on exact individual gene matches. For the protein-coding genes task, we consider exact matches as correct after applying a simple vocabulary mapping that converts modelpredicted "yes" / "no" to "TRUE" / "NA" and Latin species names to their informal names, respectively. The final answer of other datasets is "yes" / "no".

## 3.4 Results on Biological-related Tasks

To verify the effectiveness of the proposed model, we first conduct biological QA tasks. Results are shown in Table 2. We conclude with the following findings: (1) Based on the results of BioLLMs and GPT-3.5, we conclude that fine-tuning domainspecific data is helpful for domain-specific tasks. As the size of BioLLMs is much smaller than GPT3.5, their performance is on par with GPT-3.5. (2) BIORAG performs better than BioLLMs and GPT3.5, it indicates the effectiveness of local and external data sources. (3) Though the size of BIORAG is much smaller than SciRAG (NewBing), it has better performance. The gain comes from two aspects. The first one is our customized prompts. The second aspect lies in the local and external information sources. NewBing has no access to specialized databases and lacks technical biological descriptions for reasoning. (4) GeneGPT scores 0% accuracy in this task, because it is a customized model for the GeneTuring dataset, resulting in poor generalization capabilities.

## 3.5 Specialized Biological Reasoning Results

The GeneTuring dataset contains more specialized biological questions, and the corresponding reasoning process highly relies on technical biological corpus and descriptions. Results are shown in Ta-

Figure 5: A case study selected from the College Biology dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01107v2-figure-c72f06e9-1027-400d-8b5c-6d8d6129fb10.jpg)

ble 1. As this dataset does not contain the train data, BioLLMs are performed directly without finetuning. Their bad results indicate their poor generalization. In this dataset, we focus on the analyses of GeneGPT, NewBing, and BIORAG (1) For the nomenclature tasks , the performance of BIORAG and GeneGPT rank first and second respectively, as both of them have access to the Gene database. BIORAG integrates the results of search engines while GeneGPT does not, and this brings the gap. (2) The reasoning behind genomic location tasks relies on the highly specialized Gene and dbSNP database. BIORAG and GeneGPT achieve 100% accuracy in the gene SNP association sub-task, as

Figure 6: A case study conducted on the gene alias task in the GeneTuring dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01107v2-figure-80f3041c-c737-4a34-8bbe-f6867d83dc3e.jpg)

both of them have access to the dbSNP database. However, NewBing has no access to the dbSNP database, thus it gets 0% accuracy in this task. For the gene location subtask, the challenge is the variants of gene names. The interface of GeneGPT does not support advanced search, thus the retrieved names are not comprehensive. In contrast, general search engines, such as NewBing, have better retrieved results when the query entity has variants or ambiguities. Thus NewBing has a better performance in this task than GeneGPT. BIORAG supports the above two kinds of interfaces, and achieves the best results in this task. (3) Functional analysis tasks rely on both the Gene database and

Figure 7: A case study conducted on the gene disease association task in GeneTuring dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01107v2-figure-4136c33a-3cf4-4a80-809c-b85b0866c04b.jpg)

relative PubMed papers. The PubMed corpus provides detailed gene-disease relationships. Although NewBing retrieves the metadata, BIORAG combines the local PubMed database with other specialized databases to achieve the best results.

## 3.6 Ablation Study

To evaluate the contribution of each component of BIORAG, we performed an extensive ablation study using the GeneTuring dataset, systematically removing individual components to assess their impact on performance across various tasks. This study was designed to isolate the effects of different databases, components, and base models, with the experiments categorized as follows: (1) Databases : We consider three variations to evaluate the effectiveness of each data sources of our database: D1 : BIORAGwithout the Gene database; D2 : BIORAGwithout general search engines. D3 : BIORAGwithout the local PubMed database. (2) Model Components : We investigate the impact of specific components of our proposed framework: C1 : BIORAGwithout the MeSH Filter; C2 : BIORAGwithout the Query Rewrite component; C3 : BIORAGwithout the Self-Evaluation mechanism. (3) Base Models : We compare the performance when using two different base LLM models: M1 : take Llama-3-8B as the basic LLM, and M2 : take Llama-3-70B as the basic LLM of BioRAG.

Based on the results of ablation study, we highlights the following key findings: (1) Impact of Databases: The results indicate that the Gene database (D1) plays a crucial role in performance. For instance, the accuracy significantly drops in tasks such as Gene\_location when this component is removed. The general search engines (D2) and local PubMed database (D3) also contribute positively, but their impact is less pronounced compared to the Gene database. (2) Component Contributions: Among the components, the SelfEvaluation mechanism (C3) is vital for maintaining high accuracy across most tasks. The MeSH Filter (C1) and Query Rewrite (C2) also enhance performance, but their absence does not degrade the results as severely as the removal of Self-Evaluation. (3) Effects of Basic Language Models: Comparing the two base models, Llama-3-70B (M2) generally outperforms Llama-3-8B (M1) across all tasks, indicating that the larger model size contributes to better handling of complex biological queries. These findings underscore the importance of integrating diverse data sources and advanced compo- nents within the BIORAG framework to achieve optimal performance in biological question reasoning tasks. By understanding the contribution of each component, we can better optimize BIORAG for different tasks and datasets.

## 3.7 Case Study

To compare reasoning differences among BIORAG and the baselines in a more intuitive manner, we select three typical case studies in this section. We first provide a case study to show the workflow of BIORAG (Figure 5). It is selected from the College Biology dataset. BIORAG performs self-evaluation twice: the first time it starts with a web search for general information, but the results are insufficient to support answering the question. Thus BIORAG conducts the second self-evaluation and calls for the more specialized PubMed database. The results this time are accurate and sufficient to support answering the question, thus BIORAG gives the final answer based on the results.

The second case study is conducted on the gene alias task in the GeneTuring dataset (Figure 6). The challenge of this task is the variants of gene names. NewBing gets the response from the Wikimedia. However, Wikimedia is not specialized enough to provide the alias for the input gene, which leads to the wrong answer. The prompts of GeneGPT are too complicated, none of the prompts is relevant to this task. In addition, its NCBI API returns the gene IDs, instead of the gene names. The LLM is unable to understand these IDs, and finally arrives at a wrong answer. BIORAG employs fuzzy queries, yielding a larger number of related responses with a higher error tolerance. Furthermore, each result contains detailed gene-related information and descriptions, such as the aliases. Thus BIORAG gets the correct answer.

The third case study is conducted on the genedisease association task in the GeneTuring dataset, shown in Figure 7. Reasoning behind this task relies on both the Gene database and relative PubMed papers. The PubMed abstracts provide detailed gene-disease relationships. NewBing gets the response from the Geekymedics website. Although the Geekymedics website provides general medical information, it does not offer the correct or specific details required for gene-disease associations. Consequently, NewBing's response is inaccurate due to the reliance on a non-specialized source. GeneGPT chose the wrong NCBI API. The API's feedback is a complicated and interminable HTML page, with massive irrelevant information or descriptions. Based on the ambiguous backgrounds, GeneGPT outputs the wrong answer. In the reasoning process of BIORAG, BioRAG uses multiple tools, i.e., Gene database, local PubMed database, and Web search, to gather and conduct mutual confirmation on the information of genes associated with B-cell immunodeficiency. The process involves preprocessing queries, executing searches, and conducting self-evaluations at each step to ensure comprehensive and accurate results. The reasoning process is thorough, incorporating various data sources to confirm the association of specific genes with B-cell immunodeficiency.

## 4 Conclusion

This paper introduces BIORAG, an innovative framework that integrates Retrieval-Augmented Generation with Large Language Models to enhance biological question-reasoning. The framework's ability to obtain relevant and current information from a blend of traditional databases, toolkits, and modern search engines ensures the accuracy of the generated answers. Through extensive validation, including rigorous testing on widely recognized biology QA datasets and extensive case studies, BIORAG has demonstrated its superior ability to handle complex biological queries. These results underscore the framework's potential as a valuable tool for the scientific community, facilitating more accurate and efficient information processing.

## References

Peter William Atkins, George Ratcliffe, Julio de Paula, and Mark Wormald. 2023. Physical chemistry for the life sciences . Oxford University Press.

Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. 2023. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports , 13(1):7240.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Apretrained language model for scientific text. arXiv preprint arXiv:1903.10676 .

Letícia MF Bertoline, Angélica N Lima, Jose E Krieger, and Samantha K Teixeira. 2023. Before and after alphafold2: An overview of protein structure prediction. Frontiers in Bioinformatics , 3:1120370.

Cayque Monteiro Castro Nascimento and André Silva Pimentel. 2023. Do large language models understand chemistry? a conversation with chatgpt. Journal of Chemical Information and Modeling , 63(6):1649-1655.

Benjamin Cole, Dominique Bergmann, Crysten E Blaby-Haas, Ian K Blaby, Kristofer E Bouchard, Siobhan M Brady, Doina Ciobanu, Devin ColemanDerr, Samuel Leiboff, Jenny C Mortimer, et al. 2021. Plant single-cell solutions for energy and the environment. Communications biology , 4(1):962.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) , 3(1):1-23.

Zhicheng Guo, Sijie Cheng, Yile Wang, Peng Li, and Yang Liu. 2023. Prompt-guided retrieval augmentation for non-knowledge-intensive tasks. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 10896-10912.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 .

Andreas Holzinger, Katharina Keiblinger, Petr Holub, Kurt Zatloukal, and Heimo Müller. 2023. Ai for life: Trends in artificial intelligence for biotechnology. New Biotechnology , 74:16-24.

Wenpin Hou and Zhicheng Ji. 2023. Geneturing tests gpt models in genomics. BioRxiv .

Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. 2024. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics , 40(2):btae075.

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, PierreAntoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of opensource pretrained large language models for medical domains. arXiv preprint arXiv:2402.10373 .

Augustin Lecler, Loïc Duron, and Philippe Soyer. 2023. Revolutionizing radiology with gpt-based models: current applications, future possibilities and limitations of chatgpt. Diagnostic and Interventional Imaging , 104(6):269-274.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234-1240.

Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine , 388(13):1233-1239.

Dominique Lepore, Koustabh Dolui, Oleksandr Tomashchuk, Heereen Shim, Chetanya Puri, Yuan Li, Nuoya Chen, and Francesca Spigarelli. 2023. Interdisciplinary research unlocking innovative solutions in healthcare. Technovation , 120:102511.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llavamed: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems , 36.

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. 2021. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 .

Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, and Tianyu Du. 2024. Ra-isf: Learning to answer and understand from retrieval augmentation via iterative self-feedback. arXiv preprint arXiv:2403.06840 .

Qingqing Long, Yilun Jin, Yi Wu, and Guojie Song. 2021a. Theoretically improving graph neural networks via anonymous walk graph kernels. In Proceedings of the Web Conference 2021 , pages 12041214.

Qingqing Long, Lingjun Xu, Zheng Fang, and Guojie Song. 2021b. Hgk-gnn: Heterogeneous graph kernel based graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &amp; Data Mining , pages 1129-1138.

Zooey Nguyen, Anthony Annunziata, Vinh Luong, Sang Dinh, Quynh Le, Anh Hai Ha, Chanh Le, Hong An Phan, Shruti Raghavan, and Christopher Nguyen. 2024. Enhancing q&amp;a with domain-specific finetuning and iterative reasoning: A comparative study. arXiv preprint arXiv:2404.11792 .

Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613 .

Bob O'Donnell. 2023. New bing brings ai to search engine. USA Today , pages 01B-01B.

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning , pages 248-260. PMLR.

Conrad L Schoch, Stacy Ciufo, Mikhail Domrachev, Carol L Hotton, Sivakumar Kannan, Rogneda Khovanskaya, Detlef Leipe, Richard Mcveigh, Kathleen O'Neill, Barbara Robbertse, et al. 2020. Ncbi taxonomy: a comprehensive update on curation, resources and tools. Database , 2020:baaa062.

Guojie Song, Qingqing Long, Yi Luo, Yiming Wang, and Yilun Jin. 2020. Deep convolutional neural network based medical concept normalization. IEEE Transactions on Big Data , 8(5):1195-1208.

Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. Self-knowledge guided retrieval augmentation for large language models. arXiv preprint arXiv:2310.05002 .

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. 2024. Pmc-llama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association , page ocae045.

Jasper Xian, Tommaso Teofili, Ronak Pradeep, and Jimmy Lin. 2024. Vector search with openai embeddings: Lucene is all you need. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining , pages 1090-1093.

Meng Xiao, Ziyue Qiao, Yanjie Fu, Hao Dong, Yi Du, Pengyang Wang, Hui Xiong, and Yuanchun Zhou. 2023. Hierarchical interdisciplinary topic detection model for research proposal classification. IEEE Transactions on Knowledge and Data Engineering .

Meng Xiao, Min Wu, Ziyue Qiao, Yanjie Fu, Zhiyuan Ning, Yi Du, and Yuanchun Zhou. Interdisciplinary fairness in imbalanced research proposal topic inference: A hierarchical transformer-based method with selective interpolation. ACM Transactions on Knowledge Discovery from Data .

Fangkai Yang, Pu Zhao, Zezhong Wang, Lu Wang, Jue Zhang, Mohit Garg, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. Empower large language model to perform better on industrial domain-specific question answering. arXiv preprint arXiv:2305.11541 .
Chengrui Wang Qingqing Long Meng Xiao Xunxin Cai Chengjun Wu Zhen Meng Xuezhi Wang Yuanchun Zhou
BioRAG: A RAG-LLM Framework for Biological Question Reasoning
The question-answering system for Life science research, which is
characterized by the rapid pace of discovery, evolving insights, and complex
interactions among knowledge entities, presents unique challenges in
maintaining a comprehensive knowledge warehouse and accurate information
retrieval. To address these issues, we introduce BioRAG, a novel
Retrieval-Augmented Generation (RAG) with the Large Language Models (LLMs)
framework. Our approach starts with parsing, indexing, and segmenting an
extensive collection of 22 million scientific papers as the basic knowledge,
followed by training a specialized embedding model tailored to this domain.
Additionally, we enhance the vector retrieval process by incorporating a
domain-specific knowledge hierarchy, which aids in modeling the intricate
interrelationships among each query and context. For queries requiring the most
current information, BioRAG deconstructs the question and employs an iterative
retrieval process incorporated with the search engine for step-by-step
reasoning. Rigorous experiments have demonstrated that our model outperforms
fine-tuned LLM, LLM with search engines, and other scientific RAG frameworks
across multiple life science question-answering tasks.