title: BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge
    Bases
summary: Biomedical researchers increasingly rely on large-scale structured databases
  for complex analytical tasks. However, current text-to-SQL systems often
  struggle to map qualitative scientific questions into executable SQL,
  particularly when implicit domain reasoning is required. We introduce
  BiomedSQL, the first benchmark explicitly designed to evaluate scientific
  reasoning in text-to-SQL generation over a real-world biomedical knowledge
  base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in
  a harmonized BigQuery knowledge base that integrates gene-disease associations,
  causal inference from omics data, and drug approval records. Each question
  requires models to infer domain-specific criteria, such as genome-wide
  significance thresholds, effect directionality, or trial phase filtering,
  rather than rely on syntactic translation alone. We evaluate a range of open-
  and closed-source LLMs across prompting strategies and interaction paradigms.
  Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0%
  execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%,
  both well below the expert baseline of 90.0%. BiomedSQL provides a new
  foundation for advancing text-to-SQL systems capable of supporting scientific
  discovery through robust reasoning over structured biomedical knowledge bases.
  Our dataset is publicly available at
  https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source
  at https://github.com/NIH-CARD/biomedsql.
paper_authors._id: 68370a124a68a691a29331cb
paper_authors.name: Mathew J. Koretsky
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:08.928Z
paper_authors._id: 672e33ad6ff4cb430622f915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_HBL0daPqqdN8rNU6l25i.png
paper_authors.fullname: Mathew Koretsky
paper_authors.type: user
paper_authors.user: mkoretsky1
paper_authors._id: 68370a124a68a691a29331cc
paper_authors.name: Maya Willey
paper_authors._id: 68370a124a68a691a29331cd
paper_authors.name: Adi Asija
paper_authors._id: 68370a124a68a691a29331ce
paper_authors.name: Owen Bianchi
paper_authors._id: 68370a124a68a691a29331cf
paper_authors.name: Chelsea X. Alvarado
paper_authors._id: 68370a124a68a691a29331d0
paper_authors.name: Tanay Nayak
paper_authors._id: 68370a124a68a691a29331d1
paper_authors.name: Nicole Kuznetsov
paper_authors._id: 68370a124a68a691a29331d2
paper_authors.name: Sungwon Kim
paper_authors._id: 68370a124a68a691a29331d3
paper_authors.name: Mike A. Nalls
paper_authors._id: 68370a124a68a691a29331d4
paper_authors.name: Daniel Khashabi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:06.658Z
paper_authors._id: 5f6540c65e78cc6b0ed3199d
paper_authors.avatarUrl: /avatars/0280d4df417855965a0964d22766c012.svg
paper_authors.fullname: Daniel Khashabi
paper_authors.type: user
paper_authors.user: danyaljj
paper_authors._id: 68370a124a68a691a29331d5
paper_authors.name: Faraz Faghri
paper_title: BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge
    Bases
paper_summary: Biomedical researchers increasingly rely on large-scale structured databases
  for complex analytical tasks. However, current text-to-SQL systems often
  struggle to map qualitative scientific questions into executable SQL,
  particularly when implicit domain reasoning is required. We introduce
  BiomedSQL, the first benchmark explicitly designed to evaluate scientific
  reasoning in text-to-SQL generation over a real-world biomedical knowledge
  base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in
  a harmonized BigQuery knowledge base that integrates gene-disease associations,
  causal inference from omics data, and drug approval records. Each question
  requires models to infer domain-specific criteria, such as genome-wide
  significance thresholds, effect directionality, or trial phase filtering,
  rather than rely on syntactic translation alone. We evaluate a range of open-
  and closed-source LLMs across prompting strategies and interaction paradigms.
  Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0%
  execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%,
  both well below the expert baseline of 90.0%. BiomedSQL provides a new
  foundation for advancing text-to-SQL systems capable of supporting scientific
  discovery through robust reasoning over structured biomedical knowledge bases.
  Our dataset is publicly available at
  https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source
  at https://github.com/NIH-CARD/biomedsql.
paper_ai_summary: BiomedSQL evaluates scientific reasoning in text-to-SQL tasks using a large biomedical knowledge base, highlighting performance gaps in existing models.
paper_ai_keywords: text-to-SQL biomedical knowledge base gene-disease associations causal inference omics data drug approval records genome-wide significance effect directionality trial phase filtering multi-step agent expert baseline