DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models
<|reference_start|>DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models: Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub.<|reference_end|>
@article{zhou2024db-gpt-hub:,
    title={DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large
  Language Models},
    author={Fan Zhou, Siqiao Xue, Danrui Qi, Wenhui Shi, Wang Zhao, Ganglin Wei,
  Hongyang Zhang, Caigai Jiang, Gangwei Jiang, Zhixuan Chu, Faqiang Chen},
    journal={arXiv preprint arXiv:2406.11434},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11434},
    primaryClass={cs.DB}
}