Can LLM Already Serve as A Database Interface? A BIg Bench for
  Large-Scale Database Grounded Text-to-SQLs
Text-to-SQL parsing, which aims at converting natural language instructions
into executable SQLs, has gained increasing attention in recent years. In
particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
database schema with few rows of database contents leaving the gap between
academic study and real-world applications. To mitigate this gap, we present
Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
highlights the new challenges of dirty database contents, external knowledge
between NL questions and database contents, and SQL efficiency, particularly in
the context of massive databases. To solve these problems, text-to-SQL models
must feature database value comprehension in addition to semantic parsing. The
experimental results demonstrate the significance of database values in
generating accurate text-to-SQLs for big databases. Furthermore, even the most
effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
accuracy, which is still far from the human result of 92.96%, proving that
challenges still stand. Besides, we also provide an efficiency analysis to
offer insights into generating text-to-efficient-SQLs that are beneficial to
industries. We believe that BIRD will contribute to advancing real-world
applications of text-to-SQL research. The leaderboard and source code are
available: https://bird-bench.github.io/.
645845588eef89b5c3c1c878
Jinyang Li
claimed_verified
2023-06-05T10:06:17.085Z
647c09e942b0647cd745f373
/avatars/a75bfd25e84c70cdc64bcbc9d2b3d5f9.svg
Malou Lab
user
patrickNLP
645845588eef89b5c3c1c879
Binyuan Hui
admin_assigned
2023-05-25T07:23:08.538Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
645845588eef89b5c3c1c87a
Ge Qu
claimed_verified
2025-06-04T09:07:35.296Z
6419435385030eca6ac94701
/avatars/c49ff1991739de49ec98c8310ab21e46.svg
Ge Qu
user
gq2138
645845588eef89b5c3c1c87b
Binhua Li
645845588eef89b5c3c1c87c
Jiaxi Yang
645845588eef89b5c3c1c87d
Bowen Li
645845588eef89b5c3c1c87e
Bailin Wang
645845588eef89b5c3c1c87f
Bowen Qin
645845588eef89b5c3c1c880
Rongyu Cao
645845588eef89b5c3c1c881
Ruiying Geng
645845588eef89b5c3c1c882
Nan Huo
claimed_verified
2025-06-06T07:44:52.745Z
60adfff0306d6873ec42d545
/avatars/4a63f90638dbffebfeeee181a6d0220c.svg
Nan
user
NanHUO
645845588eef89b5c3c1c883
Chenhao Ma
645845588eef89b5c3c1c884
Kevin C. C. Chang
645845588eef89b5c3c1c885
Fei Huang
645845588eef89b5c3c1c886
Reynold Cheng
645845588eef89b5c3c1c887
Yongbin Li
claimed_verified
2024-10-14T10:48:41.773Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
Can LLM Already Serve as A Database Interface? A BIg Bench for
  Large-Scale Database Grounded Text-to-SQLs
Text-to-SQL parsing, which aims at converting natural language instructions
into executable SQLs, has gained increasing attention in recent years. In
particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
database schema with few rows of database contents leaving the gap between
academic study and real-world applications. To mitigate this gap, we present
Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
highlights the new challenges of dirty database contents, external knowledge
between NL questions and database contents, and SQL efficiency, particularly in
the context of massive databases. To solve these problems, text-to-SQL models
must feature database value comprehension in addition to semantic parsing. The
experimental results demonstrate the significance of database values in
generating accurate text-to-SQLs for big databases. Furthermore, even the most
effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
accuracy, which is still far from the human result of 92.96%, proving that
challenges still stand. Besides, we also provide an efficiency analysis to
offer insights into generating text-to-efficient-SQLs that are beneficial to
industries. We believe that BIRD will contribute to advancing real-world
applications of text-to-SQL research. The leaderboard and source code are
available: https://bird-bench.github.io/.
Bird, a large-scale text-to-SQL benchmark, addresses challenges posed by big databases, emphasizing database values, external knowledge, and SQL efficiency.
Text-to-SQL Codex ChatGPT database schema Spider WikiSQL big benchmark semantic parsing dirty database contents external knowledge SQL efficiency leaderboard source code