title: Can LLM Already Serve as A Database Interface? A BIg Bench for
    Large-Scale Database Grounded Text-to-SQLs
summary: Text-to-SQL parsing, which aims at converting natural language instructions
  into executable SQLs, has gained increasing attention in recent years. In
  particular, Codex and ChatGPT have shown impressive results in this task.
  However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
  database schema with few rows of database contents leaving the gap between
  academic study and real-world applications. To mitigate this gap, we present
  Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
  containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
  of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
  highlights the new challenges of dirty database contents, external knowledge
  between NL questions and database contents, and SQL efficiency, particularly in
  the context of massive databases. To solve these problems, text-to-SQL models
  must feature database value comprehension in addition to semantic parsing. The
  experimental results demonstrate the significance of database values in
  generating accurate text-to-SQLs for big databases. Furthermore, even the most
  effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
  accuracy, which is still far from the human result of 92.96%, proving that
  challenges still stand. Besides, we also provide an efficiency analysis to
  offer insights into generating text-to-efficient-SQLs that are beneficial to
  industries. We believe that BIRD will contribute to advancing real-world
  applications of text-to-SQL research. The leaderboard and source code are
  available: https://bird-bench.github.io/.
paper_authors._id: 645845588eef89b5c3c1c878
paper_authors.name: Jinyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:06:17.085Z
paper_authors._id: 647c09e942b0647cd745f373
paper_authors.avatarUrl: /avatars/a75bfd25e84c70cdc64bcbc9d2b3d5f9.svg
paper_authors.fullname: Malou Lab
paper_authors.type: user
paper_authors.user: patrickNLP
paper_authors._id: 645845588eef89b5c3c1c879
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-25T07:23:08.538Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 645845588eef89b5c3c1c87a
paper_authors.name: Ge Qu
paper_authors._id: 645845588eef89b5c3c1c87b
paper_authors.name: Binhua Li
paper_authors._id: 645845588eef89b5c3c1c87c
paper_authors.name: Jiaxi Yang
paper_authors._id: 645845588eef89b5c3c1c87d
paper_authors.name: Bowen Li
paper_authors._id: 645845588eef89b5c3c1c87e
paper_authors.name: Bailin Wang
paper_authors._id: 645845588eef89b5c3c1c87f
paper_authors.name: Bowen Qin
paper_authors._id: 645845588eef89b5c3c1c880
paper_authors.name: Rongyu Cao
paper_authors._id: 645845588eef89b5c3c1c881
paper_authors.name: Ruiying Geng
paper_authors._id: 645845588eef89b5c3c1c882
paper_authors.name: Nan Huo
paper_authors._id: 645845588eef89b5c3c1c883
paper_authors.name: Chenhao Ma
paper_authors._id: 645845588eef89b5c3c1c884
paper_authors.name: Kevin C. C. Chang
paper_authors._id: 645845588eef89b5c3c1c885
paper_authors.name: Fei Huang
paper_authors._id: 645845588eef89b5c3c1c886
paper_authors.name: Reynold Cheng
paper_authors._id: 645845588eef89b5c3c1c887
paper_authors.name: Yongbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:48:41.773Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_title: Can LLM Already Serve as A Database Interface? A BIg Bench for
    Large-Scale Database Grounded Text-to-SQLs
paper_summary: Text-to-SQL parsing, which aims at converting natural language instructions
  into executable SQLs, has gained increasing attention in recent years. In
  particular, Codex and ChatGPT have shown impressive results in this task.
  However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
  database schema with few rows of database contents leaving the gap between
  academic study and real-world applications. To mitigate this gap, we present
  Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
  containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
  of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
  highlights the new challenges of dirty database contents, external knowledge
  between NL questions and database contents, and SQL efficiency, particularly in
  the context of massive databases. To solve these problems, text-to-SQL models
  must feature database value comprehension in addition to semantic parsing. The
  experimental results demonstrate the significance of database values in
  generating accurate text-to-SQLs for big databases. Furthermore, even the most
  effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
  accuracy, which is still far from the human result of 92.96%, proving that
  challenges still stand. Besides, we also provide an efficiency analysis to
  offer insights into generating text-to-efficient-SQLs that are beneficial to
  industries. We believe that BIRD will contribute to advancing real-world
  applications of text-to-SQL research. The leaderboard and source code are
  available: https://bird-bench.github.io/.
paper_ai_keywords: text-to-SQL parsing Codex ChatGPT SQL Spider WikiSQL benchmark database text-to-SQL data database values semantic parsing database value comprehension SQL efficiency execution accuracy efficiency analysis big databases professional domains gold descriptions text-to-efficient-SQLs