Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance
<|reference_start|>Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance: Relational databases often suffer from uninformative descriptors of table contents, such as ambiguous columns and hard-to-interpret values, impacting both human users and text-to-SQL models. In this paper, we explore the use of large language models (LLMs) to automatically generate detailed natural language descriptions for SQL database columns, aiming to improve text-to-SQL performance and automate metadata creation. We create a dataset of gold column descriptions based on the BIRD-Bench benchmark, manually refining its column descriptions and creating a taxonomy for categorizing column difficulty. Through evaluating several LLMs, we find that incorporating these column descriptions consistently enhances text-to-SQL model performance, particularly for larger models like GPT-4o, Qwen2 72B and Mixtral 22Bx8. However, models struggle with columns that exhibit inherent ambiguity, highlighting the need for manual expert input. Notably, Qwen2-generated descriptions, containing by annotators deemed superfluous information, outperform manually curated gold descriptions, suggesting that models benefit from more detailed metadata than humans expect. Future work will investigate the specific features of these high-performing descriptions and explore other types of metadata, such as numerical reasoning and synonyms, to further improve text-to-SQL systems. The dataset, annotations and code will all be made available.<|reference_end|>
@article{wretblad2024synthetic,
    title={Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL
  Performance},
    author={Niklas Wretblad, Oskar Holmstr"om, Erik Larsson, Axel Wiks"ater,
  Oscar S"oderlund, Hjalmar "Ohman, Ture Pont'en, Martin Forsberg, Martin
  S"orme, Fredrik Heintz},
    journal={arXiv preprint arXiv:2408.04691},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04691},
    primaryClass={cs.CL cs.AI cs.DB}
}