CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL
<|reference_start|>CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL: In tackling the challenges of large language model (LLM) performance for Text-to-SQL tasks, we introduce CHASE-SQL, a new framework that employs innovative strategies, using test-time compute in multi-agent modeling to improve candidate generation and selection. CHASE-SQL leverages LLMs' intrinsic knowledge to generate diverse and high-quality SQL candidates using different LLM generators with: (1) a divide-and-conquer method that decomposes complex queries into manageable sub-queries in a single LLM call; (2) chain-of-thought reasoning based on query execution plans, reflecting the steps a database engine takes during execution; and (3) a unique instance-aware synthetic example generation technique, which offers specific few-shot demonstrations tailored to test questions.To identify the best candidate, a selection agent is employed to rank the candidates through pairwise comparisons with a fine-tuned binary-candidates selection LLM. This selection approach has been demonstrated to be more robust over alternatives. The proposed generators-selector framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods. Overall, our proposed CHASE-SQL achieves the state-of-the-art execution accuracy of 73.0% and 73.01% on the test set and development set of the notable BIRD Text-to-SQL dataset benchmark, rendering CHASE-SQL the top submission of the leaderboard (at the time of paper submission).<|reference_end|>
@article{pourreza2024chase-sql:,
    title={CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate
  Selection in Text-to-SQL},
    author={Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan
  Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, Sercan O.
  Arik},
    journal={arXiv preprint arXiv:2410.01943},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01943},
    primaryClass={cs.LG cs.AI cs.CL cs.DB}
}