Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization
<|reference_start|>Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization: Recent Large Language Models (LLMs) have demonstrated impressive capabilities at tasks that require human intelligence and are a significant step towards human-like artificial intelligence (AI). Yet the performance of LLMs at reasoning tasks have been subpar and the reasoning capability of LLMs is a matter of significant debate. While it has been shown that the choice of the prompting technique to the LLM can alter its performance on a multitude of tasks, including reasoning, the best performing techniques require human-made prompts with the knowledge of the tasks at hand. We introduce a framework for what we call Combinatorial Reasoning (CR), a fully-automated prompting method, where reasons are sampled from an LLM pipeline and mapped into a Quadratic Unconstrained Binary Optimization (QUBO) problem. The framework investigates whether QUBO solutions can be profitably used to select a useful subset of the reasons to construct a Chain-of-Thought style prompt. We explore the acceleration of CR with specialized solvers. We also investigate the performance of simpler zero-shot strategies such as linear majority rule or random selection of reasons. Our preliminary study indicates that coupling a combinatorial solver to generative AI pipelines is an interesting avenue for AI reasoning and elucidates design principles for future CR methods.<|reference_end|>
@article{esencan2024combinatorial,
    title={Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines
  via Combinatorial Optimization},
    author={Mert Esencan, Tarun Advaith Kumar, Ata Akbari Asanjan, P. Aaron Lott,
  Masoud Mohseni, Can Unlu, Davide Venturelli, Alan Ho},
    journal={arXiv preprint arXiv:2407.00071},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.00071},
    primaryClass={cs.AI cs.CL cs.ET cs.LG}
}