Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A
<|reference_start|>Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A: While LLMs in the RAG paradigm have shown remarkable performance on a variety of tasks, they still under-perform on unseen domains, especially on complex tasks like procedural question answering. In this work, we introduce a novel formalism and structure for manipulating text-based procedures. Based on this formalism, we further present a novel dataset called LCStep, scraped from the LangChain Python docs. Moreover, we extend the traditional RAG system to propose a novel system called analogy-augmented generation (AAG), that draws inspiration from human analogical reasoning and ability to assimilate past experiences to solve unseen problems. The proposed method uses a frozen language model with a custom procedure memory store to adapt to specialized knowledge. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG.<|reference_end|>
@article{roth2024pairing,
    title={Pairing Analogy-Augmented Generation with Procedural Memory for
  Procedural Q&A},
    author={K Roth and Rushil Gupta and Simon Halle and Bang Liu},
    journal={arXiv preprint arXiv:2409.01344},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01344},
    primaryClass={cs.AI cs.CL}
}