Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models
<|reference_start|>Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models: We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.<|reference_end|>
@article{pan2024chain-of-action:,
    title={Chain-of-Action: Faithful and Multimodal Question Answering through
  Large Language Models},
    author={Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu},
    journal={arXiv preprint arXiv:2403.17359},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.17359},
    primaryClass={cs.CL}
}