Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports
<|reference_start|>Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports: As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization.<|reference_end|>
@article{cao2024characterizing,
    title={Characterizing Multimodal Long-form Summarization: A Case Study on
  Financial Reports},
    author={Tianyu Cao, Natraj Raman, Danial Dervovic, Chenhao Tan},
    journal={arXiv preprint arXiv:2404.06162},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06162},
    primaryClass={cs.CL cs.AI cs.LG}
}