Formulation Comparison for Timeline Construction using LLMs
<|reference_start|>Formulation Comparison for Timeline Construction using LLMs: Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information. To mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of single-document timelines with document-level order annotation. TimeSET features saliency-based event selection and partial ordering, which enable a practical annotation workload. Aiming to build better automatic timeline construction systems, we propose a novel evaluation framework to compare multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2 and Flan-T5. Considering that identifying temporal orders of events is a core subtask in timeline construction, we further benchmark open LLMs on existing event temporal ordering datasets to gain a robust understanding of their capabilities. Our experiments show that (1) NLI formulation with Flan-T5 demonstrates a strong performance among others, while (2) timeline construction and event temporal ordering are still challenging tasks for few-shot LLMs. Our code and data are available at https://github.com/kimihiroh/timeset.<|reference_end|>
@article{hasegawa2024formulation,
    title={Formulation Comparison for Timeline Construction using LLMs},
    author={Kimihiro Hasegawa, Nikhil Kandukuri, Susan Holm, Yukari Yamakawa,
  Teruko Mitamura},
    journal={arXiv preprint arXiv:2403.00990},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00990},
    primaryClass={cs.CL}
}