Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset
<|reference_start|>Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset: The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game commentary is not enough to fully understand the game situation. It will be richer by including diverse multimodal esports information, including audiences' talks/emotions, game audio, and game match event information. This paper introduces GAME-MUG, a new multimodal game situation understanding and audience-engaged commentary generation dataset and its strong baseline. Our dataset is collected from 2020-2022 LOL game live streams from YouTube and Twitch, and includes multimodal esports game information, including text, audio, and time-series event logs, for detecting the game situation. In addition, we also propose a new audience conversation augmented commentary dataset by covering the game situation and audience conversation understanding, and introducing a robust joint multimodal dual learning model as a baseline. We examine the model's game situation/event understanding ability and commentary generation capability to show the effectiveness of the multimodal aspects coverage and the joint integration learning approach.<|reference_end|>
@article{zhang2024game-mug:,
    title={Game-MUG: Multimodal Oriented Game Situation Understanding and
  Commentary Generation Dataset},
    author={Zhihao Zhang, Feiqi Cao, Yingbin Mo, Yiran Zhang, Josiah Poon, Caren
  Han},
    journal={arXiv preprint arXiv:2404.19175},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.19175},
    primaryClass={cs.CL}
}