title: A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them
    In Zero Shot
summary: Multimedia content, such as advertisements and story videos, exhibit a rich
  blend of creativity and multiple modalities. They incorporate elements like
  text, visuals, audio, and storytelling techniques, employing devices like
  emotions, symbolism, and slogans to convey meaning. While previous research in
  multimedia understanding has focused mainly on videos with specific actions
  like cooking, there is a dearth of large annotated training datasets, hindering
  the development of supervised learning models with satisfactory performance for
  real-world applications. However, the rise of large language models (LLMs) has
  witnessed remarkable zero-shot performance in various natural language
  processing (NLP) tasks, such as emotion classification, question-answering, and
  topic classification. To bridge this performance gap in multimedia
  understanding, we propose verbalizing story videos to generate their
  descriptions in natural language and then performing video-understanding tasks
  on the generated story as opposed to the original video. Through extensive
  experiments on five video-understanding tasks, we demonstrate that our method,
  despite being zero-shot, achieves significantly better results than supervised
  baselines for video understanding. Further, alleviating a lack of story
  understanding benchmarks, we publicly release the first dataset on a crucial
  task in computational social science, persuasion strategy identification.
paper_authors._id: 646582770e6c7618f61506c5
paper_authors.name: Aanisha Bhattacharya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T15:14:32.432Z
paper_authors._id: 60a7d1ca23ce37179774a210
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651500117101-60a7d1ca23ce37179774a210.jpeg
paper_authors.fullname: Aanisha Bhattacharyya
paper_authors.type: user
paper_authors.user: Aanisha
paper_authors._id: 646582770e6c7618f61506c6
paper_authors.name: Yaman K Singla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T09:37:00.156Z
paper_authors._id: 62be9baeae56e75e4d68a289
paper_authors.avatarUrl: /avatars/6a9415a05b4c217433bdb1b414ccd221.svg
paper_authors.fullname: Yaman
paper_authors.type: user
paper_authors.user: yamanksingla
paper_authors._id: 646582770e6c7618f61506c7
paper_authors.name: Balaji Krishnamurthy
paper_authors._id: 646582770e6c7618f61506c8
paper_authors.name: Rajiv Ratn Shah
paper_authors._id: 646582770e6c7618f61506c9
paper_authors.name: Changyou Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:03:42.355Z
paper_authors._id: 64678f41946476c5d2170609
paper_authors.avatarUrl: /avatars/deadb2c5552b314f3a14c89a9b353769.svg
paper_authors.fullname: Changyou Chen
paper_authors.type: user
paper_authors.user: cchangyou
paper_title: A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them
    In Zero Shot
paper_summary: Multimedia content, such as advertisements and story videos, exhibit a rich
  blend of creativity and multiple modalities. They incorporate elements like
  text, visuals, audio, and storytelling techniques, employing devices like
  emotions, symbolism, and slogans to convey meaning. While previous research in
  multimedia understanding has focused mainly on videos with specific actions
  like cooking, there is a dearth of large annotated training datasets, hindering
  the development of supervised learning models with satisfactory performance for
  real-world applications. However, the rise of large language models (LLMs) has
  witnessed remarkable zero-shot performance in various natural language
  processing (NLP) tasks, such as emotion classification, question-answering, and
  topic classification. To bridge this performance gap in multimedia
  understanding, we propose verbalizing story videos to generate their
  descriptions in natural language and then performing video-understanding tasks
  on the generated story as opposed to the original video. Through extensive
  experiments on five video-understanding tasks, we demonstrate that our method,
  despite being zero-shot, achieves significantly better results than supervised
  baselines for video understanding. Further, alleviating a lack of story
  understanding benchmarks, we publicly release the first dataset on a crucial
  task in computational social science, persuasion strategy identification.
paper_ai_summary: The proposed method of verbalizing story videos into natural language descriptions improves zero-shot performance in video understanding tasks, surpassing supervised baselines.
paper_ai_keywords: large language models zero-shot performance natural language processing video-understanding tasks zero-shot supervised baselines story-understanding benchmarks persuasion strategy identification