title: MovieChat: From Dense Token to Sparse Memory for Long Video
    Understanding
summary: Recently, integrating video foundation models and large language models to
  build a video understanding system overcoming the limitations of specific
  pre-defined vision tasks. Yet, existing systems can only handle videos with
  very few frames. For long videos, the computation complexity, memory cost, and
  long-term temporal connection are the remaining challenges. Inspired by
  Atkinson-Shiffrin memory model, we develop an memory mechanism including a
  rapidly updated short-term memory and a compact thus sustained long-term
  memory. We employ tokens in Transformers as the carriers of memory. MovieChat
  achieves state-of-the-art performace in long video understanding.
paper_authors._id: 64c86c213137cc529dddb414
paper_authors.name: Enxin Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:30:28.314Z
paper_authors._id: 63a7b196e27a6dbd4861e275
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a7b196e27a6dbd4861e275/QGn4hdDPNuSRgZVKxzDpq.jpeg
paper_authors.fullname: EnxinSong
paper_authors.type: user
paper_authors.user: Enxin
paper_authors._id: 64c86c213137cc529dddb415
paper_authors.name: Wenhao Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:33:29.799Z
paper_authors._id: 637c7503fe115289cfecbe6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
paper_authors.fullname: Wenhao Chai
paper_authors.type: user
paper_authors.user: wchai
paper_authors._id: 64c86c213137cc529dddb416
paper_authors.name: Guanhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:34:01.719Z
paper_authors._id: 64b621c25b4a27b5dd1a252d
paper_authors.avatarUrl: /avatars/93d984b29177e083e1416747005a3520.svg
paper_authors.fullname: Guanhong Wang
paper_authors.type: user
paper_authors.user: Rejoicelf
paper_authors._id: 64c86c213137cc529dddb417
paper_authors.name: Yucheng Zhang
paper_authors._id: 64c86c213137cc529dddb418
paper_authors.name: Haoyang Zhou
paper_authors._id: 64c86c213137cc529dddb419
paper_authors.name: Feiyang Wu
paper_authors._id: 64c86c213137cc529dddb41a
paper_authors.name: Xun Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:35:13.711Z
paper_authors._id: 64ae7ddf407a5cae8579c171
paper_authors.avatarUrl: /avatars/f78e8958db16f5f5603ece527951ac23.svg
paper_authors.fullname: Xun Guo
paper_authors.type: user
paper_authors.user: xunguohf
paper_authors._id: 64c86c213137cc529dddb41b
paper_authors.name: Tian Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:21:26.183Z
paper_authors._id: 66015e8aa4d296af07de538e
paper_authors.avatarUrl: /avatars/a1295c631cc2646282c545859975ce4c.svg
paper_authors.fullname: Ye
paper_authors.type: user
paper_authors.user: Owen777
paper_authors._id: 64c86c213137cc529dddb41c
paper_authors.name: Yan Lu
paper_authors._id: 64c86c213137cc529dddb41d
paper_authors.name: Jenq-Neng Hwang
paper_authors._id: 64c86c213137cc529dddb41e
paper_authors.name: Gaoang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T13:08:52.689Z
paper_authors._id: 64c9033c5381684d3eaac7f1
paper_authors.avatarUrl: /avatars/07d36ca193826044b0df04e3602b9ef8.svg
paper_authors.fullname: Gaoang Wang
paper_authors.type: user
paper_authors.user: GaoangWang
paper_title: MovieChat: From Dense Token to Sparse Memory for Long Video
    Understanding
paper_summary: Recently, integrating video foundation models and large language models to
  build a video understanding system overcoming the limitations of specific
  pre-defined vision tasks. Yet, existing systems can only handle videos with
  very few frames. For long videos, the computation complexity, memory cost, and
  long-term temporal connection are the remaining challenges. Inspired by
  Atkinson-Shiffrin memory model, we develop an memory mechanism including a
  rapidly updated short-term memory and a compact thus sustained long-term
  memory. We employ tokens in Transformers as the carriers of memory. MovieChat
  achieves state-of-the-art performace in long video understanding.
paper_ai_summary: A video understanding system using a dual-memory mechanism based on the Atkinson-Shiffrin model, leveraging Transformers, achieves leading performance on long videos.
paper_ai_keywords: video foundation models large language models long videos computation complexity memory cost long-term temporal connection Atkinson-Shiffrin memory model short-term memory long-term memory Transformers MovieChat