title: InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
    Long-term Streaming Video and Audio Interactions
summary: Creating AI systems that can interact with environments over long periods,
  similar to human cognition, has been a longstanding research goal. Recent
  advancements in multimodal large language models (MLLMs) have made significant
  strides in open-world understanding. However, the challenge of continuous and
  simultaneous streaming perception, memory, and reasoning remains largely
  unexplored. Current MLLMs are constrained by their sequence-to-sequence
  architecture, which limits their ability to process inputs and generate
  responses simultaneously, akin to being unable to think while perceiving.
  Furthermore, relying on long contexts to store historical data is impractical
  for long-term interactions, as retaining all information becomes costly and
  inefficient. Therefore, rather than relying on a single foundation model to
  perform all functions, this project draws inspiration from the concept of the
  Specialized Generalist AI and introduces disentangled streaming perception,
  reasoning, and memory mechanisms, enabling real-time interaction with streaming
  video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
  (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
  Processes multimodal information in real-time, storing key details in memory
  and triggering reasoning in response to user queries. (2) Multi-modal Long
  Memory Module: Integrates short-term and long-term memory, compressing
  short-term memories into long-term ones for efficient retrieval and improved
  accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
  tasks, coordinating with the perception and memory modules. This project
  simulates human-like cognition, enabling multimodal large language models to
  provide continuous and adaptive service over time.
paper_authors._id: 675ba1de7ce703a00f19b4e4
paper_authors.name: Pan Zhang
paper_authors._id: 675ba1de7ce703a00f19b4e5
paper_authors.name: Xiaoyi Dong
paper_authors._id: 675ba1de7ce703a00f19b4e6
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:31:34.553Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 675ba1de7ce703a00f19b4e7
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:29.113Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 675ba1de7ce703a00f19b4e8
paper_authors.name: Rui Qian
paper_authors._id: 675ba1de7ce703a00f19b4e9
paper_authors.name: Xilin Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:27.497Z
paper_authors._id: 62eb70462f0f5e54df42f778
paper_authors.avatarUrl: /avatars/456049dba67638d3cdb330cdf383f272.svg
paper_authors.fullname: Xilin Wei
paper_authors.type: user
paper_authors.user: Wiselnn
paper_authors._id: 675ba1de7ce703a00f19b4ea
paper_authors.name: Lin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:25.665Z
paper_authors._id: 64b02ec0e5000ae8a572ced5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
paper_authors.fullname: Lin Chen
paper_authors.type: user
paper_authors.user: Lin-Chen
paper_authors._id: 675ba1de7ce703a00f19b4eb
paper_authors.name: Yifei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:24.684Z
paper_authors._id: 64adfeac4beffa272dfaef21
paper_authors.avatarUrl: /avatars/883f6ba38b993476115dfafcef9ce3c1.svg
paper_authors.fullname: Yifei Li
paper_authors.type: user
paper_authors.user: JoeLeelyf
paper_authors._id: 675ba1de7ce703a00f19b4ec
paper_authors.name: Junbo Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-13T17:42:32.064Z
paper_authors._id: 650abbb71aece923f21d87fc
paper_authors.avatarUrl: /avatars/f09ff031c278bc42bfd7a563853e142c.svg
paper_authors.fullname: Junbo Niu
paper_authors.type: user
paper_authors.user: Niujunbo2002
paper_authors._id: 675ba1de7ce703a00f19b4ed
paper_authors.name: Shuangrui Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:31:46.441Z
paper_authors._id: 65a7c0335e79abfa2ec30c52
paper_authors.avatarUrl: /avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
paper_authors.fullname: Shuangrui Ding
paper_authors.type: user
paper_authors.user: Mar2Ding
paper_authors._id: 675ba1de7ce703a00f19b4ee
paper_authors.name: Qipeng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:31:52.372Z
paper_authors._id: 6491cd52b1e5d3444528edb1
paper_authors.avatarUrl: /avatars/a85635d886c7f157b6723dec5c01c030.svg
paper_authors.fullname: Qipeng Guo
paper_authors.type: user
paper_authors.user: QipengGuo
paper_authors._id: 675ba1de7ce703a00f19b4ef
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:31:57.973Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 675ba1de7ce703a00f19b4f0
paper_authors.name: Xin Chen
paper_authors._id: 675ba1de7ce703a00f19b4f1
paper_authors.name: Han Lv
paper_authors._id: 675ba1de7ce703a00f19b4f2
paper_authors.name: Zheng Nie
paper_authors._id: 675ba1de7ce703a00f19b4f3
paper_authors.name: Min Zhang
paper_authors._id: 675ba1de7ce703a00f19b4f4
paper_authors.name: Bin Wang
paper_authors._id: 675ba1de7ce703a00f19b4f5
paper_authors.name: Wenwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:32:53.099Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 675ba1de7ce703a00f19b4f6
paper_authors.name: Xinyue Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:33:53.973Z
paper_authors._id: 670a40f75054a602d11988f1
paper_authors.avatarUrl: /avatars/60ba3e211c73b1c6c78460aff8699584.svg
paper_authors.fullname: XinyueZhang
paper_authors.type: user
paper_authors.user: XinyueZhang1997
paper_authors._id: 675ba1de7ce703a00f19b4f7
paper_authors.name: Jiaye Ge
paper_authors._id: 675ba1de7ce703a00f19b4f8
paper_authors.name: Wei Li
paper_authors._id: 675ba1de7ce703a00f19b4f9
paper_authors.name: Jingwen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:32:16.439Z
paper_authors._id: 61740f8f8a50963ce71ffff3
paper_authors.avatarUrl: /avatars/b07b264669fce4c75e063e6392ebd74b.svg
paper_authors.fullname: 李静雯
paper_authors.type: user
paper_authors.user: Jingwen
paper_authors._id: 675ba1de7ce703a00f19b4fa
paper_authors.name: Zhongying Tu
paper_authors._id: 675ba1de7ce703a00f19b4fb
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:32:46.981Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 675ba1de7ce703a00f19b4fc
paper_authors.name: Xingcheng Zhang
paper_authors._id: 675ba1de7ce703a00f19b4fd
paper_authors.name: Kai Chen
paper_authors._id: 675ba1de7ce703a00f19b4fe
paper_authors.name: Yu Qiao
paper_authors._id: 675ba1de7ce703a00f19b4ff
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-13T17:42:17.927Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 675ba1de7ce703a00f19b500
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:46.738Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
    Long-term Streaming Video and Audio Interactions
paper_summary: Creating AI systems that can interact with environments over long periods,
  similar to human cognition, has been a longstanding research goal. Recent
  advancements in multimodal large language models (MLLMs) have made significant
  strides in open-world understanding. However, the challenge of continuous and
  simultaneous streaming perception, memory, and reasoning remains largely
  unexplored. Current MLLMs are constrained by their sequence-to-sequence
  architecture, which limits their ability to process inputs and generate
  responses simultaneously, akin to being unable to think while perceiving.
  Furthermore, relying on long contexts to store historical data is impractical
  for long-term interactions, as retaining all information becomes costly and
  inefficient. Therefore, rather than relying on a single foundation model to
  perform all functions, this project draws inspiration from the concept of the
  Specialized Generalist AI and introduces disentangled streaming perception,
  reasoning, and memory mechanisms, enabling real-time interaction with streaming
  video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
  (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
  Processes multimodal information in real-time, storing key details in memory
  and triggering reasoning in response to user queries. (2) Multi-modal Long
  Memory Module: Integrates short-term and long-term memory, compressing
  short-term memories into long-term ones for efficient retrieval and improved
  accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
  tasks, coordinating with the perception and memory modules. This project
  simulates human-like cognition, enabling multimodal large language models to
  provide continuous and adaptive service over time.
paper_ai_keywords: multimodal large language models (MLLMs) continuous and simultaneous streaming perception memory reasoning sequence-to-sequence architecture long contexts Specialized Generalist AI disentangled streaming perception reasoning, and memory mechanisms real-time interaction streaming video and audio input Streaming Perception Module Multi-modal Long Memory Module Short-term and long-term memory Reasoning Module human-like cognition continuous and adaptive service