InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
  Long-term Streaming Video and Audio Interactions
Creating AI systems that can interact with environments over long periods,
similar to human cognition, has been a longstanding research goal. Recent
advancements in multimodal large language models (MLLMs) have made significant
strides in open-world understanding. However, the challenge of continuous and
simultaneous streaming perception, memory, and reasoning remains largely
unexplored. Current MLLMs are constrained by their sequence-to-sequence
architecture, which limits their ability to process inputs and generate
responses simultaneously, akin to being unable to think while perceiving.
Furthermore, relying on long contexts to store historical data is impractical
for long-term interactions, as retaining all information becomes costly and
inefficient. Therefore, rather than relying on a single foundation model to
perform all functions, this project draws inspiration from the concept of the
Specialized Generalist AI and introduces disentangled streaming perception,
reasoning, and memory mechanisms, enabling real-time interaction with streaming
video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
(IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
Processes multimodal information in real-time, storing key details in memory
and triggering reasoning in response to user queries. (2) Multi-modal Long
Memory Module: Integrates short-term and long-term memory, compressing
short-term memories into long-term ones for efficient retrieval and improved
accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
tasks, coordinating with the perception and memory modules. This project
simulates human-like cognition, enabling multimodal large language models to
provide continuous and adaptive service over time.
675ba1de7ce703a00f19b4e4
Pan Zhang
675ba1de7ce703a00f19b4e5
Xiaoyi Dong
675ba1de7ce703a00f19b4e6
Yuhang Cao
admin_assigned
2024-12-16T10:31:34.553Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
675ba1de7ce703a00f19b4e7
Yuhang Zang
claimed_verified
2024-12-13T15:22:29.113Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
675ba1de7ce703a00f19b4e8
Rui Qian
675ba1de7ce703a00f19b4e9
Xilin Wei
claimed_verified
2024-12-13T15:22:27.497Z
62eb70462f0f5e54df42f778
/avatars/456049dba67638d3cdb330cdf383f272.svg
Xilin Wei
user
Wiselnn
675ba1de7ce703a00f19b4ea
Lin Chen
claimed_verified
2024-12-17T08:04:25.665Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
675ba1de7ce703a00f19b4eb
Yifei Li
claimed_verified
2024-12-16T09:42:24.684Z
64adfeac4beffa272dfaef21
/avatars/883f6ba38b993476115dfafcef9ce3c1.svg
Yifei Li
user
JoeLeelyf
675ba1de7ce703a00f19b4ec
Junbo Niu
admin_assigned
2024-12-13T17:42:32.064Z
650abbb71aece923f21d87fc
/avatars/f09ff031c278bc42bfd7a563853e142c.svg
Junbo Niu
user
Niujunbo2002
675ba1de7ce703a00f19b4ed
Shuangrui Ding
admin_assigned
2024-12-16T10:31:46.441Z
65a7c0335e79abfa2ec30c52
/avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
Shuangrui Ding
user
Mar2Ding
675ba1de7ce703a00f19b4ee
Qipeng Guo
admin_assigned
2024-12-16T10:31:52.372Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
675ba1de7ce703a00f19b4ef
Haodong Duan
admin_assigned
2024-12-16T10:31:57.973Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
675ba1de7ce703a00f19b4f0
Xin Chen
675ba1de7ce703a00f19b4f1
Han Lv
675ba1de7ce703a00f19b4f2
Zheng Nie
claimed_verified
2025-05-26T08:44:03.047Z
646d89a3534e52f8c307742b
/avatars/ca2b79eeaa6c9755dfced4245b2e2309.svg
niezheng
user
niez
675ba1de7ce703a00f19b4f3
Min Zhang
675ba1de7ce703a00f19b4f4
Bin Wang
675ba1de7ce703a00f19b4f5
Wenwei Zhang
admin_assigned
2024-12-16T10:32:53.099Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
675ba1de7ce703a00f19b4f6
Xinyue Zhang
admin_assigned
2024-12-16T10:33:53.973Z
670a40f75054a602d11988f1
/avatars/60ba3e211c73b1c6c78460aff8699584.svg
XinyueZhang
user
XinyueZhang1997
675ba1de7ce703a00f19b4f7
Jiaye Ge
675ba1de7ce703a00f19b4f8
Wei Li
675ba1de7ce703a00f19b4f9
Jingwen Li
admin_assigned
2024-12-16T10:32:16.439Z
61740f8f8a50963ce71ffff3
/avatars/b07b264669fce4c75e063e6392ebd74b.svg
李静雯
user
Jingwen
675ba1de7ce703a00f19b4fa
Zhongying Tu
675ba1de7ce703a00f19b4fb
Conghui He
admin_assigned
2024-12-16T10:32:46.981Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
675ba1de7ce703a00f19b4fc
Xingcheng Zhang
675ba1de7ce703a00f19b4fd
Kai Chen
675ba1de7ce703a00f19b4fe
Yu Qiao
675ba1de7ce703a00f19b4ff
Dahua Lin
admin_assigned
2024-12-13T17:42:17.927Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
675ba1de7ce703a00f19b500
Jiaqi Wang
claimed_verified
2025-01-13T08:54:46.738Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
  Long-term Streaming Video and Audio Interactions
Creating AI systems that can interact with environments over long periods,
similar to human cognition, has been a longstanding research goal. Recent
advancements in multimodal large language models (MLLMs) have made significant
strides in open-world understanding. However, the challenge of continuous and
simultaneous streaming perception, memory, and reasoning remains largely
unexplored. Current MLLMs are constrained by their sequence-to-sequence
architecture, which limits their ability to process inputs and generate
responses simultaneously, akin to being unable to think while perceiving.
Furthermore, relying on long contexts to store historical data is impractical
for long-term interactions, as retaining all information becomes costly and
inefficient. Therefore, rather than relying on a single foundation model to
perform all functions, this project draws inspiration from the concept of the
Specialized Generalist AI and introduces disentangled streaming perception,
reasoning, and memory mechanisms, enabling real-time interaction with streaming
video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
(IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
Processes multimodal information in real-time, storing key details in memory
and triggering reasoning in response to user queries. (2) Multi-modal Long
Memory Module: Integrates short-term and long-term memory, compressing
short-term memories into long-term ones for efficient retrieval and improved
accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
tasks, coordinating with the perception and memory modules. This project
simulates human-like cognition, enabling multimodal large language models to
provide continuous and adaptive service over time.
A framework combining disentangled perception, reasoning, and memory mechanisms enables real-time interaction for long-term multimodal AI systems.
multimodal large language models MLLMs continuous interaction streaming perception long memory multi-modal long memory real-time interaction specialized generalist AI