WORLDMEM: Long-term Consistent World Simulation with Memory
World simulation has gained increasing popularity due to its ability to model
virtual environments and predict the consequences of actions. However, the
limited temporal context window often leads to failures in maintaining
long-term consistency, particularly in preserving 3D spatial consistency. In
this work, we present WorldMem, a framework that enhances scene generation with
a memory bank consisting of memory units that store memory frames and states
(e.g., poses and timestamps). By employing a memory attention mechanism that
effectively extracts relevant information from these memory frames based on
their states, our method is capable of accurately reconstructing previously
observed scenes, even under significant viewpoint or temporal gaps.
Furthermore, by incorporating timestamps into the states, our framework not
only models a static world but also captures its dynamic evolution over time,
enabling both perception and interaction within the simulated world. Extensive
experiments in both virtual and real scenarios validate the effectiveness of
our approach.
6801a8453c431c2bbe3b5f94
Zeqi Xiao
6801a8453c431c2bbe3b5f95
Yushi Lan
6801a8453c431c2bbe3b5f96
Yifan Zhou
6801a8453c431c2bbe3b5f97
Wenqi Ouyang
claimed_verified
2025-04-22T10:13:22.717Z
6487d9415d164d8c7f905bcf
/avatars/1c01dd4b88f88a9863a12c09a9a28bed.svg
Wenqi Ouyang
user
Vicky0522
6801a8453c431c2bbe3b5f98
Shuai Yang
6801a8453c431c2bbe3b5f99
Yanhong Zeng
6801a8453c431c2bbe3b5f9a
Xingang Pan
WORLDMEM: Long-term Consistent World Simulation with Memory
World simulation has gained increasing popularity due to its ability to model
virtual environments and predict the consequences of actions. However, the
limited temporal context window often leads to failures in maintaining
long-term consistency, particularly in preserving 3D spatial consistency. In
this work, we present WorldMem, a framework that enhances scene generation with
a memory bank consisting of memory units that store memory frames and states
(e.g., poses and timestamps). By employing a memory attention mechanism that
effectively extracts relevant information from these memory frames based on
their states, our method is capable of accurately reconstructing previously
observed scenes, even under significant viewpoint or temporal gaps.
Furthermore, by incorporating timestamps into the states, our framework not
only models a static world but also captures its dynamic evolution over time,
enabling both perception and interaction within the simulated world. Extensive
experiments in both virtual and real scenarios validate the effectiveness of
our approach.
WorldMem uses a memory bank with attention mechanism to enhance scene generation by preserving 3D spatial and temporal consistency across viewpoints and time.
memory bank memory units memory frames states memory attention mechanism scene generation 3D spatial consistency temporal consistency viewpoints timestamps static world dynamic evolution perception interaction virtual scenarios real scenarios