title: WORLDMEM: Long-term Consistent World Simulation with Memory
summary: World simulation has gained increasing popularity due to its ability to model
  virtual environments and predict the consequences of actions. However, the
  limited temporal context window often leads to failures in maintaining
  long-term consistency, particularly in preserving 3D spatial consistency. In
  this work, we present WorldMem, a framework that enhances scene generation with
  a memory bank consisting of memory units that store memory frames and states
  (e.g., poses and timestamps). By employing a memory attention mechanism that
  effectively extracts relevant information from these memory frames based on
  their states, our method is capable of accurately reconstructing previously
  observed scenes, even under significant viewpoint or temporal gaps.
  Furthermore, by incorporating timestamps into the states, our framework not
  only models a static world but also captures its dynamic evolution over time,
  enabling both perception and interaction within the simulated world. Extensive
  experiments in both virtual and real scenarios validate the effectiveness of
  our approach.
paper_authors._id: 6801a8453c431c2bbe3b5f94
paper_authors.name: Zeqi Xiao
paper_authors._id: 6801a8453c431c2bbe3b5f95
paper_authors.name: Yushi Lan
paper_authors._id: 6801a8453c431c2bbe3b5f96
paper_authors.name: Yifan Zhou
paper_authors._id: 6801a8453c431c2bbe3b5f97
paper_authors.name: Wenqi Ouyang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:13:22.717Z
paper_authors._id: 6487d9415d164d8c7f905bcf
paper_authors.avatarUrl: /avatars/1c01dd4b88f88a9863a12c09a9a28bed.svg
paper_authors.fullname: Wenqi Ouyang
paper_authors.type: user
paper_authors.user: Vicky0522
paper_authors._id: 6801a8453c431c2bbe3b5f98
paper_authors.name: Shuai Yang
paper_authors._id: 6801a8453c431c2bbe3b5f99
paper_authors.name: Yanhong Zeng
paper_authors._id: 6801a8453c431c2bbe3b5f9a
paper_authors.name: Xingang Pan
paper_title: WORLDMEM: Long-term Consistent World Simulation with Memory
paper_summary: World simulation has gained increasing popularity due to its ability to model
  virtual environments and predict the consequences of actions. However, the
  limited temporal context window often leads to failures in maintaining
  long-term consistency, particularly in preserving 3D spatial consistency. In
  this work, we present WorldMem, a framework that enhances scene generation with
  a memory bank consisting of memory units that store memory frames and states
  (e.g., poses and timestamps). By employing a memory attention mechanism that
  effectively extracts relevant information from these memory frames based on
  their states, our method is capable of accurately reconstructing previously
  observed scenes, even under significant viewpoint or temporal gaps.
  Furthermore, by incorporating timestamps into the states, our framework not
  only models a static world but also captures its dynamic evolution over time,
  enabling both perception and interaction within the simulated world. Extensive
  experiments in both virtual and real scenarios validate the effectiveness of
  our approach.
paper_ai_summary: WorldMem uses a memory bank with attention mechanism to enhance scene generation by preserving 3D spatial and temporal consistency across viewpoints and time.
paper_ai_keywords: memory bank memory units memory frames states memory attention mechanism scene generation 3D spatial consistency temporal consistency viewpoints timestamps static world dynamic evolution perception interaction virtual scenarios real scenarios