title: KMM: Key Frame Mask Mamba for Extended Motion Generation
summary: Human motion generation is a cut-edge area of research in generative computer
  vision, with promising applications in video creation, game development, and
  robotic manipulation. The recent Mamba architecture shows promising results in
  efficiently modeling long and complex sequences, yet two significant challenges
  remain: Firstly, directly applying Mamba to extended motion generation is
  ineffective, as the limited capacity of the implicit memory leads to memory
  decay. Secondly, Mamba struggles with multimodal fusion compared to
  Transformers, and lack alignment with textual queries, often confusing
  directions (left or right) or omitting parts of longer text queries. To address
  these challenges, our paper presents three key contributions: Firstly, we
  introduce KMM, a novel architecture featuring Key frame Masking Modeling,
  designed to enhance Mamba's focus on key actions in motion segments. This
  approach addresses the memory decay problem and represents a pioneering method
  in customizing strategic frame-level masking in SSMs. Additionally, we designed
  a contrastive learning paradigm for addressing the multimodal fusion problem in
  Mamba and improving the motion-text alignment. Finally, we conducted extensive
  experiments on the go-to dataset, BABEL, achieving state-of-the-art performance
  with a reduction of more than 57% in FID and 70% parameters compared to
  previous state-of-the-art methods. See project website:
  https://steve-zeyu-zhang.github.io/KMM
paper_authors._id: 67331674dfcbdbfec8c1ab64
paper_authors.name: Zeyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:26:26.169Z
paper_authors._id: 64ec877bb93654d4ca5c92e9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ec877bb93654d4ca5c92e9/GvHk_KSdE9Rhnk_o-NaZX.jpeg
paper_authors.fullname: Zeyu Zhang
paper_authors.type: user
paper_authors.user: SteveZeyuZhang
paper_authors._id: 67331674dfcbdbfec8c1ab65
paper_authors.name: Hang Gao
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-12T08:48:53.847Z
paper_authors._id: 62ba40650e2fb1dcddc6dad1
paper_authors.avatarUrl: /avatars/0f1d303373494d6ca809c123e5e528a6.svg
paper_authors.fullname: Hao Tang
paper_authors.type: user
paper_authors.user: Ha0Tang
paper_authors._id: 67331674dfcbdbfec8c1ab66
paper_authors.name: Akide Liu
paper_authors._id: 67331674dfcbdbfec8c1ab67
paper_authors.name: Qi Chen
paper_authors._id: 67331674dfcbdbfec8c1ab68
paper_authors.name: Feng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T11:16:41.540Z
paper_authors._id: 64574380f3ef144c0e69d484
paper_authors.avatarUrl: /avatars/a0a84757cb0bf09c24291803e1389b49.svg
paper_authors.fullname: Feng Chen
paper_authors.type: user
paper_authors.user: chenfeng1271
paper_authors._id: 67331674dfcbdbfec8c1ab69
paper_authors.name: Yiran Wang
paper_authors._id: 67331674dfcbdbfec8c1ab6a
paper_authors.name: Danning Li
paper_authors._id: 67331674dfcbdbfec8c1ab6b
paper_authors.name: Hao Tang
paper_title: KMM: Key Frame Mask Mamba for Extended Motion Generation
paper_summary: Human motion generation is a cut-edge area of research in generative computer
  vision, with promising applications in video creation, game development, and
  robotic manipulation. The recent Mamba architecture shows promising results in
  efficiently modeling long and complex sequences, yet two significant challenges
  remain: Firstly, directly applying Mamba to extended motion generation is
  ineffective, as the limited capacity of the implicit memory leads to memory
  decay. Secondly, Mamba struggles with multimodal fusion compared to
  Transformers, and lack alignment with textual queries, often confusing
  directions (left or right) or omitting parts of longer text queries. To address
  these challenges, our paper presents three key contributions: Firstly, we
  introduce KMM, a novel architecture featuring Key frame Masking Modeling,
  designed to enhance Mamba's focus on key actions in motion segments. This
  approach addresses the memory decay problem and represents a pioneering method
  in customizing strategic frame-level masking in SSMs. Additionally, we designed
  a contrastive learning paradigm for addressing the multimodal fusion problem in
  Mamba and improving the motion-text alignment. Finally, we conducted extensive
  experiments on the go-to dataset, BABEL, achieving state-of-the-art performance
  with a reduction of more than 57% in FID and 70% parameters compared to
  previous state-of-the-art methods. See project website:
  https://steve-zeyu-zhang.github.io/KMM
paper_ai_summary: KMM enhances Mamba's motion generation through key frame masking and contrastive learning, achieving state-of-the-art results with reduced parameters.
paper_ai_keywords: Key frame Masking Modeling SSMs contrastive learning multimodal fusion memory decay motion-text alignment BABEL FID