title: World Model on Million-Length Video And Language With RingAttention
summary: Current language models fall short in understanding aspects of the world not
  easily described in words, and struggle with complex, long-form tasks. Video
  sequences offer valuable temporal information absent in language and static
  images, making them attractive for joint modeling with language. Such models
  could develop a understanding of both human textual knowledge and the physical
  world, enabling broader AI capabilities for assisting humans. However, learning
  from millions of tokens of video and language sequences poses challenges due to
  memory constraints, computational complexity, and limited datasets. To address
  these challenges, we curate a large dataset of diverse videos and books,
  utilize the RingAttention technique to scalably train on long sequences, and
  gradually increase context size from 4K to 1M tokens. This paper makes the
  following contributions: (a) Largest context size neural network: We train one
  of the largest context size transformers on long video and language sequences,
  setting new benchmarks in difficult retrieval tasks and long video
  understanding. (b) Solutions for overcoming vision-language training
  challenges, including using masked sequence packing for mixing different
  sequence lengths, loss weighting to balance language and vision, and
  model-generated QA dataset for long sequence chat. (c) A highly-optimized
  implementation with RingAttention, masked sequence packing, and other key
  features for training on millions-length multimodal sequences. (d) Fully
  open-sourced a family of 7B parameter models capable of processing long text
  documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M
  tokens. This work paves the way for training on massive datasets of long video
  and language to develop understanding of both human knowledge and the
  multimodal world, and broader capabilities.
paper_authors._id: 65cc2e8f9a4ee02bbbbf78e5
paper_authors.name: Hao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T10:52:25.419Z
paper_authors._id: 62aa53fc5df2251f7df798c6
paper_authors.avatarUrl: /avatars/1c7828ce861c30ddd0afceff355a92ec.svg
paper_authors.fullname: Hao
paper_authors.type: user
paper_authors.user: haoliu
paper_authors._id: 65cc2e8f9a4ee02bbbbf78e6
paper_authors.name: Wilson Yan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-14T03:08:00.254Z
paper_authors._id: 63443747bcabbbc21a4c4aae
paper_authors.avatarUrl: /avatars/822dcdd9d94a79f04073ef64df0a0a9e.svg
paper_authors.fullname: Wilson
paper_authors.type: user
paper_authors.user: wilson1yan
paper_authors._id: 65cc2e8f9a4ee02bbbbf78e7
paper_authors.name: Matei Zaharia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T10:50:28.065Z
paper_authors._id: 642509c6d476e4ad5568dd31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
paper_authors.fullname: Matei Zaharia
paper_authors.type: user
paper_authors.user: mateiz
paper_authors._id: 65cc2e8f9a4ee02bbbbf78e8
paper_authors.name: Pieter Abbeel
paper_title: World Model on Million-Length Video And Language With RingAttention
paper_summary: Current language models fall short in understanding aspects of the world not
  easily described in words, and struggle with complex, long-form tasks. Video
  sequences offer valuable temporal information absent in language and static
  images, making them attractive for joint modeling with language. Such models
  could develop a understanding of both human textual knowledge and the physical
  world, enabling broader AI capabilities for assisting humans. However, learning
  from millions of tokens of video and language sequences poses challenges due to
  memory constraints, computational complexity, and limited datasets. To address
  these challenges, we curate a large dataset of diverse videos and books,
  utilize the RingAttention technique to scalably train on long sequences, and
  gradually increase context size from 4K to 1M tokens. This paper makes the
  following contributions: (a) Largest context size neural network: We train one
  of the largest context size transformers on long video and language sequences,
  setting new benchmarks in difficult retrieval tasks and long video
  understanding. (b) Solutions for overcoming vision-language training
  challenges, including using masked sequence packing for mixing different
  sequence lengths, loss weighting to balance language and vision, and
  model-generated QA dataset for long sequence chat. (c) A highly-optimized
  implementation with RingAttention, masked sequence packing, and other key
  features for training on millions-length multimodal sequences. (d) Fully
  open-sourced a family of 7B parameter models capable of processing long text
  documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M
  tokens. This work paves the way for training on massive datasets of long video
  and language to develop understanding of both human knowledge and the
  multimodal world, and broader capabilities.
paper_ai_summary: Large-scale training of a transformer model for long video and language sequences using RingAttention and masked sequence packing improves performance on retrieval and long-form understanding tasks.
paper_ai_keywords: RingAttention masked sequence packing long video large context size transformers multimodal sequences loss weighting model-generated QA dataset