title: Loong: Generating Minute-level Long Videos with Autoregressive Language
    Models
summary: It is desirable but challenging to generate content-rich long videos in the
  scale of minutes. Autoregressive large language models (LLMs) have achieved
  great success in generating coherent and long sequences of tokens in the domain
  of natural language processing, while the exploration of autoregressive LLMs
  for video generation is limited to generating short videos of several seconds.
  In this work, we conduct a deep analysis of the challenges that prevent
  autoregressive LLM-based video generators from generating long videos. Based on
  the observations and analysis, we propose Loong, a new autoregressive LLM-based
  video generator that can generate minute-long videos. Specifically, we model
  the text tokens and video tokens as a unified sequence for autoregressive LLMs
  and train the model from scratch. We propose progressive short-to-long training
  with a loss re-weighting scheme to mitigate the loss imbalance problem for long
  video training. We further investigate inference strategies, including video
  token re-encoding and sampling strategies, to diminish error accumulation
  during inference. Our proposed Loong can be trained on 10-second videos and be
  extended to generate minute-level long videos conditioned on text prompts, as
  demonstrated by the results. More samples are available at:
  https://epiphqny.github.io/Loong-video.
paper_authors._id: 66ff590f1687036580ad3899
paper_authors.name: Yuqing Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:30:54.287Z
paper_authors._id: 63ea23b9dedfeebe54d02bdf
paper_authors.avatarUrl: /avatars/4d9f9a546aa8c63e277161ea700075c4.svg
paper_authors.fullname: Yuqing Wang
paper_authors.type: user
paper_authors.user: Epiphqny
paper_authors._id: 66ff590f1687036580ad389a
paper_authors.name: Tianwei Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:32.472Z
paper_authors._id: 668125557b50b433cda2a211
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/668125557b50b433cda2a211/j3z3wT5Rv9IyUKtbzQpnc.png
paper_authors.fullname: Tianwei Xiong
paper_authors.type: user
paper_authors.user: YuuTennYi
paper_authors._id: 66ff590f1687036580ad389b
paper_authors.name: Daquan Zhou
paper_authors._id: 66ff590f1687036580ad389c
paper_authors.name: Zhijie Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:35:48.411Z
paper_authors._id: 64415957bd0c9726529802f6
paper_authors.avatarUrl: /avatars/1132d1ee68fb58ec635d57c8175caacd.svg
paper_authors.fullname: Zhijie Lin
paper_authors.type: user
paper_authors.user: Ikuinen
paper_authors._id: 66ff590f1687036580ad389d
paper_authors.name: Yang Zhao
paper_authors._id: 66ff590f1687036580ad389e
paper_authors.name: Bingyi Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:36:06.799Z
paper_authors._id: 647b5fef6a79fbf5e996c47c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
paper_authors.fullname: Bingyi Kang
paper_authors.type: user
paper_authors.user: bykang
paper_authors._id: 66ff590f1687036580ad389f
paper_authors.name: Jiashi Feng
paper_authors._id: 66ff590f1687036580ad38a0
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:26:25.925Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: Loong: Generating Minute-level Long Videos with Autoregressive Language
    Models
paper_summary: It is desirable but challenging to generate content-rich long videos in the
  scale of minutes. Autoregressive large language models (LLMs) have achieved
  great success in generating coherent and long sequences of tokens in the domain
  of natural language processing, while the exploration of autoregressive LLMs
  for video generation is limited to generating short videos of several seconds.
  In this work, we conduct a deep analysis of the challenges that prevent
  autoregressive LLM-based video generators from generating long videos. Based on
  the observations and analysis, we propose Loong, a new autoregressive LLM-based
  video generator that can generate minute-long videos. Specifically, we model
  the text tokens and video tokens as a unified sequence for autoregressive LLMs
  and train the model from scratch. We propose progressive short-to-long training
  with a loss re-weighting scheme to mitigate the loss imbalance problem for long
  video training. We further investigate inference strategies, including video
  token re-encoding and sampling strategies, to diminish error accumulation
  during inference. Our proposed Loong can be trained on 10-second videos and be
  extended to generate minute-level long videos conditioned on text prompts, as
  demonstrated by the results. More samples are available at:
  https://epiphqny.github.io/Loong-video.
paper_ai_summary: A novel autoregressive large language model (LLM)-based video generator, Loong, is proposed to produce minute-long videos by addressing challenges related to long video generation.
paper_ai_keywords: autoregressive LLMs video generation text tokens video tokens progressive short-to-long training loss re-weighting scheme inference strategies video token re-encoding sampling strategies