LTX-Video: Realtime Video Latent Diffusion
We introduce LTX-Video, a transformer-based latent diffusion model that
adopts a holistic approach to video generation by seamlessly integrating the
responsibilities of the Video-VAE and the denoising transformer. Unlike
existing methods, which treat these components as independent, LTX-Video aims
to optimize their interaction for improved efficiency and quality. At its core
is a carefully designed Video-VAE that achieves a high compression ratio of
1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled
by relocating the patchifying operation from the transformer's input to the
VAE's input. Operating in this highly compressed latent space enables the
transformer to efficiently perform full spatiotemporal self-attention, which is
essential for generating high-resolution videos with temporal consistency.
However, the high compression inherently limits the representation of fine
details. To address this, our VAE decoder is tasked with both latent-to-pixel
conversion and the final denoising step, producing the clean result directly in
pixel space. This approach preserves the ability to generate fine details
without incurring the runtime cost of a separate upsampling module. Our model
supports diverse use cases, including text-to-video and image-to-video
generation, with both capabilities trained simultaneously. It achieves
faster-than-real-time generation, producing 5 seconds of 24 fps video at
768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all
existing models of similar scale. The source code and pre-trained models are
publicly available, setting a new benchmark for accessible and scalable video
generation.
6777ef01cbb36f508a37aa44
Yoav HaCohen
claimed_verified
2025-01-05T23:02:34.545Z
6303cc5e0547362a22a51af0
/avatars/8f3348f121565bf6c5e1af0e559a43a3.svg
Yoav HaCohen
user
yoavhacohen
6777ef01cbb36f508a37aa45
Nisan Chiprut
admin_assigned
2025-01-06T14:08:26.318Z
62dd30a8d43078cd49ac8ad8
/avatars/ad599719290637f7817b7508a91c2e2c.svg
Nisan Chiprut
user
nisan
6777ef01cbb36f508a37aa46
Benny Brazowski
claimed_verified
2025-01-05T23:02:32.419Z
6489c487b9e9258ba065418f
https://cdn-avatars.huggingface.co/v1/production/uploads/6489c487b9e9258ba065418f/6rzmV3bQ3YxswG6NP2hDW.png
Benny Brazowski
user
benibraz
6777ef01cbb36f508a37aa47
Daniel Shalem
admin_assigned
2025-01-06T14:08:45.579Z
672c8f988e6f8ed5741f4917
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/LPjQ1suwUAfipCIe1HbaB.png
daniel shalem
user
danielshalemltx
6777ef01cbb36f508a37aa48
Dudu Moshe
admin_assigned
2025-01-06T14:08:51.844Z
636b97a57631fe5e86fe1fa2
/avatars/c568ae26fd4fc2655cd12f15d539db58.svg
Dudu Moshe
user
dudumoshe
6777ef01cbb36f508a37aa49
Eitan Richardson
admin_assigned
2025-01-06T14:08:57.988Z
677a422979d3c32a5dd87a0a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WUa6E68GpnT2mEMJ41nDd.png
Eitan Richardson
user
eitanrich
6777ef01cbb36f508a37aa4a
Eran Levin
admin_assigned
2025-01-06T14:15:13.163Z
670d24e76ad961f79c00fe9b
/avatars/7407fa476f1b227e35817578632e839f.svg
Eran Levin
user
eranlevinlt
6777ef01cbb36f508a37aa4b
Guy Shiran
admin_assigned
2025-01-06T14:15:21.216Z
673f6911d83832a6ce15e7bf
/avatars/0da6cded3b0e785241a6ba5fdb5d8ceb.svg
Guy Shiran
user
guysrn
6777ef01cbb36f508a37aa4c
Nir Zabari
admin_assigned
2025-01-06T14:15:47.837Z
673f29b568595672b8d3e90e
https://cdn-avatars.huggingface.co/v1/production/uploads/673f29b568595672b8d3e90e/4sYADg3mpqMKmJ4fQwaTl.png
Nir Zabari
user
NirZabariLTX
6777ef01cbb36f508a37aa4d
Ori Gordon
claimed_verified
2025-01-05T23:02:14.348Z
674348b46215a2c0878e219b
/avatars/8a213e431a1583d1a93377410907c059.svg
Ori Gordon
user
origordon
6777ef01cbb36f508a37aa4e
Poriya Panet
6777ef01cbb36f508a37aa4f
Sapir Weissbuch
admin_assigned
2025-01-06T14:16:36.788Z
5fd0b693a23a118c84d66a2d
/avatars/a469a67da5765b557444a40264dbe73f.svg
Sapir Weissbuch
user
sapirw
6777ef01cbb36f508a37aa50
Victor Kulikov
6777ef01cbb36f508a37aa51
Yaki Bitterman
6777ef01cbb36f508a37aa52
Zeev Melumian
6777ef01cbb36f508a37aa53
Ofir Bibi
admin_assigned
2025-01-06T14:16:55.345Z
631f58935ba8c026340b377c
https://cdn-avatars.huggingface.co/v1/production/uploads/631f58935ba8c026340b377c/4yoHLdNE99VBb7ji_Mzzj.jpeg
Ofir Bibi
user
ofirbibi
LTX-Video: Realtime Video Latent Diffusion
We introduce LTX-Video, a transformer-based latent diffusion model that
adopts a holistic approach to video generation by seamlessly integrating the
responsibilities of the Video-VAE and the denoising transformer. Unlike
existing methods, which treat these components as independent, LTX-Video aims
to optimize their interaction for improved efficiency and quality. At its core
is a carefully designed Video-VAE that achieves a high compression ratio of
1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled
by relocating the patchifying operation from the transformer's input to the
VAE's input. Operating in this highly compressed latent space enables the
transformer to efficiently perform full spatiotemporal self-attention, which is
essential for generating high-resolution videos with temporal consistency.
However, the high compression inherently limits the representation of fine
details. To address this, our VAE decoder is tasked with both latent-to-pixel
conversion and the final denoising step, producing the clean result directly in
pixel space. This approach preserves the ability to generate fine details
without incurring the runtime cost of a separate upsampling module. Our model
supports diverse use cases, including text-to-video and image-to-video
generation, with both capabilities trained simultaneously. It achieves
faster-than-real-time generation, producing 5 seconds of 24 fps video at
768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all
existing models of similar scale. The source code and pre-trained models are
publicly available, setting a new benchmark for accessible and scalable video
generation.
LTX-Video, a transformer-based latent diffusion model, integrates Video-VAE and denoising transformer for efficient high-resolution video generation with temporal consistency.
transformer-based latent diffusion model Video-VAE denoising transformer spatiotemporal downscaling patchifying operation self-attention latent space latent-to-pixel conversion text-to-video generation image-to-video generation Nvidia H100 GPU accessible and scalable video generation