Update README.md

README.md

@@ -1,8 +1,10 @@
 ---
 tags:
 - ltx-video
-- image-to-video
-pinned: true
+- video-upscaling
+- diffusers
+- video-to-video
+pinned: false
 language:
 - en
 license: other
@@ -10,11 +12,15 @@ pipeline_tag: text-to-video
 library_name: diffusers
 ---
 
-# LTX-Video 0.9.7 Distilled Model Card
-This model card focuses on the model associated with the LTX-Video model, codebase available [here](https://github.com/Lightricks/LTX-Video).
+# LTX Video Spatial Upscaler 0.9.7 Model Card
+
+This model card focuses on the LTX Video Spatial Upscaler 0.9.7, a component model designed to work in conjunction with the LTX-Video generation models. 
+The main LTX-Video codebase is available [here](https://github.com/Lightricks/LTX-Video).
 
 LTX-Video is the first DiT-based video generation model capable of generating high-quality videos in real-time. It produces 30 FPS videos at a 1216×704 resolution faster than they can be watched. Trained on a large-scale dataset of diverse videos, the model generates high-resolution videos with realistic and varied content.
-We provide a model for both text-to-video as well as image+text-to-video usecases
+We provide a model for both text-to-video as well as image+text-to-video usecases.
+
+**The LTX Video Spatial Upscaler** is a diffusion-based model that enhances the spatial resolution of videos. It is specifically trained to upscale the latent representations of videos generated by LTX Video models. 
 
 <img src="./media/trailer.gif" alt="trailer" width="512">
 
@@ -26,24 +32,18 @@ We provide a model for both text-to-video as well as image+text-to-video usecase
 | ![example9](./media/ltx-video_example_00009.gif)<br><details style="max-width: 300px; margin: auto;"><summary>A man with graying hair, a beard, and a gray shirt...</summary>A man with graying hair, a beard, and a gray shirt looks down and to his right, then turns his head to the left. The camera angle is a close-up, focused on the man's face. The lighting is dim, with a greenish tint. The scene appears to be real-life footage. Step</details> | ![example10](./media/ltx-video_example_00010.gif)<br><details style="max-width: 300px; margin: auto;"><summary>A clear, turquoise river flows through a rocky canyon...</summary>A clear, turquoise river flows through a rocky canyon, cascading over a small waterfall and forming a pool of water at the bottom.The river is the main focus of the scene, with its clear water reflecting the surrounding trees and rocks. The canyon walls are steep and rocky, with some vegetation growing on them. The trees are mostly pine trees, with their green needles contrasting with the brown and gray rocks. The overall tone of the scene is one of peace and tranquility.</details> | ![example11](./media/ltx-video_example_00011.gif)<br><details style="max-width: 300px; margin: auto;"><summary>A man in a suit enters a room and speaks to two women...</summary>A man in a suit enters a room and speaks to two women sitting on a couch. The man, wearing a dark suit with a gold tie, enters the room from the left and walks towards the center of the frame. He has short gray hair, light skin, and a serious expression. He places his right hand on the back of a chair as he approaches the couch. Two women are seated on a light-colored couch in the background. The woman on the left wears a light blue sweater and has short blonde hair. The woman on the right wears a white sweater and has short blonde hair. The camera remains stationary, focusing on the man as he enters the room. The room is brightly lit, with warm tones reflecting off the walls and furniture. The scene appears to be from a film or television show.</details> | ![example12](./media/ltx-video_example_00012.gif)<br><details style="max-width: 300px; margin: auto;"><summary>The waves crash against the jagged rocks of the shoreline...</summary>The waves crash against the jagged rocks of the shoreline, sending spray high into the air.The rocks are a dark gray color, with sharp edges and deep crevices. The water is a clear blue-green, with white foam where the waves break against the rocks. The sky is a light gray, with a few white clouds dotting the horizon.</details> |
 | ![example13](./media/ltx-video_example_00013.gif)<br><details style="max-width: 300px; margin: auto;"><summary>The camera pans across a cityscape of tall buildings...</summary>The camera pans across a cityscape of tall buildings with a circular building in the center. The camera moves from left to right, showing the tops of the buildings and the circular building in the center. The buildings are various shades of gray and white, and the circular building has a green roof. The camera angle is high, looking down at the city. The lighting is bright, with the sun shining from the upper left, casting shadows from the buildings. The scene is computer-generated imagery.</details> | ![example14](./media/ltx-video_example_00014.gif)<br><details style="max-width: 300px; margin: auto;"><summary>A man walks towards a window, looks out, and then turns around...</summary>A man walks towards a window, looks out, and then turns around. He has short, dark hair, dark skin, and is wearing a brown coat over a red and gray scarf. He walks from left to right towards a window, his gaze fixed on something outside. The camera follows him from behind at a medium distance. The room is brightly lit, with white walls and a large window covered by a white curtain. As he approaches the window, he turns his head slightly to the left, then back to the right. He then turns his entire body to the right, facing the window. The camera remains stationary as he stands in front of the window. The scene is captured in real-life footage.</details> | ![example15](./media/ltx-video_example_00015.gif)<br><details style="max-width: 300px; margin: auto;"><summary>Two police officers in dark blue uniforms and matching hats...</summary>Two police officers in dark blue uniforms and matching hats enter a dimly lit room through a doorway on the left side of the frame. The first officer, with short brown hair and a mustache, steps inside first, followed by his partner, who has a shaved head and a goatee. Both officers have serious expressions and maintain a steady pace as they move deeper into the room. The camera remains stationary, capturing them from a slightly low angle as they enter. The room has exposed brick walls and a corrugated metal ceiling, with a barred window visible in the background. The lighting is low-key, casting shadows on the officers' faces and emphasizing the grim atmosphere. The scene appears to be from a film or television show.</details> | ![example16](./media/ltx-video_example_00016.gif)<br><details style="max-width: 300px; margin: auto;"><summary>A woman with short brown hair, wearing a maroon sleeveless top...</summary>A woman with short brown hair, wearing a maroon sleeveless top and a silver necklace, walks through a room while talking, then a woman with pink hair and a white shirt appears in the doorway and yells. The first woman walks from left to right, her expression serious; she has light skin and her eyebrows are slightly furrowed. The second woman stands in the doorway, her mouth open in a yell; she has light skin and her eyes are wide. The room is dimly lit, with a bookshelf visible in the background. The camera follows the first woman as she walks, then cuts to a close-up of the second woman's face. The scene is captured in real-life footage.</details> |
 
-# Models & Workflows
 
-| Name                                                                                                                                   | Notes                                                                                                         | inference.py config                                                                                                    | ComfyUI workflow (Recommended)                                                                                                                                |
-|----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| ltxv-13b-0.9.7-dev                                                                                                                     | Highest quality, requires more VRAM                                                                           | [ltxv-13b-0.9.7-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.7-dev.yaml)           | [ltxv-13b-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv-13b-i2v-base.json)                                 |
-| [ltxv-13b-0.9.7-mix](https://app.ltx.studio/motion-workspace?videoModel=ltxv-13b)                                                      | Mix ltxv-13b-dev and ltxv-13b-distilled in the same multi-scale rendering workflow for balanced speed-quality | N/A                                                                                                                    | [ltxv-13b-i2v-mix.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv13b-i2v-mixed-multiscale.json)                       |
-| [ltxv-13b-0.9.7-distilled](https://app.ltx.studio/motion-workspace?videoModel=ltxv)                                                    | Faster, less VRAM usage, slight quality reduction compared to 13b. Ideal for rapid iterations                 | [ltxv-13b-0.9.7-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.7-dev.yaml)     | [ltxv-13b-dist-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/13b-distilled/ltxv-13b-dist-i2v-base.json)         |
-| [ltxv-13b-0.9.7-distilled-lora128](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-distilled-lora128.safetensors) | LoRA to make ltxv-13b-dev behave like the distilled model                                                     | N/A                                                                                                                    | N/A                                                                                                                                                           |
-| ltxv-13b-0.9.7-fp8                                                                                                                     | Quantized version of ltxv-13b                                                                                 | Coming soon                                                                                                            | [ltxv-13b-i2v-base-fp8.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv-13b-i2v-base-fp8.json)                         |
-| ltxv-13b-0.9.7-distilled-fp8                                                                                                           | Quantized version of ltxv-13b-distilled                                                                       | Coming soon                                                                                                            | [ltxv-13b-dist-fp8-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/13b-distilled/ltxv-13b-dist-fp8-i2v-base.json) |
-| ltxv-2b-0.9.6                                                                                                                          | Good quality, lower VRAM requirement than ltxv-13b                                                            | [ltxv-2b-0.9.6-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.6-dev.yaml)             | [ltxvideo-i2v.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/low_level/ltxvideo-i2v.json)                                 |
-| ltxv-2b-0.9.6-distilled                                                                                                                | 15× faster, real-time capable, fewer steps needed, no STG/CFG required                                        | [ltxv-2b-0.9.6-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.6-distilled.yaml) | [ltxvideo-i2v-distilled.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/low_level/ltxvideo-i2v-distilled.json)             |
+**This upscaler model is compatible with and can be used to improve the output quality of videos generated by both:**
+*   `Lightricks/LTX-Video-0.9.7-dev`
+*   `Lightricks/LTX-Video-0.9.7-distilled`
+
 
 ## Model Details
 - **Developed by:** Lightricks
-- **Model type:** Diffusion-based text-to-video and image-to-video generation model
-- **Language(s):** English
-
+- **Model type:** Latent Diffusion Video Spatial Upscaler
+- **Input:** Latent video frames from an LTX Video model.
+- **Output:** Higher-resolution latent video frames.
+- **Compatibility:** can be used with `Lightricks/LTX-Video-0.9.7-dev` and `Lightricks/LTX-Video-0.9.7-distilled`.
 
 ## Usage
 
@@ -70,8 +70,7 @@ You can use the model for purposes under the license:
 
 ### Online demo
 The model is accessible right away via the following links:
-- [LTX-Studio image-to-video (13B-mix)](https://app.ltx.studio/motion-workspace?videoModel=ltxv-13b)
-- [LTX-Studio image-to-video (13B distilled)](https://app.ltx.studio/motion-workspace?videoModel=ltxv)
+- [LTX-Studio image-to-video](https://app.ltx.studio/ltx-video)
 - [Fal.ai text-to-video](https://fal.ai/models/fal-ai/ltx-video)
 - [Fal.ai image-to-video](https://fal.ai/models/fal-ai/ltx-video/image-to-video)
 - [Replicate text-to-video and image-to-video](https://replicate.com/lightricks/ltx-video)
@@ -99,18 +98,6 @@ python -m pip install -e .\[inference-script\]
 
 To use our model, please follow the inference code in [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py):
 
-##### For text-to-video generation:
-
-```bash
-python inference.py --prompt "PROMPT" --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED --pipeline_config configs/ltxv-13b-0.9.7-distilled.yaml
-```
-
-##### For image-to-video generation:
-
-```bash
-python inference.py --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED --pipeline_config configs/ltxv-13b-0.9.7-distilled.yaml
-```
-
 ### Diffusers 🧨
 
 LTX Video is compatible with the [Diffusers Python library](https://huggingface.co/docs/diffusers/main/en/index). It supports both text-to-video and image-to-video generation.
@@ -121,77 +108,9 @@ Make sure you install `diffusers` before trying out the examples below.
 pip install -U git+https://github.com/huggingface/diffusers
 ```
 
-Now, you can run the examples below (note that the upsampling stage is optional but reccomeneded):
-
-### text-to-video:
-```
-```py
-import torch
-from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
-from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
-from diffusers.utils import export_to_video
-
-pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-distilled", torch_dtype=torch.bfloat16)
-pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)
-pipe.to("cuda")
-pipe_upsample.to("cuda")
-pipe.vae.enable_tiling()
-
-prompt = "The video depicts a winding mountain road covered in snow, with a single vehicle traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. The landscape is characterized by rugged terrain and a river visible in the distance. The scene captures the solitude and beauty of a winter drive through a mountainous region."
-negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
-expected_height, expected_width = 704, 512
-downscale_factor = 2 / 3
-num_frames = 121
-
-# Part 1. Generate video at smaller resolution
-downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
-latents = pipe(
-    conditions=None,
-    prompt=prompt,
-    negative_prompt=negative_prompt,
-    width=downscaled_width,
-    height=downscaled_height,
-    num_frames=num_frames,
-    num_inference_steps=7,
-    decode_timestep = 0.05,
-    guidnace_scale=1.0,
-    decode_noise_scale = 0.025,
-    generator=torch.Generator().manual_seed(0),
-    output_type="latent",
-).frames
-
-# Part 2. Upscale generated video using latent upsampler with fewer inference steps
-# The available latent upsampler upscales the height/width by 2x
-upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2
-upscaled_latents = pipe_upsample(
-    latents=latents,
-    output_type="latent"
-).frames
-# Part 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)
-video = pipe(
-    prompt=prompt,
-    negative_prompt=negative_prompt,
-    width=upscaled_width,
-    height=upscaled_height,
-    num_frames=num_frames,
-    denoise_strength=0.3,  # Effectively, 4 inference steps out of 10
-    num_inference_steps=10,
-    latents=upscaled_latents,
-    decode_timestep = 0.05,
-    guidnace_scale=1.0,
-    decode_noise_scale = 0.025,
-    image_cond_noise_scale=0.025,
-    generator=torch.Generator().manual_seed(0),
-    output_type="pil",
-).frames[0]
-
-# Part 4. Downscale the video to the expected resolution
-video = [frame.resize((expected_width, expected_height)) for frame in video]
-
-export_to_video(video, "output.mp4", fps=24)
-```
+The LTX Video Spatial Upscaler is used via the `LTXLatentUpsamplePipeline` in the `diffusers` library. It is intended to be part of a multi-stage generation process.
 
-### For image-to-video:
+Below is an example demonstrating how to use the spatial upsampler with a base LTX Video model (either the 'dev' or 'distilled' version).
 
 ```py
 import torch
@@ -199,91 +118,25 @@ from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
 from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
 from diffusers.utils import export_to_video, load_image
 
-pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-distilled", torch_dtype=torch.bfloat16)
-pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)
+# Choose your base LTX Video model:
+# base_model_id = "Lightricks/LTX-Video-0.9.7-dev"
+base_model_id = "Lightricks/LTX-Video-0.9.7-distilled" # Using distilled for this example
+
+# 0. Load base model and upsampler
+pipe = LTXConditionPipeline.from_pretrained(base_model_id, torch_dtype=torch.bfloat16)
+pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained(
+    "Lightricks/ltxv-spatial-upscaler-0.9.7",
+    vae=pipe.vae, 
+    torch_dtype=torch.bfloat16
+)
 pipe.to("cuda")
 pipe_upsample.to("cuda")
-pipe.vae.enable_tiling()
-
-image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png")
-video = [image]
-condition1 = LTXVideoCondition(video=video, frame_index=0)
-
-prompt = "The video depicts a winding mountain road covered in snow, with a single vehicle traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. The landscape is characterized by rugged terrain and a river visible in the distance. The scene captures the solitude and beauty of a winter drive through a mountainous region."
-negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
-expected_height, expected_width = 832, 480
-downscale_factor = 2 / 3
-num_frames = 96
-
-# Part 1. Generate video at smaller resolution
-downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
-downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)
-latents = pipe(
-    conditions=[condition1],
-    prompt=prompt,
-    negative_prompt=negative_prompt,
-    width=downscaled_width,
-    height=downscaled_height,
-    num_frames=num_frames,
-    num_inference_steps=7,
-    guidnace_scale=1.0,
-    decode_timestep = 0.05,
-    decode_noise_scale = 0.025,
-    generator=torch.Generator().manual_seed(0),
-    output_type="latent",
-).frames
-
-# Part 2. Upscale generated video using latent upsampler with fewer inference steps
-# The available latent upsampler upscales the height/width by 2x
-upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2
-upscaled_latents = pipe_upsample(
-    latents=latents,
-    output_type="latent"
-).frames
-# Part 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)
-video = pipe(
-    conditions=[condition1],
-    prompt=prompt,
-    negative_prompt=negative_prompt,
-    width=upscaled_width,
-    height=upscaled_height,
-    num_frames=num_frames,
-    denoise_strength=0.3,  # Effectively, 4 inference steps out of 10
-    num_inference_steps=10,
-    guidnace_scale=1.0,
-    latents=upscaled_latents,
-    decode_timestep = 0.05,
-    decode_noise_scale = 0.025,
-    image_cond_noise_scale=0.025,
-    generator=torch.Generator().manual_seed(0),
-    output_type="pil",
-).frames[0]
-
-# Part 4. Downscale the video to the expected resolution
-video = [frame.resize((expected_width, expected_height)) for frame in video]
-
-export_to_video(video, "output.mp4", fps=24)
-
-```
-
-### For video-to-video: 
-
-```py
-import torch
-from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
-from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
-from diffusers.utils import export_to_video, load_video
-
-pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-distilled", torch_dtype=torch.bfloat16)
-pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)
-pipe.to("cuda")
-pipe_upsample.to("cuda")
-pipe.vae.enable_tiling()
 
 def round_to_nearest_resolution_acceptable_by_vae(height, width):
     height = height - (height % pipe.vae_temporal_compression_ratio)
     width = width - (width % pipe.vae_temporal_compression_ratio)
     return height, width
+
 video = load_video(
     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cosmos/cosmos-video2world-input-vid.mp4"
 )[:21]  # Use only the first 21 frames as conditioning
@@ -298,6 +151,7 @@ num_frames = 161
 # Part 1. Generate video at smaller resolution
 downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
 downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)
+
 latents = pipe(
     conditions=[condition1],
     prompt=prompt,
@@ -305,10 +159,7 @@ latents = pipe(
     width=downscaled_width,
     height=downscaled_height,
     num_frames=num_frames,
-    num_inference_steps=7,
-    guidnace_scale=1.0,
-    decode_timestep = 0.05,
-    decode_noise_scale = 0.025,
+    num_inference_steps=30,
     generator=torch.Generator().manual_seed(0),
     output_type="latent",
 ).frames
@@ -329,12 +180,10 @@ video = pipe(
     width=upscaled_width,
     height=upscaled_height,
     num_frames=num_frames,
-    denoise_strength=0.3,  # Effectively, 4 inference steps out of 10
+    denoise_strength=0.4,  # Effectively, 4 inference steps out of 10
     num_inference_steps=10,
-    guidnace_scale=1.0,
     latents=upscaled_latents,
-    decode_timestep = 0.05,
-    decode_noise_scale = 0.025,
+    decode_timestep=0.05,
     image_cond_noise_scale=0.025,
     generator=torch.Generator().manual_seed(0),
     output_type="pil",
@@ -346,10 +195,13 @@ video = [frame.resize((expected_width, expected_height)) for frame in video]
 export_to_video(video, "output.mp4", fps=24)
 
 ```
-To learn more, check out the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video). 
 
+for more details and inference examples using 🧨 diffusers, check out the [diffusers documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video)
 Diffusers also supports directly loading from the original LTX checkpoints using the `from_single_file()` method. Check out [this section](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video#loading-single-files) to learn more.
 
+To learn more, check out the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video). 
+
+
 ## Limitations
 - This model is not intended or able to provide factual information.
 - As a statistical model this checkpoint might amplify existing societal biases.