Update README.md to include diffusers usage

#42
by sayakpaul HF staff - opened
Files changed (1) hide show
  1. README.md +66 -0
README.md CHANGED
@@ -92,6 +92,72 @@ python inference.py --ckpt_dir 'PATH' --prompt "PROMPT" --height HEIGHT --width
92
  python inference.py --ckpt_dir 'PATH' --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED
93
  ```
94
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
  ## Limitations
96
  - This model is not intended or able to provide factual information.
97
  - As a statistical model this checkpoint might amplify existing societal biases.
 
92
  python inference.py --ckpt_dir 'PATH' --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED
93
  ```
94
 
95
+ ### Diffusers 🧨
96
+
97
+ LTX Video is compatible with the [Diffusers Python library](https://huggingface.co/docs/diffusers/main/en/index). It supports both text-to-video and image-to-video generation.
98
+
99
+ Make sure you install `diffusers` before trying out the examples below.
100
+
101
+ ```bash
102
+ pip install -U git+https://github.com/huggingface/diffusers
103
+ ```
104
+
105
+ Now, you can run the examples below:
106
+
107
+ ```py
108
+ import torch
109
+ from diffusers import LTXPipeline
110
+ from diffusers.utils import export_to_video
111
+
112
+ pipe = LTXPipeline.from_pretrained("Lightricks/LTX-Video", torch_dtype=torch.bfloat16)
113
+ pipe.to("cuda")
114
+
115
+ prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
116
+ negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
117
+
118
+ video = pipe(
119
+ prompt=prompt,
120
+ negative_prompt=negative_prompt,
121
+ width=704,
122
+ height=480,
123
+ num_frames=161,
124
+ num_inference_steps=50,
125
+ ).frames[0]
126
+ export_to_video(video, "output.mp4", fps=24)
127
+ ```
128
+
129
+ For image-to-video:
130
+
131
+ ```py
132
+ import torch
133
+ from diffusers import LTXImageToVideoPipeline
134
+ from diffusers.utils import export_to_video, load_image
135
+
136
+ pipe = LTXImageToVideoPipeline.from_pretrained("Lightricks/LTX-Video", torch_dtype=torch.bfloat16)
137
+ pipe.to("cuda")
138
+
139
+ image = load_image(
140
+ "https://huggingface.co/datasets/a-r-r-o-w/tiny-meme-dataset-captioned/resolve/main/images/8.png"
141
+ )
142
+ prompt = "A young girl stands calmly in the foreground, looking directly at the camera, as a house fire rages in the background. Flames engulf the structure, with smoke billowing into the air. Firefighters in protective gear rush to the scene, a fire truck labeled '38' visible behind them. The girl's neutral expression contrasts sharply with the chaos of the fire, creating a poignant and emotionally charged scene."
143
+ negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
144
+
145
+ video = pipe(
146
+ image=image,
147
+ prompt=prompt,
148
+ negative_prompt=negative_prompt,
149
+ width=704,
150
+ height=480,
151
+ num_frames=161,
152
+ num_inference_steps=50,
153
+ ).frames[0]
154
+ export_to_video(video, "output.mp4", fps=24)
155
+ ```
156
+
157
+ To learn more, check out the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video).
158
+
159
+ Diffusers also supports directly loading from the original LTX checkpoints using the `from_single_file()` method. Check out [this section](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video#loading-single-files) to learn more.
160
+
161
  ## Limitations
162
  - This model is not intended or able to provide factual information.
163
  - As a statistical model this checkpoint might amplify existing societal biases.