Fast LoRA inference for Flux with Diffusers and PEFT 🚨
text
Fast LoRA inference for Flux with Diffusers and PEFT 🚨
new_line
new_line
There are great materials that demonstrate how to optimize inference for popular image generation models, such as Flux. However, very few cover how to serve LoRAs fast, despite LoRAs being an inseparable part of their adoption.
text
There are great materials that demonstrate how to optimize inference for popular image generation models, such as Flux. However, very few cover how to serve LoRAs fast, despite LoRAs being an inseparable part of their adoption.
new_line
new_line
In our latest post, 
text
In our latest post, 
@BenjaminB
mention
BenjaminB
 and I show different techniques to optimize LoRA inference for the Flux family of models for image generation. Our recipe includes the use of:
text
 and I show different techniques to optimize LoRA inference for the Flux family of models for image generation. Our recipe includes the use of:
new_line
new_line
1. 
text
1. 
torch.compile
`torch.compile`
inline_code
new_line
2. Flash Attention 3 (when compatible)
text
2. Flash Attention 3 (when compatible)
new_line
3. Dynamic FP8 weight quantization (when compatible)
text
3. Dynamic FP8 weight quantization (when compatible)
new_line
4. Hotswapping for avoiding recompilation during swapping new LoRAs 🤯
text
4. Hotswapping for avoiding recompilation during swapping new LoRAs 🤯
new_line
new_line
We have tested our recipe with Flux.1-Dev on both H100 and RTX 4090. We achieve at least a *2x speedup* in either of the GPUs. We believe our recipe is grounded in the reality of how LoRA-based use cases are generally served. So, we hope this will be beneficial to the community 🤗
text
We have tested our recipe with Flux.1-Dev on both H100 and RTX 4090. We achieve at least a *2x speedup* in either of the GPUs. We believe our recipe is grounded in the reality of how LoRA-based use cases are generally served. So, we hope this will be beneficial to the community 🤗
new_line
new_line
Even though our recipe was tested primarily with NVIDIA GPUs, it should also work with AMD GPUs.
text
Even though our recipe was tested primarily with NVIDIA GPUs, it should also work with AMD GPUs.
new_line
new_line
Learn the details and the full code here:
text
Learn the details and the full code here:
new_line
https://huggingface.co/blog/lora-fast
https://huggingface.co/blog/lora-fast
link
Fast LoRA inference for Flux with Diffusers and PEFT 🚨

There are great materials that demonstrate how to optimize inference for popular image generation models, such as Flux. However, very few cover how to serve LoRAs fast, despite LoRAs being an inseparable part of their adoption.

In our latest post, @BenjaminB and I show different techniques to optimize LoRA inference for the Flux family of models for image generation. Our recipe includes the use of:

1. `torch.compile`
2. Flash Attention 3 (when compatible)
3. Dynamic FP8 weight quantization (when compatible)
4. Hotswapping for avoiding recompilation during swapping new LoRAs 🤯

We have tested our recipe with Flux.1-Dev on both H100 and RTX 4090. We achieve at least a *2x speedup* in either of the GPUs. We believe our recipe is grounded in the reality of how LoRA-based use cases are generally served. So, we hope this will be beneficial to the community 🤗

Even though our recipe was tested primarily with NVIDIA GPUs, it should also work with AMD GPUs.

Learn the details and the full code here:
https://huggingface.co/blog/lora-fast
🤗
John6666 9voltfan2009 ethix
🔥
John6666 9voltfan2009