Latent Flow Transformer
Transformers, the standard implementation for large language models (LLMs),
typically consist of tens to hundreds of discrete layers. While more layers can
lead to better performance, this approach has been challenged as far from
efficient, especially given the superiority of continuous layers demonstrated
by diffusion and flow-based models for image generation. We propose the Latent
Flow Transformer (LFT), which replaces a block of layers with a single learned
transport operator trained via flow matching, offering significant compression
while maintaining compatibility with the original architecture. Additionally,
we address the limitations of existing flow-based methods in preserving
coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M
model, LFT trained with flow matching compresses 6 of 24 layers and outperforms
directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529),
demonstrating the feasibility of this design. When trained with FW, LFT further
distills 12 layers into one while reducing the KL to 0.736 surpassing that from
skipping 3 layers (0.932), significantly narrowing the gap between
autoregressive and flow-based generation paradigms.
682d334862cadf615f5f73e6
Yen-Chen Wu
admin_assigned
2025-05-21T10:11:09.090Z
63ad217b9fc40b14560e9e06
/avatars/c2d33830b141fc9c73ad8302ff35ed9d.svg
Yen-Chen Wu
user
yenchen
682d334862cadf615f5f73e7
Feng-Ting Liao
claimed_verified
2025-05-21T08:41:00.821Z
643fb7332397d8eef5b844cd
/avatars/e403a19fc13e478d5929c67028230b0e.svg
Feng-Ting Liao
user
FengTing
682d334862cadf615f5f73e8
Meng-Hsi Chen
admin_assigned
2025-05-21T10:11:15.604Z
66018c8eb1e509e1e4d9196f
/avatars/5d6cbfc7b6c435264d271c958607630f.svg
Meng-Hsi Chen
user
menghsichen
682d334862cadf615f5f73e9
Pei-Chen Ho
682d334862cadf615f5f73ea
Farhang Nabiei
682d334862cadf615f5f73eb
Da-shan Shiu
admin_assigned
2025-05-21T10:11:37.639Z
6811b1294119e4ecc92fc93b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/srmY2yyzOg9KRDSLYXJKf.png
Dashan Shiu
user
dsshiu
Latent Flow Transformer
Transformers, the standard implementation for large language models (LLMs),
typically consist of tens to hundreds of discrete layers. While more layers can
lead to better performance, this approach has been challenged as far from
efficient, especially given the superiority of continuous layers demonstrated
by diffusion and flow-based models for image generation. We propose the Latent
Flow Transformer (LFT), which replaces a block of layers with a single learned
transport operator trained via flow matching, offering significant compression
while maintaining compatibility with the original architecture. Additionally,
we address the limitations of existing flow-based methods in preserving
coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M
model, LFT trained with flow matching compresses 6 of 24 layers and outperforms
directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529),
demonstrating the feasibility of this design. When trained with FW, LFT further
distills 12 layers into one while reducing the KL to 0.736 surpassing that from
skipping 3 layers (0.932), significantly narrowing the gap between
autoregressive and flow-based generation paradigms.
The Latent Flow Transformer (LFT) compresses layers by replacing them with learned transport operators using flow matching and Flow Walking, showing improved performance over layer skipping and reducing the gap between autoregressive and flow-based models.
Latent Flow Transformer LFT flow matching Flow Walking KL Divergence LM logits autoregressive flow-based generation paradigms