title: Latent Flow Transformer
summary: Transformers, the standard implementation for large language models (LLMs),
  typically consist of tens to hundreds of discrete layers. While more layers can
  lead to better performance, this approach has been challenged as far from
  efficient, especially given the superiority of continuous layers demonstrated
  by diffusion and flow-based models for image generation. We propose the Latent
  Flow Transformer (LFT), which replaces a block of layers with a single learned
  transport operator trained via flow matching, offering significant compression
  while maintaining compatibility with the original architecture. Additionally,
  we address the limitations of existing flow-based methods in preserving
  coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M
  model, LFT trained with flow matching compresses 6 of 24 layers and outperforms
  directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529),
  demonstrating the feasibility of this design. When trained with FW, LFT further
  distills 12 layers into one while reducing the KL to 0.736 surpassing that from
  skipping 3 layers (0.932), significantly narrowing the gap between
  autoregressive and flow-based generation paradigms.
paper_authors._id: 682d334862cadf615f5f73e6
paper_authors.name: Yen-Chen Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:11:09.090Z
paper_authors._id: 63ad217b9fc40b14560e9e06
paper_authors.avatarUrl: /avatars/c2d33830b141fc9c73ad8302ff35ed9d.svg
paper_authors.fullname: Yen-Chen Wu
paper_authors.type: user
paper_authors.user: yenchen
paper_authors._id: 682d334862cadf615f5f73e7
paper_authors.name: Feng-Ting Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:41:00.821Z
paper_authors._id: 643fb7332397d8eef5b844cd
paper_authors.avatarUrl: /avatars/e403a19fc13e478d5929c67028230b0e.svg
paper_authors.fullname: Feng-Ting Liao
paper_authors.type: user
paper_authors.user: FengTing
paper_authors._id: 682d334862cadf615f5f73e8
paper_authors.name: Meng-Hsi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:11:15.604Z
paper_authors._id: 66018c8eb1e509e1e4d9196f
paper_authors.avatarUrl: /avatars/5d6cbfc7b6c435264d271c958607630f.svg
paper_authors.fullname: Meng-Hsi Chen
paper_authors.type: user
paper_authors.user: menghsichen
paper_authors._id: 682d334862cadf615f5f73e9
paper_authors.name: Pei-Chen Ho
paper_authors._id: 682d334862cadf615f5f73ea
paper_authors.name: Farhang Nabiei
paper_authors._id: 682d334862cadf615f5f73eb
paper_authors.name: Da-shan Shiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:11:37.639Z
paper_authors._id: 6811b1294119e4ecc92fc93b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/srmY2yyzOg9KRDSLYXJKf.png
paper_authors.fullname: Dashan Shiu
paper_authors.type: user
paper_authors.user: dsshiu
paper_title: Latent Flow Transformer
paper_summary: Transformers, the standard implementation for large language models (LLMs),
  typically consist of tens to hundreds of discrete layers. While more layers can
  lead to better performance, this approach has been challenged as far from
  efficient, especially given the superiority of continuous layers demonstrated
  by diffusion and flow-based models for image generation. We propose the Latent
  Flow Transformer (LFT), which replaces a block of layers with a single learned
  transport operator trained via flow matching, offering significant compression
  while maintaining compatibility with the original architecture. Additionally,
  we address the limitations of existing flow-based methods in preserving
  coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M
  model, LFT trained with flow matching compresses 6 of 24 layers and outperforms
  directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529),
  demonstrating the feasibility of this design. When trained with FW, LFT further
  distills 12 layers into one while reducing the KL to 0.736 surpassing that from
  skipping 3 layers (0.932), significantly narrowing the gap between
  autoregressive and flow-based generation paradigms.
paper_ai_summary: The Latent Flow Transformer (LFT) compresses layers by replacing them with learned transport operators using flow matching and Flow Walking, showing improved performance over layer skipping and reducing the gap between autoregressive and flow-based models.
paper_ai_keywords: Latent Flow Transformer LFT flow matching Flow Walking KL Divergence LM logits autoregressive flow-based generation paradigms