title: Generative AI Beyond LLMs: System Implications of Multi-Modal Generation
summary: As the development of large-scale Generative AI models evolve beyond text
  (1D) generation to include image (2D) and video (3D) generation, processing
  spatial and temporal information presents unique challenges to quality,
  performance, and efficiency. We present the first work towards understanding
  this new system design space for multi-modal text-to-image (TTI) and
  text-to-video (TTV) generation models. Current model architecture designs are
  bifurcated into 2 categories: Diffusion- and Transformer-based models. Our
  systematic performance characterization on a suite of eight representative
  TTI/TTV models shows that after state-of-the-art optimization techniques such
  as Flash Attention are applied, Convolution accounts for up to 44% of execution
  time for Diffusion-based TTI models, while Linear layers consume up to 49% of
  execution time for Transformer-based models. We additionally observe that
  Diffusion-based TTI models resemble the Prefill stage of LLM inference, and
  benefit from 1.1-2.5x greater speedup from Flash Attention than
  Transformer-based TTI models that resemble the Decode phase. Since
  optimizations designed for LLMs do not map directly onto TTI/TTV models, we
  must conduct a thorough characterization of these workloads to gain insights
  for new optimization opportunities. In doing so, we define sequence length in
  the context of TTI/TTV models and observe sequence length can vary up to 4x in
  Diffusion model inference. We additionally observe temporal aspects of TTV
  workloads pose unique system bottlenecks, with Temporal Attention accounting
  for over 60% of total Attention time. Overall, our in-depth system performance
  characterization is a critical first step towards designing efficient and
  deployable systems for emerging TTI/TTV workloads.
paper_authors._id: 658a5bb56cf9325ae6010e73
paper_authors.name: Alicia Golden
paper_authors._id: 658a5bb56cf9325ae6010e74
paper_authors.name: Samuel Hsia
paper_authors._id: 658a5bb56cf9325ae6010e75
paper_authors.name: Fei Sun
paper_authors._id: 658a5bb56cf9325ae6010e76
paper_authors.name: Bilge Acun
paper_authors._id: 658a5bb56cf9325ae6010e77
paper_authors.name: Basil Hosmer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T13:40:19.390Z
paper_authors._id: 631183a88894270062a2ea36
paper_authors.avatarUrl: /avatars/5da43a25768f5b677bbd02bd955210e7.svg
paper_authors.fullname: Basil Hosmer
paper_authors.type: user
paper_authors.user: bhosmer
paper_authors._id: 658a5bb56cf9325ae6010e78
paper_authors.name: Yejin Lee
paper_authors._id: 658a5bb56cf9325ae6010e79
paper_authors.name: Zachary DeVito
paper_authors._id: 658a5bb56cf9325ae6010e7a
paper_authors.name: Jeff Johnson
paper_authors._id: 658a5bb56cf9325ae6010e7b
paper_authors.name: Gu-Yeon Wei
paper_authors._id: 658a5bb56cf9325ae6010e7c
paper_authors.name: David Brooks
paper_authors._id: 658a5bb56cf9325ae6010e7d
paper_authors.name: Carole-Jean Wu
paper_title: Generative AI Beyond LLMs: System Implications of Multi-Modal Generation
paper_summary: As the development of large-scale Generative AI models evolve beyond text
  (1D) generation to include image (2D) and video (3D) generation, processing
  spatial and temporal information presents unique challenges to quality,
  performance, and efficiency. We present the first work towards understanding
  this new system design space for multi-modal text-to-image (TTI) and
  text-to-video (TTV) generation models. Current model architecture designs are
  bifurcated into 2 categories: Diffusion- and Transformer-based models. Our
  systematic performance characterization on a suite of eight representative
  TTI/TTV models shows that after state-of-the-art optimization techniques such
  as Flash Attention are applied, Convolution accounts for up to 44% of execution
  time for Diffusion-based TTI models, while Linear layers consume up to 49% of
  execution time for Transformer-based models. We additionally observe that
  Diffusion-based TTI models resemble the Prefill stage of LLM inference, and
  benefit from 1.1-2.5x greater speedup from Flash Attention than
  Transformer-based TTI models that resemble the Decode phase. Since
  optimizations designed for LLMs do not map directly onto TTI/TTV models, we
  must conduct a thorough characterization of these workloads to gain insights
  for new optimization opportunities. In doing so, we define sequence length in
  the context of TTI/TTV models and observe sequence length can vary up to 4x in
  Diffusion model inference. We additionally observe temporal aspects of TTV
  workloads pose unique system bottlenecks, with Temporal Attention accounting
  for over 60% of total Attention time. Overall, our in-depth system performance
  characterization is a critical first step towards designing efficient and
  deployable systems for emerging TTI/TTV workloads.
paper_ai_keywords: Diffusion-based models Transformer-Based models Prefill stage Decode phase Flash Attention Convolution Linear layers Sequence length Temporal Attention