title: HarmoniCa: Harmonizing Training and Inference for Better Feature Cache
    in Diffusion Transformer Acceleration
summary: Diffusion Transformers (DiTs) have gained prominence for outstanding
  scalability and extraordinary performance in generative tasks. However, their
  considerable inference costs impede practical deployment. The feature cache
  mechanism, which involves storing and retrieving redundant computations across
  timesteps, holds promise for reducing per-step inference time in diffusion
  models. Most existing caching methods for DiT are manually designed. Although
  the learning-based approach attempts to optimize strategies adaptively, it
  suffers from discrepancies between training and inference, which hampers both
  the performance and acceleration ratio. Upon detailed analysis, we pinpoint
  that these discrepancies primarily stem from two aspects: (1) Prior Timestep
  Disregard, where training ignores the effect of cache usage at earlier
  timesteps, and (2) Objective Mismatch, where the training target (align
  predicted noise in each timestep) deviates from the goal of inference (generate
  the high-quality image). To alleviate these discrepancies, we propose
  HarmoniCa, a novel method that Harmonizes training and inference with a novel
  learning-based Caching framework built upon Step-Wise Denoising Training (SDT)
  and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional
  training paradigm, the newly proposed SDT maintains the continuity of the
  denoising process, enabling the model to leverage information from prior
  timesteps during training, similar to the way it operates during inference.
  Furthermore, we design IEPO, which integrates an efficient proxy mechanism to
  approximate the final image error caused by reusing the cached feature.
  Therefore, IEPO helps balance final image quality and cache utilization,
  resolving the issue of training that only considers the impact of cache usage
  on the predicted output at each timestep.
paper_authors._id: 66fe82a327821b16ec531774
paper_authors.name: Yushi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:28:38.685Z
paper_authors._id: 64b500fdf460afaefc5c64b3
paper_authors.avatarUrl: /avatars/0cb90e3fdd116e1a49209b222125c76e.svg
paper_authors.fullname: Yushi Huang
paper_authors.type: user
paper_authors.user: Harahan
paper_authors._id: 66fe82a327821b16ec531775
paper_authors.name: Zining Wang
paper_authors._id: 66fe82a327821b16ec531776
paper_authors.name: Ruihao Gong
paper_authors._id: 66fe82a327821b16ec531777
paper_authors.name: Jing Liu
paper_authors._id: 66fe82a327821b16ec531778
paper_authors.name: Xinjie Zhang
paper_authors._id: 66fe82a327821b16ec531779
paper_authors.name: Jinyang Guo
paper_authors._id: 66fe82a327821b16ec53177a
paper_authors.name: Xianglong Liu
paper_authors._id: 66fe82a327821b16ec53177b
paper_authors.name: Jun Zhang
paper_title: HarmoniCa: Harmonizing Training and Inference for Better Feature Cache
    in Diffusion Transformer Acceleration
paper_summary: Diffusion Transformers (DiTs) have gained prominence for outstanding
  scalability and extraordinary performance in generative tasks. However, their
  considerable inference costs impede practical deployment. The feature cache
  mechanism, which involves storing and retrieving redundant computations across
  timesteps, holds promise for reducing per-step inference time in diffusion
  models. Most existing caching methods for DiT are manually designed. Although
  the learning-based approach attempts to optimize strategies adaptively, it
  suffers from discrepancies between training and inference, which hampers both
  the performance and acceleration ratio. Upon detailed analysis, we pinpoint
  that these discrepancies primarily stem from two aspects: (1) Prior Timestep
  Disregard, where training ignores the effect of cache usage at earlier
  timesteps, and (2) Objective Mismatch, where the training target (align
  predicted noise in each timestep) deviates from the goal of inference (generate
  the high-quality image). To alleviate these discrepancies, we propose
  HarmoniCa, a novel method that Harmonizes training and inference with a novel
  learning-based Caching framework built upon Step-Wise Denoising Training (SDT)
  and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional
  training paradigm, the newly proposed SDT maintains the continuity of the
  denoising process, enabling the model to leverage information from prior
  timesteps during training, similar to the way it operates during inference.
  Furthermore, we design IEPO, which integrates an efficient proxy mechanism to
  approximate the final image error caused by reusing the cached feature.
  Therefore, IEPO helps balance final image quality and cache utilization,
  resolving the issue of training that only considers the impact of cache usage
  on the predicted output at each timestep.
paper_ai_summary: HarmoniCa is a novel caching method for diffusion transformers that resolves discrepancies between training and inference by maintaining denoising continuity and balancing image quality with cache utilization.
paper_ai_keywords: Diffusion Transformers feature cache mechanism inference costs caching methods learning-based approach training-discrepancy Prior Timestep Disregard Objective Mismatch HarmoniCa Step-Wise Denoising Training Image Error Proxy-Guided Objective