HarmoniCa: Harmonizing Training and Inference for Better Feature Cache
  in Diffusion Transformer Acceleration
Diffusion Transformers (DiTs) have gained prominence for outstanding
scalability and extraordinary performance in generative tasks. However, their
considerable inference costs impede practical deployment. The feature cache
mechanism, which involves storing and retrieving redundant computations across
timesteps, holds promise for reducing per-step inference time in diffusion
models. Most existing caching methods for DiT are manually designed. Although
the learning-based approach attempts to optimize strategies adaptively, it
suffers from discrepancies between training and inference, which hampers both
the performance and acceleration ratio. Upon detailed analysis, we pinpoint
that these discrepancies primarily stem from two aspects: (1) Prior Timestep
Disregard, where training ignores the effect of cache usage at earlier
timesteps, and (2) Objective Mismatch, where the training target (align
predicted noise in each timestep) deviates from the goal of inference (generate
the high-quality image). To alleviate these discrepancies, we propose
HarmoniCa, a novel method that Harmonizes training and inference with a novel
learning-based Caching framework built upon Step-Wise Denoising Training (SDT)
and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional
training paradigm, the newly proposed SDT maintains the continuity of the
denoising process, enabling the model to leverage information from prior
timesteps during training, similar to the way it operates during inference.
Furthermore, we design IEPO, which integrates an efficient proxy mechanism to
approximate the final image error caused by reusing the cached feature.
Therefore, IEPO helps balance final image quality and cache utilization,
resolving the issue of training that only considers the impact of cache usage
on the predicted output at each timestep.
66fe82a327821b16ec531774
Yushi Huang
claimed_verified
2024-10-07T07:28:38.685Z
64b500fdf460afaefc5c64b3
/avatars/0cb90e3fdd116e1a49209b222125c76e.svg
Yushi Huang
user
Harahan
66fe82a327821b16ec531775
Zining Wang
66fe82a327821b16ec531776
Ruihao Gong
66fe82a327821b16ec531777
Jing Liu
66fe82a327821b16ec531778
Xinjie Zhang
66fe82a327821b16ec531779
Jinyang Guo
66fe82a327821b16ec53177a
Xianglong Liu
66fe82a327821b16ec53177b
Jun Zhang
HarmoniCa: Harmonizing Training and Inference for Better Feature Cache
  in Diffusion Transformer Acceleration
Diffusion Transformers (DiTs) have gained prominence for outstanding
scalability and extraordinary performance in generative tasks. However, their
considerable inference costs impede practical deployment. The feature cache
mechanism, which involves storing and retrieving redundant computations across
timesteps, holds promise for reducing per-step inference time in diffusion
models. Most existing caching methods for DiT are manually designed. Although
the learning-based approach attempts to optimize strategies adaptively, it
suffers from discrepancies between training and inference, which hampers both
the performance and acceleration ratio. Upon detailed analysis, we pinpoint
that these discrepancies primarily stem from two aspects: (1) Prior Timestep
Disregard, where training ignores the effect of cache usage at earlier
timesteps, and (2) Objective Mismatch, where the training target (align
predicted noise in each timestep) deviates from the goal of inference (generate
the high-quality image). To alleviate these discrepancies, we propose
HarmoniCa, a novel method that Harmonizes training and inference with a novel
learning-based Caching framework built upon Step-Wise Denoising Training (SDT)
and Image Error Proxy-Guided Objective (IEPO). Compared to the traditional
training paradigm, the newly proposed SDT maintains the continuity of the
denoising process, enabling the model to leverage information from prior
timesteps during training, similar to the way it operates during inference.
Furthermore, we design IEPO, which integrates an efficient proxy mechanism to
approximate the final image error caused by reusing the cached feature.
Therefore, IEPO helps balance final image quality and cache utilization,
resolving the issue of training that only considers the impact of cache usage
on the predicted output at each timestep.
HarmoniCa is a novel caching method for diffusion transformers that resolves discrepancies between training and inference by maintaining denoising continuity and balancing image quality with cache utilization.
Diffusion Transformers feature cache mechanism inference costs caching methods learning-based approach training-discrepancy Prior Timestep Disregard Objective Mismatch HarmoniCa Step-Wise Denoising Training Image Error Proxy-Guided Objective