## FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation

Xiang Gao, Jiaying Liu*, Senior Member, IEEE

Abstract -Large-scale text-to-image diffusion models have been a revolutionary milestone in the evolution of generative AI and multimodal technology, allowing wonderful image generation with natural-language text prompt. However, the issue of lacking controllability of such models restricts their practical applicability for real-life content creation. Thus, attention has been focused on leveraging a reference image to control text-to-image synthesis, which is also regarded as manipulating (or editing) a reference image as per a text prompt, namely, text-driven image-to-image translation. This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plugand-play manner, realizing high-quality and versatile text-driven I2I translation without any model training, model fine-tuning, or online optimization process. To guide T2I generation with a reference image, we propose to decompose diverse guiding factors with different frequency bands of diffusion features in the DCT spectral space, and accordingly devise a novel frequency band substitution layer which realizes dynamic control of the reference image to the T2I generation result in a plug-andplay manner. We demonstrate that our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band, respectively. Extensive qualitative and quantitative experiments verify superiority of our approach over related methods in I2I translation visual quality, versatility, and controllability. The code is publicly available at: https://github.com/XiangGao1102/FBSDiff.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1aa04861-a6e6-44d1-a67c-275ba6167a9b.jpg)

Index Terms -Diffusion model, image-to-image translation, text-driven image translation.

## I. INTRODUCTION

As a typical application of the booming multimodal technology, text-driven I2I translation is an appealing computer vision problem that aims to translate a reference image as per a text prompt. It extends text-to-image (T2I) synthesis to more controllability by controlling T2I generation result with a reference image. Since the advent of CLIP [1] bridging vision and language through large-scale contrastive pre-training, attempts have been made to instruct image manipulation with text by combining CLIP with generative models. VQGANCLIP [2] pioneers text-driven image translation by optimizing VQGAN [3] latent image embedding with CLIP text-image similarity loss. DiffusionCLIP [4] fine-tunes diffusion model [5] under the same CLIP loss to manipulate an image as

Accepted by ACMMM 2024 . Xiang Gao is with Wangxuan Institute of Computer Technology, Peking University, Beijing, 100871 China, email: (gaoxiang1102@pku.edu.cn). Jiaying Liu is with Wangxuan Institute of Computer Technology, Peking University, Beijing, 100871 China, e-mail: (liujiaying@pku.edu.cn). * is the corresponding author.

Fig. 1. Based on the pre-trained text-to-image diffusion model, FBSDiff enables efficient text-driven image-to-image translation by proposing a plugand-play reference image guidance mechanism. It allows flexible control over different guiding factors (e.g., image appearance, image layout, image contours) of the reference image to the T2I generated image, simply by dynamically substituting different types of DCT frequency bands during the reverse sampling process of the diffusion model. Better viewed with zoom-in.

per a text. DiffuseIT [6] combines VIT-based structure loss [7] and CLIP-based semantic loss to guide diffusion model's reverse sampling process via manifold constrained gradient [8], synthesizing translated image that complies with the target text while maintaining the structure of the reference image. However, these methods are not competitive in generation visual quality due to the limited model capacity of backbone generative model as well as the inherent unstability caused by online fine-tuning or optimization process.

To promote image translation visual quality, efforts have been made to train large models on massive data. InstructPix2Pix [9] employs GPT-3 [10] and Stable Diffusion [11] to synthesize huge amounts of paired training data, based on which trains a supervised text-driven I2I mapping for general image manipulation task. Design Booster [12] trains a latent diffusion model [11] conditioned on both text embedding and image embedding, realizing layout-preserved text-driven I2I translation. Nevertheless, these methods are computationally

intensive in training large models from scratch and less efficient in collecting immense training data.

To circumvent formidable training costs, research has been focused on leveraging off-the-shelf large-scale T2I diffusion models for text-driven I2I translation. This type of methods further divide into fine-tuning-based methods and inversionbased methods.

The former type of fine-tuning-based methods represented by SINE [13] and Imagic [14] fine-tune the pre-trained T2I diffusion model to reconstruct an input reference image before manipulating it with a target text. These methods require separate fine-tuning of the entire diffusion model for each time of image manipulation, which is less efficient and prone to underfitting or overfitting to the reference image.

The latter type of inversion-based methods invert reference image into diffusion model's Gaussian noise space and then generate the translated image via the reverse sampling process guided by the target text. A pivotal challenge of this pipeline is that the sampling trajectory may severely deviate from the inversion trajectory due to the error accumulation caused by the classifier-free guidance technique [15], which severely impairs the correlation between the reference image and the translated image. To remedy this issue, Null-text Inversion [16] optimizes the unconditional null-text embedding to calibrate the sampling trajectory step by step. Prompt Tuning Inversion [17] proposes to minimize trajectory divergence with an optimization to encode the reference image into a learnable prompt embedding. Similarly, StyleDiffusion [18] opts to optimize the 'value' embedding of the cross-attention layer as the visual encoding of the reference image. Pix2Pix-zero [19] penalizes trajectory deviation by matching cross-attention maps between the two trajectories with least-square loss. These methods apply per-step online optimization to calibrate the whole sampling trajectory, introducing additional computational cost and time overhead. Moreover, most of these methods adopt the cross-attention control technique introduced in Promptto-Prompt [20] for image structure preservation. This makes them rely on a paired source text of the reference image, which is not flexible or even available in most cases. Plug-andPlay (PAP) [21] proposes to leverage feature maps and selfattention maps extracted from internal layers of the denoising U-Net to maintain image structure, realizing optimization-free text-driven I2I translation. However, the algorithm is sensitive to specific layer selection, and the feature extraction process is also time-consuming.

In this paper, we propose a concise and efficient approach termed FBSDiff, realizing plug-and-play and highly controllable text-driven I2I translation from a frequency-domain perspective. To guide T2I generation with a reference image, a key missing ingredient of existing methods is the mechanism to control the guiding factor (e.g., image appearance, layout, contours) and guiding intensity of the reference image. Since different image guiding factors are difficult to isolate in the spatial domain, we consider decomposing them in the frequency domain by modeling them with different frequency bands of diffusion features in the Discrete Cosine Transform (DCT) spectral space. Based on this motivation, we propose an inversion-based text-driven I2I translation framework featured with a novel frequency band substitution mechanism, which efficiently enables reference image guidance of the T2I generation by dynamically substituting a certain DCT frequency band of diffusion features with the corresponding counterpart of the reference image along the reverse sampling process. As displayed in Fig. 1, T2I generation with appearance and layout control, pure layout control, and contour control of the reference image can be respectively realized by transplanting low-frequency band, mid-frequency band, and high-frequency band between diffusion features, allowing versatile and highly controllable text-driven I2I translation.

The strengths of our method are fourfold: (I) plug-and-play efficiency: our method extends pre-trained T2I diffusion model to the realm of I2I in a plug-and-play manner; (II) conciseness: our method dispenses with the need for the paired source text of the reference image as well as cumbersome attention modulation process as compared with existing advanced methods, all while achieving leading I2I translation performance; (III) model generalizability: our method transplants frequency band of diffusion features along the reverse sampling trajectory, requiring no access to any internal features of the denoising network, and thus decouples with the specific diffusion model backbone architecture as compared with existing methods; (IV) controllability: our method allows flexible control over the guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band. The effectiveness of our method is fully demonstrated both qualitatively and quantitatively. To summarize, we make the following key contributions:

- · We provide new insights about controllable diffusion process from a novel frequency-domain perspective.
- · We propose a novel frequency band substitution technique, realizing plug-and-play text-driven I2I translation without any model training, model fine-tuning, and online optimization process.
- · We contribute a concise and efficient text-driven I2I framework that is free from source text and cumbersome attention modulations, highly controllable in both guiding factor and guiding intensity of the reference image, and invariant to the architecture of the used diffusion model backbone, all while achieving superior I2I translation performance compared with existing advanced methods.

## II. RELATED WORK

## A. Diffusion Model

Since the advent of DDPM [5], diffusion model has soon dominated the family of generative models [22]. Afterwards, much progress have been made to improve diffusion model in both methodology and application. DDIM [23] and its variants [24], [24] accelerate diffusion model sampling process to tens of times with only marginal drop in generation quality, promoting its practicability dramatically. Palette [25] extends diffusion model from unconditional image generation to the realm of conditional image synthesis, opening the door of diffusionbased image-to-image translation. With the advancement of multimodal technology, large-scale T2I diffusion models [26], [27], [28] are proposed to generate high-resolution images

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b6f33631-5315-421f-a3e3-a0cf4d310b4c.jpg)

Fig. 2. Overview of FBSDiff. Based on the pre-trained latent diffusion model (LDM), FBSDiff starts with an inversion trajectory that inverts reference image into the LDM Gaussian noise space, then a reconstruction trajectory is applied to reconstruct the reference image from the inverted Gaussian noise, providing intermediate denoising results as pivotal guidance features. The guidance features are leveraged to guide the text-driven sampling trajectory of the LDM to exert reference image control, which is realized by dynamically transplanting certain DCT frequency bands from diffusion features along the reconstruction trajectory into the corresponding features along the sampling trajectory. The dynamic DCT frequency band transplantation is implemented in a plug-and-play manner with our proposed frequency band substitution layer (FBS layer).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f5928d72-9dfa-497e-8e9d-c77ae24ebcf5.jpg)

## B. Computer Vision in Frequency Perspective

Fig. 3. Illustration of the proposed frequency band substitution (FBS) layer. The FBS layer takes in two diffusion features and substitutes a certain frequency band of one feature with the corresponding frequency band of the other feature. This is realized by converting the two diffusion features into the frequency domain via 2D DCT, extracting and transplanting a certain DCT frequency band, and converting the fused DCT features back to spatial domain via 2D IDCT. The frequency band extraction and transplantation are implemented with binary masking.

with open-domain text prompts, bringing content creation to an unprecedented level. To lower computational overhead of large-scale T2I model, Latent Diffusion Model (LDM) [11] proposes to transfer diffusion model from high-dimension pixel space to low-dimensional feature space, contributing the most widely used architecture in AIGC industry. To introduce more controllability to T2I synthesis, ControlNet [29] and T2iadapter [30] add spatial control to T2I diffusion models by training a control module of the denoising U-Net conditioned on certain image priors (e.g., canny edges, depth maps, human key points, etc.). SDXL [31] and DiTs [32] propose Transformer [33] based backbone denoising network, improving T2I diffusion model to larger capacity. Up to now, diffusion model has been applied to a wide variety of vision fields such as image super-resolution [34], image inpainting [35], image colorization [36], semantic segmentation [37], point cloud generation [38], video synthesis [39], 3D reconstruction [40], etc, and is still making rapid progress in theory and potential applications.

Deep neural network models are mostly applied to tackle vision tasks in spatial or temporal domain, some research reveals that model performance can also be boosted from frequency domain. For example, Ghosh et al. [41] introduce DCT to convolutional neural network for image classification, accelerating network convergence speed. Xie et al. [42] propose a frequency-aware dynamic network for lightweight image super-resolution. Cai et al. [43] impose Fourier frequency spectrum consistency to image translation tasks, achieving better identity preservation ability. FreeU [44] improves T2I generation quality by selectively enhancing or depressing different frequency components of diffusion features inside the denoising U-Net model. ILVR [45] proposes to fuse lowfrequency information of the forward diffusion process into the reverse sampling process for conditioned image synthesis. Our method differs with ILVR in that ILVR simulates lowpass filtering with simple feature downsampling and upsampling and performs information fusion in the spatial domain, while our method explicitly extracts and transplants frequency bands of diffusion features in the DCT domain. FCDiffusion [46] shares similar spirit of frequency-based control of T2I diffusion model with our method. However, FCDiffusion relies on training multiple frequency control branches to realize versatile control effects, while our method achieves versatility and high controllability in both guiding factor and guiding intensity of the reference image in a training-free and plugand-play manner.

## III. METHOD

In this section, we first describe the overall model architecture of our FBSDiff, then elaborate on our proposed frequency band substitution mechanism, and finally summarize our algorithm and describe implementation details. For the diffusion model background, please refer to the Appendix.

## A. Overall Architecture

Established on the pre-trained Latent Diffusion Model (LDM), FBSDiff adapts it from T2I generation to the realm of text-driven I2I translation with our proposed plug-andplay reference image guidance mechanism: dynamic frequency band substitution, which efficiently realizes flexible control over both guiding factor and guiding intensity of the reference image to the T2I generated image.

As Fig. 2 shows, FBSDiff comprises three diffusion trajectories: (i) inversion trajectory ( z 0 → z T inv ); (ii) reconstruction trajectory ( z T inv = ˆ z T → ˆ z 0 ≈ z 0 ); (iii) sampling trajectory ( ˜ z T → ˜ z 0 ). Starting from the initial feature z 0 = E x ( ) extracted from the reference image x by the LDM encoder E , a T inv -step DDIM inversion is employed to project z 0 into the Gaussian noise latent space conditioned on the null-text embedding v ∅ , based on the assumption that the ODE process can be reversed in the limit of small steps:

$$z _ { t + 1 } = \sqrt { \bar { \alpha } _ { t + 1 } } f _ { \theta } ( z _ { t }, t, v _ { \emptyset } ) + \sqrt { 1 - \bar { \alpha } _ { t + 1 } } \epsilon _ { \theta } ( z _ { t }, t, v _ { \emptyset } ), \ \ ( 1 ) \ \text{the scan}$$

$$\tilde { \mu } _ { + 1 } & \quad v \, \cdots \, \iota _ { + 1 }, \iota _ { \, \nu } \, \cdots \, \iota _ { \, \nu } / \, \cdot \, v \, \cdots \, \iota _ { + 1 } \, \cdots \, \iota _ { \, \nu } \, \cdots \, \iota _ { \, \nu } / \, \cdots \, \iota _ { \, \nu } \, \cdots \, \iota _ { \, \nu } \, \cdots \\ & f _ { \theta } ( z _ { t }, t, v _ { \emptyset } ) = \frac { z _ { t } - \sqrt { 1 - \bar { \alpha } _ { t } } \epsilon _ { \theta } ( z _ { t }, t, v _ { \emptyset } ) } { \sqrt { \bar { \alpha } _ { t } } }, \quad \ ( 2 ) \quad \text{form}$$

where { ¯ α t } are schedule parameters that follows the same setting as DDPM [5], ϵ θ is the denoising U-Net of the pretrained LDM. The Gaussian noise z T inv obtained after the T inv -step DDIM inversion is directly used as the initial noise feature of the subsequent reconstruction trajectory, which is a T -step DDIM sampling process that reconstructs ˆ z 0 ≈ z 0 from the inverted noise feature ˆ z T = z T inv :

$$\hat { z } _ { t - 1 } = \sqrt { \bar { \alpha } _ { t - 1 } } f _ { \theta } ( \hat { z } _ { t }, t, v _ { \emptyset } ) + \sqrt { 1 - \bar { \alpha } _ { t - 1 } } \epsilon _ { \theta } ( \hat { z } _ { t }, t, v _ { \emptyset } ), \ \ ( 3 ) \ \underset { \text{produc} } { \text{decode} }$$

in which f θ (ˆ z , t, v t ∅ ) follows the same form as Eq. 2. The length of the reconstruction trajectory could be much smaller than that of the inversion trajectory (i.e., T ≪ T inv ) to save inference time. The reconstruction trajectory is conditioned on the same null-text embedding v ∅ as the inversion trajectory to ensure feature reconstructability (i.e., ˆ z 0 ≈ z 0 ).

Meanwhile, an equal-length sampling trajectory is applied in parallel with the reconstruction trajectory for T2I synthesis. The sampling trajectory is also a T -step DDIM sampling process that progressively denoises a randomly initialized Gaussian noise feature ˜ x T ∼ N (0 , I ) into ˜ x 0 conditioned on the text embedding v of the target text prompt. To amplify the effect of text guidance, we employ classifier-free guidance technique [15] by interpolating conditional (target text) and unconditional (null text) noise prediction at each time step with a guidance scale ω along the sampling process:

$$\tilde { z } _ { t - 1 } = \sqrt { \bar { \alpha } _ { t - 1 } } f _ { \theta } ( \tilde { z } _ { t }, t, v, v _ { \emptyset } ) + \sqrt { 1 - \bar { \alpha } _ { t - 1 } } \epsilon _ { \theta } ( \tilde { z } _ { t }, t, v, v _ { \emptyset } ), \quad \text{threshold} \quad \text{and ban} \\ \tilde { z } _ { \ast } - \sqrt { 1 - \bar { \alpha } _ { \ast } } \epsilon _ { \theta } ( \tilde { z } _ { \ast }, t, v, v _ { \omega } ) \quad \text{sign thru}.$$

$$f _ { \theta } ( \tilde { z } _ { t }, t, v, v _ { \emptyset } ) = \frac { \tilde { z } _ { t } - \sqrt { 1 - \bar { \alpha } _ { t } } \epsilon _ { \theta } ( \tilde { z } _ { t }, t, v, v _ { \emptyset } ) } { \sqrt { \bar { \alpha } _ { t } } }, \quad ^ { \searrow \prime } ( 5 ) \quad \text{$\min_{pma}$}$$

$$\epsilon _ { \theta } ( \tilde { z } _ { t }, t, v, v _ { \emptyset } ) = \omega \cdot \epsilon _ { \theta } ( \tilde { z } _ { t }, t, v ) + ( 1 - \omega ) \cdot \epsilon _ { \theta } ( \tilde { z } _ { t }, t, v _ { \emptyset } ). \ ( 6 ) \quad \text{$\mathbb{ } M_{a}$}.$$

Due to the inherent property of DDIM inversion and DDIM sampling, the reconstruction trajectory forms a deterministic denoising mapping towards the reference image, during which the intermediate denoising results { ˆ z t } can function as pivotal guidance features to calibrate the corresponding counterparts { ˜ z t } along the sampling trajectory. Thus, correlation between the reference image and the generated image can be established to allow for text-driven I2I translation. Specifically, we implement feature calibration by inserting a plug-andplay frequency band substitution (FBS) layer in between the reconstruction trajectory and the sampling trajectory. FBS layer substitutes a certain frequency band of ˜ z t in the sampling trajectory with the corresponding frequency band of ˆ z t in the reconstruction trajectory along the reverse sampling process. The frequency band substitution effectively and efficiently imposes guidance of the reference image to the T2I synthesis process. Both the guiding factor (e.g., image appearance, image layout, image contours) and guiding intensity of the reference image can be flexibly controlled simply by tuning the type and bandwidth of the substituted frequency band, respectively.

To improve I2I translation visual quality, we partition the sampling process into a calibration phase and a noncalibration phase, separated by the time step λT . In the former calibration phase ( ˜ z T → ˜ z λT ), dynamic frequency band substitution is applied at each time step for smooth calibration of the sampling trajectory; in the latter non-calibration phase ( ˜ z λT -1 → ˜ z 0 ), we remove FBS layer to avoid over-constrained sampling result, fully unleashing the generative power of the pre-trained T2I model to improve image generation quality. Here λ denotes the ratio of the length of the non-calibration phase to that of the entire sampling trajectory.

At last, the final result ˜ z 0 of the sampling trajectory is decoded back to the image space via the LDM decoder D , producing the final translated image ˜ x , i.e., ˜ = x D z (˜ ) 0 .

## B. Frequency Band Substitution Layer

As Fig. 3 illustrates, the FBS layer takes in a pair of diffusion features ˆ z t and ˜ z t , converts them from the spatial domain into the frequency domain via 2D-DCT, then transplants a certain frequency band in the DCT spectrum of ˆ z t to the same position in the DCT spectrum of ˜ z t . Finally, 2D-IDCT is applied to transform the manipulated DCT spectrum of ˜ z t back into the spatial domain as the final calibrated feature.

In 2D DCT spectrum, elements with smaller coordinates (nearer to the top-left origin) encode lower-frequency information while larger-coordinate elements (nearer to the bottomright corner) correspond to higher-frequency components. Most of the DCT spectral energy is occupied by a small proportion of low-frequency elements near the top-left origin.

In FBS layer, the sum of 2D coordinates is used as the threshold to extract DCT frequency bands of different types and bandwidths through binary masking. Specifically, we design three types of binary masks which are respectively termed the low-pass mask ( Mask lp ), high-pass mask ( Mask hp ), and mid-pass mask ( Mask mp ):

$$v _ { \emptyset } ). \ ( 6 ) \quad \begin{cases} \stackrel {. } { \ M a s k _ { l p } ( x, y ) = 1 } \ \stackrel {. } { i f } \ x + y \leq t h _ { l p } \ e l s e \ 0, \\ \ M a s k _ { h p } ( x, y ) = 1 \ \stackrel {. } { i f } \ x + y > t h _ { h p } \ e l s e \ 0, \\ \ M a s k _ { m p } ( x, y ) = 1 \ \stackrel {. } { i f } \ t h _ { m p 1 } < x + y \leq t h _ { m p 2 } \ e l s e \ 0, \end{cases}$$

where th lp is the threshold of the low-pass filtering; th hp is the threshold of the high-pass filtering; th mp 1 and th mp 2

$$\psi$$

## Algorithm 1 Complete algorithm of FBSDiff

Input: the reference image x and the target text. Output: the translated image ˜ x .

- 1: Extract the initial latent feature z 0 = E x ( ) .
- 2: for t = 0 to T inv -1 do
- 3: compute z t +1 from z t via Eq. 1;
- 4: end for { DDIM inversion }
- 5: Initialize ˆ z T = z T inv , ˜ z T ∼ N (0 , I ) .
- 6: for t = T to λT +1 do
- 7: compute ˆ z t -1 from ˆ z t via Eq. 3;
- 8: compute ˜ z t -1 from ˜ z t via Eq. 4;
- 9: substitute a certain frequency band of ˜ z t -1 with the corresponding counterpart of ˆ z t -1 via Eq. 7;
- 10: end for { DDIM sampling in the calibration phase }
- 11: for t = λT to 1 do
- 12: compute ˜ z t -1 from ˜ z via Eq. 4;
- 13: end for { DDIM sampling in the non-calibration phase }
- 14: Obtain ˜ z 0 and the final translated image ˜ = x D z (˜ ) 0 .

are respectively the lower bound and upper bound of the mid-pass filtering. Given a binary mask Mask ∗ ∈ { Mask lp , Mask hp , Mask mp } , the frequency band substitution operation in the FBS layer can be formulated as:

$$\tilde { z } _ { t } = & I D C T ( D C T ( \hat { z } _ { t } ) \cdot M a s k _ { * } + \begin{array} { c } \text{dfter} \\ D C T ( \tilde { z } _ { t } ) \cdot ( 1 - M a s k _ { * } ) ), \end{array} \quad ( 7 ) \quad \text{trans} \text{f} \\ & \text{diffus}$$

where DCT and IDCT refer to the 2D-DCT and 2D-IDCT transformations respectively, which are introduced in detail in the Appendix section. The use of the low-pass mask Mask lp , high-pass mask Mask hp , and mid-pass mask Mask mp respectively corresponds to the extraction and substitution of the low-frequency band, high-frequency band, and mid-frequency band, which controls different guiding factors of the reference image to the T2I generated result:

- · Low-frequency band substitution enables low-frequency information guidance of the reference image x , realizing image appearance (e.g., color, luminance) and layout control over the generated image ˜ x ;
- · High-frequency band substitution enables highfrequency information guidance of x , realizing image contour control over the generated image ˜ x ;
- · Mid-frequency band substitution enables mid-frequency information guidance of the reference image x . By filtering out higher-frequency contour information and lowerfrequency appearance information in the DCT spectrum, the substitution of the mid-frequency band realizes only image layout control over the generated image ˜ x .

The DCT masking type and the corresponding thresholds used in the FBS layer are hyper-parameters of our method, which could be flexibly modulated to enable control over diverse guiding factors and continuous guiding intensity of the reference image x to the T2I generated image ˜ x .

## C. Implementation Details

We use the pre-trained Stable Diffusion v1.5 as the backbone diffusion model and set the classifier-free guidance scale

ω = 7 5 . . We use 1000-step DDIM inversion to ensure highquality reconstruction, i.e., T inv =1000, and use 50-step DDIM sampling for both the reconstruction and sampling trajectory, i.e., T =50. Along the sampling trajectory, we allocate 55% time steps to the calibration phase and the remaining 45% steps for the non-calibration phase, i.e., λ =0.45. For the default DCT masking thresholds used in the FBS layer, we set th lp =80 for low-frequency band substitution (low-FBS); th hp =5 for highfrequency band substitution (high-FBS); th mp 1 =5, th mp 2 =80 for mid-frequency band substitution (mid-FBS). The complete algorithm of FBSDiff is presented in Alg. 1.

## IV. EXPERIMENTS

In this section, we first present and analyze the qualitative results of our method as well as qualitative comparison with related advanced methods; then we delve into the frequency band substitution mechanism in detail with ablation studies; finally, we show quantitative evaluations of our method and related approaches.

## A. Qualitative Results

Example text-driven I2I translation results of our method are shown in Fig. 4. Our method effectively decomposes different guiding factors of the reference image by dynamically transplanting different types of DCT frequency bands of diffusion features. The low-FBS transfers low-frequency information of the reference image into the sampling trajectory, producing translated image that inherits the original image appearance and layout. In the mode of high-FBS that dynamically transplants high-frequency components of diffusion features, the generated image is aligned with the reference image in high-frequency contours while the low-frequency appearance is not restricted. Results of mid-FBS maintain only overall image layout of the reference image, since the lower-frequency appearance information and higher-frequency contour information of the reference image are filtered out in the DCT domain. For all three modes of frequency band substitution, the image translation results exhibit high visual quality and high text fidelity for both real-world and artisticstyle reference images.

The control of our method over different guiding factors of the reference image is more clearly demonstrated in Fig. 5. The T2I generated image maintains the appearance and layout of the reference image with low-FBS; preserves detailed image contours of the reference image with high-FBS; and inherits pure image layout with mid-FBS.

We qualitatively compare our method with SOTA textdriven I2I translation methods including Plug-and-Play (PAP) [21], Null-text Inversion (Null-text) [16], Pix2Pix-zero [19], InstructPix2Pix (InsPix2Pix) [9], Prompt Tuning Inversion (PT-inversion) [17], StyleDiffusion [18], and VQGAN-CLIP (VQCLIP) [2], results are displayed in Fig. 7. The top panel of Fig. 7 shows that our method with low-FBS achieves better appearance consistency between the reference image and the translated result than related approaches, and is thus better suited to image creation scenario which favors inheriting the appearance and style from an existing image.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a326d65d-a93f-4686-af06-d7929bb7a6be.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-51a0be52-ee24-4767-9c16-cb99ab803d89.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d260e5e0-ff28-4f2e-b8cd-6b670c8b2f1a.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d96723af-64df-4757-9605-8123706cc935.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-98b3f25d-5607-44cb-ac72-dac06a167937.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-49a42298-555b-4f75-98ff-749923e798ac.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9c4c71dc-b252-40ed-9c3f-9d449c6d788b.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f72dc569-b8fb-463a-a001-6febbd9bfbce.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-98d868a3-b620-4cfa-8150-c5bdd334f9c2.jpg)

ň.QPFQPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;6QYGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-aacbf500-1b28-45ea-9a8c-6e8aa4f2c180.jpg)

ň[QWPIglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-824a5ba3-01f4-4931-8016-cfcce6ca518c.jpg)

ňECUVNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-95e9766d-6ea1-4f64-9db0-c615ee2a0df1.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d47f9336-fdf2-4e93-8a9c-6dc79a3733ce.jpg)

ňYCVGTglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;EQNQTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4f8b006b-a843-4a58-95c4-51fb0f3514b4.jpg)

ňFGUGTVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b1131ef2-f460-4ac1-ae34-18beeec3ca59.jpg)

ňUPQYglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b566fcd4-7fd2-418c-aebb-50ff6b5b1237.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1e356692-5799-4b8a-a380-deee7effab59.jpg)

ň0QVTGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;&amp;COGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-83d81315-66d3-4ee2-896b-c2a0d3a0705e.jpg)

ňGNFGTN[glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-91e2f1f3-2d7b-4677-8901-4a68f64fcce0.jpg)

ňCPEKGPVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-dd77b6ac-9aaa-4450-8faf-3c0a7fb4b7c1.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-118ac6eb-9473-49b5-95e9-e642d41b8168.jpg)

ňRGPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UMGVEJŉ

ň8CPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;)QIJglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-943b7d19-8c96-46c4-ae31-6d8b142adb44.jpg)

ňURCEGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UVCVKQPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-c084fe63-4478-40a4-968c-fd967e746dd1.jpg)

ň2CTVJGPQPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; 6GORNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6b595861-c4eb-48b0-9cad-de688ff0db36.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-31c5b72b-ddf0-47b8-9008-628f24ca3e79.jpg)

ňNKVVNGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;DQ[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a4e0405f-4864-4940-a43d-1a38bdbcc381.jpg)

ňHCEVQTKGUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-52bd7bbe-6c8f-41df-ac33-c38dcf0af194.jpg)

ňFGUGTVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8cd393f4-64d5-42eb-b5c0-c4137499902b.jpg)

ň6TKWORJCNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; #TEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b19bdfdd-1cf6-42d1-b6ad-4450544da3ba.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4f64d35a-28ba-4988-a36a-9012e1551dc5.jpg)

ňUQNFKGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b57f5ea3-4478-47f4-8e0c-a08af2401409.jpg)

ňOQWPVCKPUŉ

ňUVTGCOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f3704322-b6d7-49cc-85b5-f4c06e920875.jpg)

ňTQDQVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ff370a21-c407-4cfb-a176-1daeb40f841c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-05f070e0-0aeb-429d-a4fd-777c05d171f5.jpg)

ňYQQFGPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UEWNRVWTGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-85926f66-5e78-455e-94a6-15a18aadb4c7.jpg)

ň/QPGVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; RCKPVKPIŉ

ňDKTVJFC[glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;ECMGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-62df84a6-99ac-4a0b-a17f-ddd81ea960ab.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6c107f98-8637-4088-8edb-b78a98b0fd78.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a595c0fe-fbcc-46dd-a2ae-9cabe65ecfd5.jpg)

ňCPEKGPVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TWKPUŉ

ňHWVWTKUVKEglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;EKV[ŉ

ňQNFglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;JQWUGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-2d96ae77-cada-4e61-b516-8997d8017637.jpg)

4GHGTGPEG

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b2b49632-b0a5-413b-bd25-98c726c0d446.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7923041f-99be-4a76-896d-f668b03d8882.jpg)

ňCPEKGPVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TWKPUŉ ňOQFGTPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;DWKNFKPIUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e846b7b4-94ac-4bc0-8436-d7f5433c78ff.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-16ab49df-f64e-4bee-b1f4-505e6dcd0715.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-14abf201-e6b0-4c84-a503-9e23e977074e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3af181d0-1da7-41c9-a7a9-7615817b8d7f.jpg)

ňVGORNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b3d61830-25a7-4b42-a4e9-27f2af1f942f.jpg)

ň2CTNKCOGPVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;*QWUGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f967be38-ddde-45f9-9cd0-72d6905828cb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5d317d2c-5fa8-4aef-a0d8-582379b652c6.jpg)

ň[QWPIglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;NCF[ŉ

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cd6e9840-62f1-4294-a7f6-1a8759d9418c.jpg)

ňNKVVNGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;IKTNŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b9b8e646-f3d5-4e41-a7c9-4bafa300ca40.jpg)

ňOQFGTPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;DWKNFKPIUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-08b3769d-3897-4346-8f2f-acc5abbc87bf.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1ee765b4-66a1-45d8-903a-3e5005d56184.jpg)

ňHCEVQTKGUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-da143794-54c2-44ef-8089-32dae586eb35.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-783d397b-a8a4-447a-a7bb-ceb098ad6b4f.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-fcc95158-43aa-455b-8805-740a7ec26410.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1c687e15-6b9c-43c1-8058-ffd880836c6e.jpg)

4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b4c5edab-8cb1-4dfb-9d96-c358ded17265.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8cd8e08c-db35-45fd-8b70-3d189f19bfaa.jpg)

ňRGPEKNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UMGVEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-fda55b06-f0de-4307-9b9b-49a4d2c70524.jpg)

ňYCVGTglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;EQNQTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f9c457af-8304-45ad-92b7-34d5f0df8b51.jpg)

ňOWUKEglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TQQOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3af8b2a5-37c0-4f4d-be1a-150fe686e2d0.jpg)

ň%JKPGUGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;KPMglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; YCUJglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;RCKPVKPIŉ

ň.')1glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;DTKEMŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-c11f935d-9823-411d-97c6-a38f52309fea.jpg)

ň.')1glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;DTKEMŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-33cb1034-8bb2-4b3f-9694-4e56738bae9c.jpg)

4GHGTGPEG

ňCPEKGPVglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f80907c6-f172-4ac7-b450-dace62973d13.jpg)

ňGNFGTN[glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b94b5be1-7ce7-4d95-ad8b-b597245aa598.jpg)

ňVQYPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UVTGGVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-27815be6-04f1-4cf1-8a63-a7edcdd326f8.jpg)

ňETC[QPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;FTCYKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7307bd35-c581-4352-b571-6de11694eee9.jpg)

ňQKNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1a2164ea-ad4f-4a9d-85ea-58094511bc73.jpg)

ňRGPEKNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UMGVEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e9dae32c-7e15-4793-999b-10c9fa7ade4c.jpg)

ňNKDTCT[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ca16bbce-7d13-4b42-894c-cd294d561863.jpg)

ňGNFGTN[glyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-38056fc5-c474-4cb7-a3d7-704a97e17ebe.jpg)

ňVGORNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-565f2a0d-10b0-40b2-8a7b-e7e13d724c7f.jpg)

ňYCVGTglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;EQNQTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4592cbf2-868d-4e63-be42-42b5c3e8f639.jpg)

ň#XCVCTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-04c2578c-71a3-473e-81f3-f49f752af2ee.jpg)

ňYQQFGPglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; UEWNRVWTGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5a02abc4-9acf-4469-877d-87fcaa0479e9.jpg)

ňQHHKEGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;TQQOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9b33ca91-5e7b-44df-9c3c-84ead9dcb9b6.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cc69c86a-1f1a-43c1-91ff-7fbe732acac4.jpg)

ňYCVGTglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;EQNQTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4382ab01-36b6-41de-b057-4a03d857e9d1.jpg)

ňQKNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;RCKPVKPIŉ

ňDCVJTQQOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e4488699-32ab-43ce-a5b2-d20c258dcb59.jpg)

ňTQ[CNglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;RCNCEGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f692bcc8-0ca8-4f05-89c9-18cd37f6b94b.jpg)

ňRCKPVKPIglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;QHglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt; Cglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;ICTFGPŉ

ňURCEGglyph&lt;c=2,font=/VJBKLI+HanziPenSC-W3&gt;UVCVKQPŉ

Appearance and layout control with low-frequency band substitution

Contour control with high-frequency band substitution

Layout control with mid-frequency band substitution

Fig. 4. Qualitative results of our method with different types of frequency band substitution. For low-frequency band substitution (low-FBS), the generated image is controlled by the reference image in terms of image appearance and layout; for high-frequency band substitution (high-FBS), the reference image controls image contours of the generated image; as for mid-frequency band substitution (mid-FBS), only image layout of the generated image is controlled by the reference image. Better viewed with zoom-in .

Fig. 5. Comparison among different reference image control effects achieved by low-FBS, mid-FBS, and high-FBS. Low-FBS controls image appearance and layout, mid-FBS controls only image layout, and high-FBS controls image contours.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-df7ae3e5-6504-4111-a304-2d3942f191df.jpg)

The bottom panel of Fig. 7 shows that existing SOTA textdriven I2I methods struggle at producing I2I results with large appearance change from the reference images, while our method with high-FBS excels in generating images with significantly different appearance, and is thus more suitable to image creation scenario where appearance divergence is pursued. Among the compared approaches, our method is the only one that enables flexible control over different guiding factors of the reference image, and is also the only approach that simultaneously dispenses with model training, fine-tuning, online optimization, and attention modulations.

An advantage of our approach over related methods is sampling diversity. As displayed in Fig. 6, our FBSDiff can produce diverse text-guided I2I results by randomly sampling ˜ x T from isotropic Gaussian distribution, while other inversionbased methods [16], [21], [17], [19], [18] lack such sampling diversity due to directly initializing ˜ x T with the inverted feature embedding of the reference image.

The importance of frequency band substitution (FBS) for reference image control is clearly shown in Fig. 8, from which we see that low-FBS establishes appearance and layout correlations between the reference and the generated images, while removing frequency band substitution leads to results without any correlation to the reference images. Moreover, as Fig. 9 displays, our method robustly adapts to varying degrees of semantic gap between the reference image and the target text prompt. The translated image of our method can still comply with the target text accurately with satisfying visual quality even in the case of very large image-text semantic discrepancy.

Besides the controllability in the guiding factors of the reference image, our method also allows continuous control over the guiding intensity simply by modulating the bandwidth

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7137896a-56a5-42dc-ab2d-f0e0a076239d.jpg)

Reference

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6e2172a9-a199-44a0-9ffc-a220511edb65.jpg)

Reference

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-14361d7f-3046-4ede-b26e-d1cb43e58737.jpg)

Reference

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1485ca1b-aa3d-469a-9f11-555aed77d7f6.jpg)

## Text prompt: ňRKEVWTGglyph&lt;c=2,font=/VNZZEB+HanziPenSC-W5&gt;QHglyph&lt;c=2,font=/VNZZEB+HanziPenSC-W5&gt;Cglyph&lt;c=2,font=/VNZZEB+HanziPenSC-W5&gt;TQDQVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1e6b906d-a238-4b37-87f5-36407a91c348.jpg)

Diversified sampling results of our FBSDiff

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ef167895-3b77-4f77-aef0-c6762e42412b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5bb2621b-062d-4c7a-b87d-973eed32f19e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e4712c43-c482-4e89-88d0-d48af349c5fb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ddfc5c35-d889-4837-aa90-e412046aa2c4.jpg)

Unique sampling result of Null-text Inversion

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8149f831-b1da-433a-8ab5-a14f842c1303.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-21cf6a86-e338-4290-ba66-608cd48e0493.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e54c6350-ce0d-4966-9592-f2b8ff01426e.jpg)

Text prompt:

ňCPEKGPV TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-57feaa3f-eea3-41d2-bed7-aaea6fae1d40.jpg)

Diversified sampling results of our FBSDiff

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6cad3f47-8370-44f6-9371-04a9ded9ba99.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-65418b47-687a-4421-8e41-18309f219fe0.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e845d7d4-9624-4495-adbf-1e0f0a00a19c.jpg)

Reference

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-74fb16a3-d4e3-4357-9023-3bf12861a05c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-c83a56c5-093a-4845-8341-931f94ce51de.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-95847892-d912-4bb7-9bc8-b1c68e64a25f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a0bcab13-48b5-4c63-9864-e11ee8b0ee43.jpg)

Unique sampling result of Null-text Inversion

Fig. 6. Our method enables diverse sampling results for fixed reference image and text prompt, as contrasted with Null-text Inversion that produces unique text-driven I2I result. Our method also produces results with better visual quality than Null-text Inversion.

of the substituted frequency band. Results displayed in Fig. 10 demonstrate the image appearance and layout guiding intensity control of our method by adjusting the low-pass filtering threshold th lp in the mode of low-FBS. Enlarging the value of th lp widens the bandwidth of the transplanted low-frequency band and thus increases the amount of guiding information of the reference image, leading to the translated image with more resemblance to the reference image. Conversely, lowering the value of th lp narrows the bandwidth of the substituted frequency band, which reduces the amount of guiding information and thus brings more variations to the translated result as compared with the reference image.

Likewise, results in Fig. 11 demonstrate the image contour guiding intensity control of our method by adjusting the midpass filtering upper bound threshold th mp 2 in the mode of mid-FBS. When increasing the value of th mp 2 , more highfrequency components of the reference image (high-frequency guiding information) are included into the transplanted frequency band and transferred to the sampling trajectory, which results in more consistent image contours between the reference image and the translated image. On the contrary, decreasing the value of th mp 2 shrinks the transplanted highfrequency guiding information and thus leads to weaker image contour consistency.

## B. Ablation Study

To verify the rationality and effectiveness of our proposed method, we also explore and compare with other designs of

Fig. 7. Qualitative method comparisons. Our FBSDiff with low-FBS is more adept at appearance preservation than related methods, which better suits to I2I task pursuing appearance consistency between the reference image and the generated image (top panel). Conversely, our method with high-FBS remarkably facilitates I2I appearance change compared with related methods, which better suits to I2I task pursuing appearance divergence (bottom panel).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-433b3e4e-7286-4d73-84dd-e04eab1c6c27.jpg)

ň6JGglyph&lt;c=2,font=/RKBCUN+HanziPenSC-W3&gt;9KPFUQTglyph&lt;c=2,font=/RKBCUN+HanziPenSC-W3&gt;%CUVNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-73efe39f-e34b-47eb-80a1-c412d2b4ac37.jpg)

ňCPEKGPVglyph&lt;c=2,font=/RKBCUN+HanziPenSC-W3&gt;TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-781ae9fb-c69b-4469-9aef-4cd3ba14b1e6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-674631c3-1ed2-4f84-b911-7d15b9dfe068.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7af32f20-76b0-4746-92bc-2dd55fe10eae.jpg)

Reference

W/ low-FBS

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-31900255-a519-41ab-bb1f-1fde7dcd0eb4.jpg)

W/O FBS

Fig. 8. Comparison between results of our method with low-FBS and without frequency band substitution.

frequency band substitution, including substituting the frequency band only once at λT time step rather than along the whole calibration phase (which we denote as Once Substitution ), and substituting the full DCT spectrum rather other only a partial frequency band of it (which we refer to as Full Substitution ).

intermediate time step will cause completely incoherent 2D DCT spectrum, and thus leads to abnormal image translation results after converting the diffusion features back to the spatial domain.

The image translation results of different designs of frequency band substitution (FBS) are displayed in Fig. 12. It shows that Once Substitution produces severely noisy results rather than reasonable images, which indicates that step-bystep FBS along the whole calibration phase is of crucial importance for smooth and stable information fusion. Since image content is basically formed in the early stage of the diffusion sampling process, removing per-step feature calibration of FBS in the early sampling process will inevitably lead to large deviation of the sampling trajectory against the reconstruction trajectory. In this case, substituting a frequency band at an

Besides, it also shows that Full Substitution fails to manipulate the reference image as per the text prompt. This is because substituting the full DCT spectrum is equivalent to complete feature replacement, which makes the sampling trajectory totally the same as the reconstruction trajectory during the calibration phase, the early part of the diffusion sampling process that dominates the forming of image content. Therefore, the generated image content is forced to be the same as the reference image after the calibration phase and is difficult to be modified noticeably during the subsequent non-calibration phase, the latter part of the diffusion sampling process that focuses on refining fine-grained image details rather than coarse-grained image content. Thus, the sampling

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7a19e98a-bca2-41e1-ae2a-81180d991a95.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-958090b9-7e7c-4fac-b656-f9cb4f478179.jpg)

W/ low-FBS

W/O FBS

Reference

ňCglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;NCF[glyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

CUVTQPCWVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e5760f19-34b0-4989-a161-cdb6c8885c92.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-dd7e0a41-4478-48cc-804b-a920947ff3b3.jpg)

ňCglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;NKVVNGglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;DQ[glyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

CUVTQPCWVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-50b28e59-8834-481d-9454-c24b4ac0aed4.jpg)

ňRCKPVKPIglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;QHglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt; Cglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;JQWUGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a7675b19-33f5-4ace-956a-c791f1a79aec.jpg)

ňVTCPUHQTOGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9e593fd1-c82d-4c94-8b29-9e883e1b4a03.jpg)

ňCglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;URCEGglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt; UVCVKQPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-52b19cd9-f84e-444d-99af-a94feae03a32.jpg)

ň.')1glyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

DTKEMŉ

small semantic gap

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-370d1596-846d-430a-af4a-12e25ea4c7fd.jpg)

ňRCKPVKPIglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;QHglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

Cglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;EKV[glyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;RCTMŉ

ňRCKPVKPIglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;QHglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

Cglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;NKXKPIglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;TQQOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-22d4e011-2b81-40c9-a8c9-15185da99b18.jpg)

large semantic gap

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d13d3de9-f101-4233-8dab-62308547f33b.jpg)

Fig. 9. Our method well adapts to varying degrees of semantic gap between the reference image and the target text prompt.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-984c6487-0cf8-4d97-89bc-b60c4a26e1e8.jpg)

Fig. 10. Demonstration of our method in controlling the appearance and layout guiding intensity of the reference image by varying the th lp in low-FBS.

results of Full Substitution closely resemble the reference images, lacking editability and text fidelity.

## C. Quantitative Evaluations

Similarity ( ↑ ) metric is used to measure semantic consistency between the target text prompt and the translated image, i.e., text fidelity of the I2I translation results. Finally, we evaluate Aesthetic Score ( ↑ ) of the translated images via the pre-trained LAION Aesthetics Predictor V2 model.

For quantitative method evaluation, we separately evaluate methods on the text-driven I2I translation task pursuing image appearance consistency and the task pursuing image appearance divergence. For the former task, we assess models' appearance and layout preservation ability by measuring Structure Similarity ( ↑ ), Perceptual Similarity ( ↑ ), and Style Distance ( ↓ ) between the reference image and the translated image pair. For the latter task, we assess models' contour preservation and appearance alteration capabilities by measuring Structure Similarity ( ↑ ) and Style Distance ( ↑ ) between I2I translation pairs. For Structure Similarity measurement, we use DINO-ViT self-similarity distance [7] as the metric for Structure Distance between two images, and define Structure Similarity as 1 - Structure Distance. We use LPIPS [47] metric to measure Perceptual Similarity, and use AdaIN style loss [48] to measure Style Distance between I2I pairs. Besides, CLIP

We sample reference images from the LAION Aesthetics 6.5+ dataset for quantitative evaluation. For the abovementioned two tasks, we separately sample 500 reference images for each task and manually design 2 editing text prompts for each reference image, resulting in 1000 evaluation samples (reference image and target text pairs) for each task. For evaluation of our method, we use low-FBS for the task pursuing appearance consistency and use high-FBS for the task pursuing appearance divergence. The average values of all the evaluation metrics are reported in Tab. I. Our method achieves top rankings for all the metrics in both two tasks, indicating superiority of our method in layout and appearance preservation with low-FBS, as well as simultaneous contour preservation and appearance modification with high-FBS. Moreover, the competitive results in CLIP Similarity and Aesthetic Score

ňCPglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;KTQPglyph&lt;c=2,font=/DSEEAM+HanziPenSC-W3&gt;

TQDQVŉ

TABLE I

QUANTITATIVE EVALUATIONS OF TEXT-DRIVEN I2I TRANSLATION METHODS.

| Emphasis            | Pursuing image appearance consistency   | Pursuing image appearance consistency   | Pursuing image appearance consistency   | Pursuing image appearance consistency   | Pursuing image appearance consistency   | Pursuing image appearance divergence   | Pursuing image appearance divergence   | Pursuing image appearance divergence   | Pursuing image appearance divergence   |
|---------------------|-----------------------------------------|-----------------------------------------|-----------------------------------------|-----------------------------------------|-----------------------------------------|----------------------------------------|----------------------------------------|----------------------------------------|----------------------------------------|
| Methods Metrics     | Structure Similarity( ↑ )               | LPIPS( ↓ )                              | AdaIN Style Loss( ↓ )                   | CLIP Similarity( ↑ )                    | Aesthetic Score( ↑ )                    | Structure Similarity( ↑ )              | AdaIN Style Loss( ↑ )                  | CLIP Similarity( ↑ )                   | Aesthetic Score( ↑ )                   |
| PAP [21]            | 0.954                                   | 0.272                                   | 20.440                                  | 0.287                                   | 6.590                                   | 0.956                                  | 28.337                                 | 0.279                                  | 6.458                                  |
| Null-text [16]      | 0.948                                   | 0.247                                   | 17.546                                  | 0.276                                   | 6.505                                   | 0.952                                  | 22.545                                 | 0.270                                  | 6.402                                  |
| Pix2Pix-zero [19]   | 0.951                                   | 0.243                                   | 16.875                                  | 0.262                                   | 6.484                                   | 0.953                                  | 21.240                                 | 0.258                                  | 6.344                                  |
| InsPix2Pix [9]      | 0.958                                   | 0.266                                   | 23.373                                  | 0.258                                   | 6.269                                   | 0.965                                  | 30.804                                 | 0.264                                  | 6.196                                  |
| PT-inversion [17]   | 0.947                                   | 0.248                                   | 21.667                                  | 0.271                                   | 6.481                                   | 0.948                                  | 24.367                                 | 0.267                                  | 6.285                                  |
| StyleDiffusion [18] | 0.944                                   | 0.251                                   | 22.484                                  | 0.267                                   | 6.477                                   | 0.947                                  | 25.166                                 | 0.260                                  | 6.267                                  |
| FBSDiff (ours)      | 0.962                                   | 0.241                                   | 15.452                                  | 0.285                                   | 6.583                                   | 0.964                                  | 33.875                                 | 0.281                                  | 6.463                                  |

The red font indicates the top-ranked value and the blue font indicates the second-ranked value.

TABLE II

COMPARISON OF OUR APPROACH WITH RELATED MODELS IN METHOD PROPERTIES.

Fig. 11. Demonstration of our method in controlling the contour guiding intensity of the reference image by varying the th mp 2 in mid-FBS.

| Methods                                                                                                                                                                                            | Training free             | Fine-tuning free          | Optimization free         | Source-text free          | Attention free            | Backbone invariant        |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|
| Null-text [16] PAP [21] Pix2Pix-zero [19] InsPix2Pix [9] PT-inversion [17] StyleDiffusion [18] VQCLIP [2] DiffuseIT [6] DiffusionCLIP [4] Design Booster [12] SINE [13] Imagic [14] FBSDiff (Ours) | ! ! ! % ! ! ! ! ! % ! ! ! | ! ! ! ! ! ! ! ! % ! % % ! | % ! % ! % % % % ! ! % % ! | % % % ! % % ! ! ! ! % ! ! | % % % ! % % ! ! ! ! ! ! ! | % % % ! % % % % % % ! ! ! |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b074de0f-7e43-47df-9deb-fceabac5bf38.jpg)

Fig. 12. Ablation study w.r.t. different manners of frequency band substitution.

reflect that our method can generate I2I translation results with high text fidelity and visual quality.

We compare our FBSDiff with related text-driven I2I trans- lation methods in method properties, results are summarized in Tab. II. Among the compared approaches, our method is

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-875ffaa4-054f-42bd-be01-c8d141e131b1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cdefaadf-dbbf-4541-814b-adc4fa259dac.jpg)

Fig. 13. Visualization of the quantitative method comparison for I2I translation task pursuing image appearance consistency . Left: comparison in CLIP Similarity ( ↑ ) and Structure Similarity ( ↑ ). Right: comparison in CLIP Similarity ( ↑ ) and (1-LPIPS) ( ↑ ). Our method with low-FBS achieves the most top-right position in both two scatters, indicating the best trade-off achieved by our method (low-FBS) in I2I translation appearance consistency and text fidelity.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-76e248c9-868a-459f-b964-5daa14c81a3d.jpg)

PT-inversion

CLIP Similarity

Fig. 14. Visualization of method comparison for the image translation task pursuing image appearance divergence. Left: comparison in CLIP Similarity ( ↑ ) and Structure Similarity ( ↑ ). Right: comparison in CLIP Similarity ( ↑ ) and AdaIN Style Loss ( ↑ ). Our method with high-FBS achieves the most top-right position in both two scatters, indicating the best trade-off achieved by our method (high-FBS) in I2I translation appearance divergence and text fidelity.

the only one that possesses all the following advantages:

- · Dispense with model training;
- · Dispense with model fine-tuning;
- · Dispense with online-optimization at inference time;
- · Dispense with paired source text of the reference image;
- · Dispense with attention modulation operations inside the denoising network;
- · Invariant to the specific architecture of the backbone diffusion model.

For quantitative evaluation reported in Tab. I, we visualize partial results to highlight the superiority of our method over related approaches. For I2I task pursuing image appearance consistency, we display the scatter plot about Structure Similarity ( ↑ ) and CLIP Similarity ( ↑ ), and the scatter plot about (1-LPIPS) ( ↑ ) and CLIP Similarity ( ↑ ) in Fig. 13. Results show that our method with low-FBS achieves the most topright position in both two scatter plots, indicating the best trade-off achieved by our method (low-FBS) in I2I translation appearance consistency and text fidelity. For I2I task pursuing image appearance divergence, we display the scatter plot about Structure Similarity ( ↑ ) and CLIP Similarity ( ↑ ), and the scatter plot about AdaIN Style Loss ( ↑ ) and CLIP Similarity

32

AdaIN Style Loss

30

( ↑ ) in Fig. 14. Results also show the most top-right position achieved by our method with high-FBS in both two plots, indicating the best trade-off achieved by our method (highFBS) in I2I translation appearance divergence and text fidelity.

## V. CONCLUSION

This paper proposes FBSDiff, a plug-and-play method adapting pre-trained T2I diffusion model to highly controllable text-driven I2I translation. At the heart of our method is decomposing different guiding factors of the reference image in the diffusion feature DCT space, and dynamically transplanting a certain DCT frequency band from diffusion features along the reconstruction trajectory into the corresponding features along the sampling trajectory, which is realized via our proposed frequency band substitution layer. Experiments demonstrate that our method allows flexible control over both guiding factors and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band, respectively. In summary, our FBSDiff provides a novel solution to text-driven I2I translation from a frequency-domain perspective, integrating advantages in versatility, high controllability, high visual quality, and plugand-play efficiency.

## APPENDIX A DIFFUSION MODEL BACKGROUND

The Denoising Diffusion Probabilistic Model (DDPM) is a latent variable model that comprises a forward noising diffusion process and a reverse denoising diffusion process. Starting with a given data distribution x 0 ∼ q x ( 0 ) , the forward diffusion process employs a T-step Markov chain to repeatedly add Gaussian noise to the original data x 0 according to q x ( t | x t -1 ) defined as follows:

$$q ( x _ { t } | x _ { t - 1 } ) \coloneqq \mathcal { N } ( x _ { t } ; \sqrt { \alpha _ { t } } x _ { t - 1 }, ( 1 - \alpha _ { t } ) I ), \quad ( 8 ) \ \text{ each i}$$

where α t ∈ (0 1) , , and α t ≥ α t +1 . Using the notation ¯ α t := ∏ t i =1 α i , we can derive the marginal distribution q x ( t | x 0 ) as follows:

$$q ( x _ { t } | x _ { 0 } ) \coloneqq \mathcal { N } ( x _ { t } ; \sqrt { \bar { \alpha } _ { t } } x _ { 0 }, ( 1 - \bar { \alpha } _ { t } ) I ), \quad \ \ ( 9 )$$

where √ ¯ α T approaches to 0 . With the above forward noising diffusion process, the source data distribution will be transformed into an isotropic Gaussian distribution.

The reverse denoising diffusion process conversely converts the isotropic Gaussian distribution to the data distribution by gradually estimating and sampling from the posterior distribution q x ( t -1 | x t ) . However, q x ( t -1 | x t ) is difficult to estimate while q x ( t -1 | x , x t 0 ) is tractable with some algebraic manipulation:

$$q ( x _ { t - 1 } | x _ { t }, x _ { 0 } ) \coloneqq \mathcal { N } ( x _ { t - 1 } ; \tilde { \mu } _ { t } ( x _ { t }, x _ { 0 } ), \tilde { \beta } _ { t } \mathbf I ), \quad ( 1 0 )$$

$$\tilde { \mu } _ { t } ( x _ { t }, x _ { 0 } ) \coloneqq \frac { \sqrt { \tilde { \alpha } _ { t - 1 } } \beta _ { t } } { 1 - \bar { \alpha } _ { t } } x _ { 0 } + \frac { \sqrt { \alpha } _ { t } ( 1 - \bar { \alpha } _ { t - 1 } ) } { 1 - \bar { \alpha } _ { t } } x _ { t }, \quad ( 1 1 )$$

$$\tilde { \beta } _ { t } \coloneqq \frac { 1 - \bar { \alpha } _ { t - 1 } } { 1 - \bar { \alpha } _ { t } } \beta _ { t },$$

where β t := 1 -α t . Though no x 0 is available at inference time, its approximate value can be estimated according to Eq. 9:

$$y _ { \theta } ( x _ { t } ) \coloneqq \frac { 1 } { \sqrt { \bar { \alpha } _ { t } } } ( x _ { t } - \sqrt { 1 - \bar { \alpha } _ { t } } \epsilon _ { \theta } ( x _ { t } ) ), \quad ( 1 3 )$$

where ϵ θ ( x t ) is the prediction of the Gaussian noise sampled at time step t estimated by the denoising network ϵ θ , y θ ( x t ) is the calculated approximation of x 0 .

For image-to-image translation or text-to-image generation, additional condition (could be an image or a text) is required for noise prediction. In these cases, Eq. 13 can be updated as follows:

$$y _ { \theta } ( x _ { t }, c ) \coloneqq \frac { 1 } { \sqrt { \bar { \alpha } _ { t } } } ( x _ { t } - \sqrt { 1 - \bar { \alpha } _ { t } } \epsilon _ { \theta } ( x _ { t }, c ) ), \quad ( 1 4 )$$

where c denotes the additional condition that is involved in the noise prediction and the reverse denoising process.

## APPENDIX B

## DCT AND IDCT DETAILS

We perform 2D-DCT to project diffusion features z into the 2D DCT space, obtaining its frequency-domain counterpart f (Eq. 15). Conversely, we employ 2D-IDCT to transform diffusion features from the DCT domain back into the spatial domain (Eq. 17). The specific form of 2D-DCT and 2D-IDCT are respectively given by Eq. 16 and Eq. 18, in which f ( n ) and z ( n ) denote the n th channel of f and z respectively; i, j and u, v are two-dimensional coordinate indices of the spatial domain and DCT frequency domain respectively; h and w denote the height and width of the latent diffusion features; m (0) = 1 √ 2 , m γ ( ) = 1 for all γ &gt; 0 . It is worth mentioning that though the 2D-DCT and 2D-IDCT are performed on each individual channel of diffusion features (per-channel transformation), our PyTorch implementation with efficient GPU parallel computing capability enables to transform all channels simultaneously, and thus brings negligible additional time overhead during the sampling process.

$$\mathbf f = D C T ( \mathbf z ),$$

$$f _ { u, v } ^ { ( n ) } = & \frac { 2 } { \sqrt { h w } } m ( u ) m ( v ) \sum _ { i = 0 } ^ { h - 1 } \sum _ { j = 0 } ^ { w - 1 } [ z _ { i, j } ^ { ( n ) } \\ & \cos ( \frac { ( 2 i + 1 ) u \pi } { 2 h } ) \cos ( \frac { ( 2 j + 1 ) v \pi } { 2 w } ) ],$$

$$\mathbf z = I D C T ( \mathbf f ),$$

$$z _ { i, j } ^ { ( n ) } = & \frac { 2 } { \sqrt { h w } } \sum _ { u = 0 } ^ { h - 1 } \sum _ { v = 0 } ^ { w - 1 } [ m ( u ) m ( v ) f _ { u, v } ^ { ( n ) } \\ & \cos ( \frac { ( 2 i + 1 ) u \pi } { 2 h } ) \cos ( \frac { ( 2 j + 1 ) v \pi } { 2 w } ) ].$$

## More I2I results with low-FBS (example 1)

## 4GHGTGPEG

ňNKVVNGglyph&lt;c=2,font=/TYCFUD+HanziPenSC-W3&gt;DQ[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d558da82-1bec-4f22-9ff7-48a2d7e83ce7.jpg)

ňNKVVNGglyph&lt;c=2,font=/TYCFUD+HanziPenSC-W3&gt;IKTNŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-52d77912-b20b-4590-b8c4-a06934a16ea1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-2fe0d076-e86c-4fdd-b9d8-8a5dd2e7aa40.jpg)

ňGNFGTN[glyph&lt;c=2,font=/TYCFUD+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-44ca9c91-7521-4ee3-888d-1f4708ae2902.jpg)

ňTQDQVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-2170bb84-8529-4c9c-8945-2cb5a5d67a0f.jpg)

## More I2I results with low-FBS (example 2)

## 4GHGTGPEG

ňCglyph&lt;c=2,font=/VERYBG+HanziPenSC-W3&gt;DQ[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-adb61a01-cd66-4f9d-bbf0-7ebf1804b524.jpg)

ň[QWPIglyph&lt;c=2,font=/VERYBG+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-693315b8-7d45-4593-b9c2-d43ae2b670c7.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ed662b59-20fc-4d36-97a9-8ed2fd4fffcf.jpg)

ňGNFGTN[glyph&lt;c=2,font=/VERYBG+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cbf2a7e5-af45-48ce-b257-73bc81f89b2b.jpg)

ňUQNFKGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d6a68a0a-275f-45fe-be9d-0076c4eccc37.jpg)

## More I2I results with low-FBS (example 3)

## 4GHGTGPEG

ň[QWPIglyph&lt;c=2,font=/RJYGIH+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-712de12d-8446-4db0-9030-58d2ca726135.jpg)

## ňGNFGTN[glyph&lt;c=2,font=/RJYGIH+HanziPenSC-W3&gt;YQOCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-aaadce25-05c4-4b77-b1ad-24817d9f2e03.jpg)

ňNKVVNGglyph&lt;c=2,font=/RJYGIH+HanziPenSC-W3&gt;DQ[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-340d69fa-2fdb-4499-a386-79aabaab71a6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7f4dcb79-9de0-4975-9385-08e024f67331.jpg)

## ňGNFGTN[glyph&lt;c=2,font=/RJYGIH+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-21f85903-56e2-4453-b310-384ca188e729.jpg)

## More I2I results with low-FBS (example 4)

## 4GHGTGPEG

ňNKVVNGglyph&lt;c=2,font=/ZHEIYC+HanziPenSC-W3&gt;DQ[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-abc0c352-3848-4272-9305-3c3232ca4233.jpg)

ň[QWPIglyph&lt;c=2,font=/ZHEIYC+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e967195e-4681-41fb-bbae-9ee96d8c8713.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f124b96e-8ff4-48bd-8740-418e892b898d.jpg)

ňGNFGTN[glyph&lt;c=2,font=/ZHEIYC+HanziPenSC-W3&gt;NCF[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-edeeb410-0d2a-43a4-9e3f-a6ba3509df1c.jpg)

ňTQDQVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b2fc68bd-4847-44b3-bade-cf1d150802fb.jpg)

## More I2I results with low-FBS (example 5)

## 4GHGTGPEG

ňECUVNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b20460e3-3061-4f2f-aef8-6c4c29cf87ba.jpg)

## ňCPEKGPVglyph&lt;c=2,font=/RKKZFK+HanziPenSC-W3&gt;TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7d8cff58-9d06-4628-9a0f-4abeaf27e4e6.jpg)

## ňOQFGTPglyph&lt;c=2,font=/RKKZFK+HanziPenSC-W3&gt;JQWUGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cd82dc7f-097c-4d29-87fb-335ac99e33a3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7a58750d-6b37-4a2d-8e56-418e1019768d.jpg)

ň%JKPGUGglyph&lt;c=2,font=/RKKZFK+HanziPenSC-W3&gt;CPEKGPVglyph&lt;c=2,font=/RKKZFK+HanziPenSC-W3&gt; DWKNFKPIUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7e6ccf1d-82ad-45cc-bd4a-f0957655debc.jpg)

## More I2I results with low-FBS (example 6)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3d918d11-946f-439c-a0db-ce19a688ac17.jpg)

## ňEKV[glyph&lt;c=2,font=/HVQWTX+HanziPenSC-W3&gt;UVTGGVŉ

## ňTQ[CNglyph&lt;c=2,font=/HVQWTX+HanziPenSC-W3&gt;RCNCEGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-047cdf75-644f-4bf8-b06e-2a865f14e805.jpg)

## ňCPEKGPVglyph&lt;c=2,font=/HVQWTX+HanziPenSC-W3&gt;TWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ffab86d7-6333-4789-acf1-fafc3d3a1c1c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-91de0456-1d42-4167-a4af-491841c2beee.jpg)

ňVTCKPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-2096573d-161e-4b77-9049-93440d4c72b3.jpg)

## More I2I results with low-FBS (example 7)

## 4GHGTGPEG

ňECUVNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6d6c59bc-a7d1-482f-bdb4-fc408726a3cc.jpg)

ňEKV[glyph&lt;c=2,font=/JWLRSE+HanziPenSC-W3&gt;UVTGGVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a382c67d-a347-4933-a8ac-f76becd3995c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4750aee1-4658-4739-ba05-a59181189ea2.jpg)

ňVGORNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-db28ddbc-9ccb-4b95-8a50-b5e57798c192.jpg)

## ňVTCKPglyph&lt;c=2,font=/JWLRSE+HanziPenSC-W3&gt;UVCVKQPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-791f8319-e1c4-4752-a9a1-404e8b86dc1d.jpg)

## More I2I results with low-FBS (example 8)

## 4GHGTGPEG

ňECUVNGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-0a73a28d-dfc4-42b7-927a-bebd7fe93eaf.jpg)

## ňEKV[glyph&lt;c=2,font=/QCEQYF+HanziPenSC-W3&gt;UVTGGVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5b0ab077-3623-4d40-a959-ae0032d06cc2.jpg)

ňHCEVQTKGUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-521aa148-4fc6-465c-b0d9-dbd7e3b1f97a.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-974a95bf-f1e2-40b0-b0ee-f739afefcaee.jpg)

## ňCOWUGOGPVglyph&lt;c=2,font=/QCEQYF+HanziPenSC-W3&gt;RCTMŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-00ec58f2-4796-4b55-83c0-32de6176fd0f.jpg)

## More I2I results with low-FBS (example 9)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-17940e4b-5fcd-49c4-b74a-dd24e5f19e6b.jpg)

ňECTglyph&lt;c=2,font=/OLGPVC+HanziPenSC-W3&gt;TCEGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-c49f703d-adb7-427f-9fc1-3a9df951d6bd.jpg)

ňOQFGTPglyph&lt;c=2,font=/OLGPVC+HanziPenSC-W3&gt;EKV[ŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ec66f925-868b-473c-885c-b8a6d8d31dcd.jpg)

ňVTCKPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ee624358-da9b-41c2-960c-fbc66eeb2a2d.jpg)

## ňOQWPVCKPglyph&lt;c=2,font=/OLGPVC+HanziPenSC-W3&gt;NCPFUECRGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f0bd7c1d-95d8-4987-bccc-67b581088b5d.jpg)

## More I2I results with low-FBS (example 10)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cfff974b-ebb3-4ea6-9b31-015445b7ea38.jpg)

ňRCTMŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-460e2cda-d22d-400b-9e39-90fe77025c5e.jpg)

ňTQ[CNglyph&lt;c=2,font=/GFCUJH+HanziPenSC-W3&gt;RCNCEGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9790d307-41f4-4111-a83a-2521db0a56af.jpg)

ňTWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6eb4bb80-4191-45ca-9e8d-31f7268a4f49.jpg)

- ňCPEKGPVglyph&lt;c=2,font=/GFCUJH+HanziPenSC-W3&gt;4QOGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-c006e231-a715-4905-8bb0-a80e550d5379.jpg)

## More I2I results with high-FBS (example 1)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-92a06628-1c53-420b-bcb6-cc2a0ff39edd.jpg)

- ňUVQPGglyph&lt;c=2,font=/WRXFAL+HanziPenSC-W3&gt;UEWNRVWTGŉ
- ňRGPEKNglyph&lt;c=2,font=/WRXFAL+HanziPenSC-W3&gt;UMGVEJŉ

ňQKNglyph&lt;c=2,font=/WRXFAL+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-368f2799-4174-4347-9a3c-7f82436ce488.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-cea9bbe3-199f-4765-b34a-e67218ddfbc8.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-30be33b5-c871-4e3b-ae87-8646f406589b.jpg)

ň8CPglyph&lt;c=2,font=/WRXFAL+HanziPenSC-W3&gt;)QIJglyph&lt;c=2,font=/WRXFAL+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d434e884-2b74-4499-b85a-1c6e30e4df5c.jpg)

## More I2I results with high-FBS (example 2)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f574d72c-bb25-4565-a0a8-3943947f4123.jpg)

## ňYCVGTglyph&lt;c=2,font=/NYQBUL+HanziPenSC-W3&gt;EQNQTŉ

## ňCPKOGglyph&lt;c=2,font=/NYQBUL+HanziPenSC-W3&gt;HKIWTGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b951a4ef-3aac-4511-a0d6-cf051884de53.jpg)

- ňRGPEKNglyph&lt;c=2,font=/NYQBUL+HanziPenSC-W3&gt;UMGVEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-789516ee-0f76-43e4-a753-f55fda9f8b88.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-b65b9ce0-fed1-45d9-9cef-46407292f9dd.jpg)

ňUEWNRVWTGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1aca4530-4188-44f0-9802-b7168f95bad4.jpg)

## More I2I results with high-FBS (example 3)

## 4GHGTGPEG

ňTQDQVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-40ed7426-e30b-4bfa-a393-489ff52a4821.jpg)

## ňRNCUVGTglyph&lt;c=2,font=/UZWVEB+HanziPenSC-W3&gt;OQFGNŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-da4e522e-af42-4935-9090-22ab23b0738f.jpg)

- ňRGPEKNglyph&lt;c=2,font=/UZWVEB+HanziPenSC-W3&gt;UMGVEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9fd57eec-dd19-4e49-b67d-1e865cb6d72b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f889b8eb-f58f-4b80-9edd-33f13f77b204.jpg)

## ňYQQFGPglyph&lt;c=2,font=/UZWVEB+HanziPenSC-W3&gt;UEWNRVWTGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-28bd5417-434a-4aa0-8d89-8427b652c1fa.jpg)

## More I2I results with high-FBS (example 4)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8e40c608-4cff-405a-8310-fbd8b30614e6.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e1216811-09ef-4057-a272-47a1a1c553d0.jpg)

ňETC[QPglyph&lt;c=2,font=/CRMLWT+HanziPenSC-W3&gt;FTCYKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f80dff1b-8780-40e2-9d1c-e9df0121a34c.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-92906363-1c1a-4e5e-b945-7609a163e9b7.jpg)

ňKPMglyph&lt;c=15,font=/CRMLWT+HanziPenSC-W3&gt;YCUJglyph&lt;c=2,font=/CRMLWT+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6732b6ed-3a22-45f4-afec-d2d23c24c477.jpg)

## More I2I results with high-FBS (example 5)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-9b1ffc88-3b13-42c3-acbf-11c1c4a91044.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8a2667e3-75fd-481f-8dc1-33cb5673d44a.jpg)

ňETC[QPglyph&lt;c=2,font=/BRCNYO+HanziPenSC-W3&gt;FTCYKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a34139d3-c829-4836-9012-609560f6d8fc.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-bdd9ac3b-8388-4be5-afc9-b1aebcfa7ee5.jpg)

ňYCVGTglyph&lt;c=2,font=/BRCNYO+HanziPenSC-W3&gt;EQNQTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-4a18ea8a-029f-45e0-a438-9291eb8c02c9.jpg)

## More I2I results with high-FBS (example 6)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-666ff832-9137-4444-9fa5-591e4e2b30a6.jpg)

## ňYCVGTglyph&lt;c=2,font=/NZAXAI+HanziPenSC-W3&gt;EQNQTŉ

ňKEGDGTIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5d1c9d27-6906-4c09-91bb-1069c2f7ebc8.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a6f4bad3-a523-42da-9ee7-4ee1e514256b.jpg)

## ňEQNQTGFglyph&lt;c=2,font=/NZAXAI+HanziPenSC-W3&gt;RGPEKNglyph&lt;c=2,font=/NZAXAI+HanziPenSC-W3&gt;UMGVEJŉ ňFGUGTVglyph&lt;c=2,font=/NZAXAI+HanziPenSC-W3&gt;NCPFUECRGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a0197e7c-66c4-4c9e-964e-f3fca3dbc0a1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f1dc508b-3d30-4afd-b2e0-60bec0515456.jpg)

## More I2I results with high-FBS (example 7)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5b72097d-d9c5-49d9-8a64-987608b43cdd.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5937ab71-21ef-4ecd-8cd0-ab31fa8fe12c.jpg)

ňQKNglyph&lt;c=2,font=/UCTKCM+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-814ebf3c-35b7-4c32-9734-9234a91cebc3.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-6ed08da0-34ac-4524-8be5-97a32726cf13.jpg)

ň%JKPGUGglyph&lt;c=2,font=/UCTKCM+HanziPenSC-W3&gt;NCPFUECRGglyph&lt;c=2,font=/UCTKCM+HanziPenSC-W3&gt; RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7263979d-1917-49d2-ba9c-01d2c928a28c.jpg)

## More I2I results with high-FBS (example 8)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3e29cc71-075a-4044-bdfb-e99ec5229ea1.jpg)

ňCWVWOPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5084126f-98de-4760-98cc-ff7b6e111730.jpg)

ňQKNglyph&lt;c=2,font=/IHURDD+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d1238209-d3ab-4170-b426-b359711ecd60.jpg)

ňYKPVGTŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3c851bca-0f9b-42dc-af5d-4485eeb3eac2.jpg)

ňETC[QPglyph&lt;c=2,font=/IHURDD+HanziPenSC-W3&gt;FTCYKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d17f5740-65d2-43ff-9dca-1b0466550237.jpg)

## More I2I results with high-FBS (example 9)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-8e89d149-a567-4d6d-a9b1-62707353ffb6.jpg)

## ňYCVGTglyph&lt;c=2,font=/RHOSNE+HanziPenSC-W3&gt;EQNQTŉ

## ňRGPEKNglyph&lt;c=2,font=/RHOSNE+HanziPenSC-W3&gt;UMGVEJŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f88ef7f5-fe11-4ecf-b6c9-d57c35006590.jpg)

ňQKNglyph&lt;c=2,font=/RHOSNE+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ed987704-752f-44e7-b0b8-b542c8b8a179.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-25253f53-ec37-4942-a013-d4c644891221.jpg)

## ňECTVQQPglyph&lt;c=2,font=/RHOSNE+HanziPenSC-W3&gt;CPKOGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3cf6b559-8bc6-4ba8-a33c-a248e778d950.jpg)

## More I2I results with high-FBS (example 10)

## 4GHGTGPEG

ňFGUGTVŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-090fe130-7188-42f0-b50d-958304ff0445.jpg)

ňKEGDGTIUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-80a03a5a-d265-409d-a796-29e5bd285635.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-bbbdfe96-45ef-4750-ad9b-c7a349c537c0.jpg)

ň8CPglyph&lt;c=2,font=/VTTMFD+HanziPenSC-W3&gt;)QIJglyph&lt;c=2,font=/VTTMFD+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-442a59b0-3103-41a0-b61a-2704d15c42e3.jpg)

ň/QPGVglyph&lt;c=2,font=/VTTMFD+HanziPenSC-W3&gt;RCKPVKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-67a45041-eacd-4c5e-a2eb-5107c92d99ac.jpg)

## More I2I results with mid-FBS (example 1)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a014470c-7c71-4c34-b69f-717bd27cee95.jpg)

## ňUEKGPEGglyph&lt;c=2,font=/ODEHUI+HanziPenSC-W3&gt;NCDŉ

## ňDGFglyph&lt;c=2,font=/ODEHUI+HanziPenSC-W3&gt;TQQOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1b73561c-fbbe-4618-bf34-09f1179a32c1.jpg)

## ňUJQRRKPIglyph&lt;c=2,font=/ODEHUI+HanziPenSC-W3&gt;OCNNŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-fb619b63-1c2a-44f4-aa37-4e7a2c8e8407.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d154f283-1bf5-49b0-9a71-1f9af0cdf8f5.jpg)

ňI[Oŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-3226503a-70ae-4f28-bbe4-b392b1396b25.jpg)

## More I2I results with mid-FBS (example 2)

## 4GHGTGPEG

ňVTCKPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-46f915f6-17fe-490d-b1b7-7353dad317c0.jpg)

ňOWUGWOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-1ac83223-140b-4e9e-af20-29f5e5f9fe74.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f9683561-c0b8-4c2b-9d27-7164fce19adf.jpg)

ňCSWCTKWOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-d5d5db47-a782-4cd8-b96e-daa79ad55d99.jpg)

ňUEKGPEGglyph&lt;c=2,font=/UPXMVD+HanziPenSC-W3&gt;NCDŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-5c985fac-42be-4136-846b-0d4843c91b6e.jpg)

## More I2I results with mid-FBS (example 3)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-35f8468f-47e8-475c-a276-fadd7be1b8a8.jpg)

ňTWKPUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-642e82e4-646f-4aa4-80c9-a693dcb849ed.jpg)

## ňTCKNYC[glyph&lt;c=2,font=/GFNJLE+HanziPenSC-W3&gt;VTCEMŉ

ňNCMGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-f8d96451-4959-4bbb-8069-1ff48a2859e4.jpg)

ňUEKGPEGglyph&lt;c=2,font=/GFNJLE+HanziPenSC-W3&gt;NCDŉ

Fig. 37. More text-driven I2I results of our method with mid-FBS for image layout control.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-713b88f5-32fe-426a-99b4-0e6d9bf7f2e1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-ad778efc-05b2-4566-83e5-3fb72d0bbdd2.jpg)

## More I2I results with mid-FBS (example 4)

## 4GHGTGPEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-32717955-37cc-4819-a100-641435586349.jpg)

ňJQURKVCNŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-798ff270-c9e2-4983-89b9-9ad642583d3a.jpg)

ňTQDQVUŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-708225f3-4b20-42e1-9201-c387a7cf767b.jpg)

ň$KIglyph&lt;c=2,font=/FXCEVD+HanziPenSC-W3&gt;$GPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-35dbf124-0970-4822-9ecd-b0e014e80b82.jpg)

## ňTQEMGVglyph&lt;c=2,font=/FXCEVD+HanziPenSC-W3&gt;NCWPEJKPIŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-a2aab5da-8b80-4a41-9ada-754bf27d3ad9.jpg)

## More I2I results with mid-FBS (example 5)

## 4GHGTGPEG

ňECMGŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-0fffa7f5-a740-42e9-9c48-863b9c9c260b.jpg)

ňKEGglyph&lt;c=2,font=/RRDRMO+HanziPenSC-W3&gt;ETGCOŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-506ba2fa-2baa-46b4-a90e-286815f05311.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-e870f590-dd97-4652-9971-06dc5a9ccb47.jpg)

ňUPQYglyph&lt;c=2,font=/RRDRMO+HanziPenSC-W3&gt;OCPŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-36cfe7de-4eeb-41ee-9e99-5f1f095053ab.jpg)

## ň.')1glyph&lt;c=2,font=/RRDRMO+HanziPenSC-W3&gt;DTKEMŉ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00998v2-figure-7e8d447d-15f1-4e0b-b282-1b2ff0110615.jpg)

## REFERENCES

- [1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in Proceedings of the International Conference on Machine Learning . PMLR, 2021, pp. 8748-8763.
- [2] K. Crowson, S. Biderman, D. Kornis, D. Stander, E. Hallahan, L. Castricato, and E. Raff, 'Vqgan-clip: Open domain image generation and editing with natural language guidance,' in Proceedings of the European Conference on Computer Vision . Springer, 2022, pp. 88-105.
- [3] P. Esser, R. Rombach, and B. Ommer, 'Taming transformers for highresolution image synthesis,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 12 87312 883.
- [4] G. Kim, T. Kwon, and J. C. Ye, 'Diffusionclip: Text-guided diffusion models for robust image manipulation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 2426-2435.
- [5] J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Proceedings of the Advances in Neural Information Processing Systems , vol. 33, pp. 6840-6851, 2020.
- [6] G. Kwon and J. C. Ye, 'Diffusion-based image translation using disentangled style and content representation,' in Proceedings of the International Conference on Learning Representations , 2022.
- [7] N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel, 'Splicing vit features for semantic appearance transfer,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 748-10 757.
- [8] H. Chung, B. Sim, D. Ryu, and J. C. Ye, 'Improving diffusion models for inverse problems using manifold constraints,' Proceedings of the Advances in Neural Information Processing Systems , vol. 35, pp. 25 683-25 696, 2022.
- [9] T. Brooks, A. Holynski, and A. A. Efros, 'Instructpix2pix: Learning to follow image editing instructions,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18 392-18 402.
- [10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Proceedings of the Advances in Neural Information Processing Systems , vol. 33, pp. 1877-1901, 2020.
- [11] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'Highresolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684-10 695.
- [12] S. Sun, S. Fang, Q. He, and W. Liu, 'Design booster: A text-guided diffusion model for image translation with spatial layout preservation,' arXiv preprint arXiv:2302.02284 , 2023.
- [13] Z. Zhang, L. Han, A. Ghosh, D. N. Metaxas, and J. Ren, 'Sine: Single image editing with text-to-image diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6027-6037.
- [14] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, 'Imagic: Text-based real image editing with diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6007-6017.
- [15] J. Ho and T. Salimans, 'Classifier-free diffusion guidance,' arXiv preprint arXiv:2207.12598 , 2022.
- [16] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, 'Nulltext inversion for editing real images using guided diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6038-6047.
- [17] W. Dong, S. Xue, X. Duan, and S. Han, 'Prompt tuning inversion for text-driven image editing using diffusion models,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7430-7440.
- [18] S. Li, J. van de Weijer, T. Hu, F. S. Khan, Q. Hou, Y. Wang, and J. Yang, 'Stylediffusion: Prompt-embedding inversion for text-based editing,' arXiv preprint arXiv:2303.15649 , 2023.
- [19] G. Parmar, K. Kumar Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu, 'Zeroshot image-to-image translation,' in ACM SIGGRAPH 2023 Conference Proceedings , 2023, pp. 1-11.
- [20] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, 'Prompt-to-prompt image editing with cross attention control,' Proceedings of the International Conference on Learning Representations , 2023.
- [21] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, 'Plug-and-play diffusion features for text-driven image-to-image translation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 1921-1930.
- [22] P. Dhariwal and A. Nichol, 'Diffusion models beat gans on image synthesis,' Proceedings of the Advances in Neural Information Processing Systems , vol. 34, pp. 8780-8794, 2021.
- [23] J. Song, C. Meng, and S. Ermon, 'Denoising diffusion implicit models,' Proceedings of the International Conference on Learning Representations , 2021.
- [24] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, 'Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,' Proceedings of the Advances in Neural Information Processing Systems , vol. 35, pp. 5775-5787, 2022.
- [25] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, 'Palette: Image-to-image diffusion models,' in ACM SIGGRAPH 2022 conference proceedings , 2022, pp. 1-10.
- [26] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen, 'Glide: Towards photorealistic image generation and editing with text-guided diffusion models,' in Proceedings of the International Conference on Machine Learning . PMLR, 2022, pp. 16 784-16 804.
- [27] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , vol. 1, no. 2, p. 3, 2022.
- [28] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al. , 'Photorealistic text-to-image diffusion models with deep language understanding,' Proceedings of the Advances in Neural Information Processing Systems , vol. 35, pp. 36 479-36 494, 2022.
- [29] L. Zhang, A. Rao, and M. Agrawala, 'Adding conditional control to text-to-image diffusion models,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836-3847.
- [30] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan, 'T2iadapter: Learning adapters to dig out more controllable ability for textto-image diffusion models,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 5, 2024, pp. 4296-4304.
- [31] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. M¨ uller, J. Penna, and R. Rombach, 'Sdxl: Improving latent diffusion models for high-resolution image synthesis,' Proceedings of the International Conference on Learning Representations , 2023.
- [32] W. Peebles and S. Xie, 'Scalable diffusion models with transformers,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4195-4205.
- [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Proceedings of the Advances in neural information processing systems , vol. 30, 2017.
- [34] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, 'Image super-resolution via iterative refinement,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 4, pp. 47134726, 2022.
- [35] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. R. Van Gool, 'Inpainting using denoising diffusion probabilistic models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11 461-11 471.
- [36] Z. Liang, Z. Li, S. Zhou, C. Li, and C. C. Loy, 'Control color: Multimodal diffusion-based interactive image colorization,' arXiv preprint arXiv:2402.10855 , 2024.
- [37] W. Tan, S. Chen, and B. Yan, 'Diffss: Diffusion model for few-shot semantic segmentation,' arXiv preprint arXiv:2307.00773 , 2023.
- [38] S. Luo and W. Hu, 'Diffusion probabilistic models for 3d point cloud generation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 2837-2845.
- [39] S. Yu, K. Sohn, S. Kim, and J. Shin, 'Video probabilistic diffusion models in projected latent space,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18 456-18 466.
- [40] T. Anciukeviˇ cius, Z. Xu, M. Fisher, P. Henderson, H. Bilen, N. J. Mitra, and P. Guerrero, 'Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 12 608-12 618.
- [41] A. Ghosh and R. Chellappa, 'Deep feature extraction in the dct domain,' in Proceedings of the International Conference on Pattern Recognition , 2016, pp. 3536-3541.
- [42] W. Xie, D. Song, C. Xu, C. Xu, H. Zhang, and Y. Wang, 'Learning frequency-aware dynamic network for efficient super-resolution,' in

Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 4308-4317.

- [43] M. Cai, H. Zhang, H. Huang, Q. Geng, Y. Li, and G. Huang, 'Frequency domain image translation: More photo-realistic, better identitypreserving,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 13 930-13 940.
- [44] C. Si, Z. Huang, Y. Jiang, and Z. Liu, 'Freeu: Free lunch in diffusion u-net,' pp. 4733-4743, 2024.
- [45] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, 'Ilvr: Conditioning method for denoising diffusion probabilistic models,' arXiv preprint arXiv:2108.02938 , 2021.
- [46] X. Gao, Z. Xu, J. Zhao, and J. Liu, 'Frequency-controlled diffusion model for versatile text-guided image-to-image translation,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 3, 2024, pp. 1824-1832.
- [47] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The unreasonable effectiveness of deep features as a perceptual metric,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 586-595.
- [48] X. Huang and S. Belongie, 'Arbitrary style transfer in real-time with adaptive instance normalization,' in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 1501-1510.
Xiang Gao Jiaying Liu
FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation
Large-scale text-to-image diffusion models have been a revolutionary
milestone in the evolution of generative AI and multimodal technology, allowing
wonderful image generation with natural-language text prompt. However, the
issue of lacking controllability of such models restricts their practical
applicability for real-life content creation. Thus, attention has been focused
on leveraging a reference image to control text-to-image synthesis, which is
also regarded as manipulating (or editing) a reference image as per a text
prompt, namely, text-driven image-to-image translation. This paper contributes
a novel, concise, and efficient approach that adapts pre-trained large-scale
text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a
plug-and-play manner, realizing high-quality and versatile text-driven I2I
translation without any model training, model fine-tuning, or online
optimization process. To guide T2I generation with a reference image, we
propose to decompose diverse guiding factors with different frequency bands of
diffusion features in the DCT spectral space, and accordingly devise a novel
frequency band substitution layer which realizes dynamic control of the
reference image to the T2I generation result in a plug-and-play manner. We
demonstrate that our method allows flexible control over both guiding factor
and guiding intensity of the reference image simply by tuning the type and
bandwidth of the substituted frequency band, respectively. Extensive
qualitative and quantitative experiments verify superiority of our approach
over related methods in I2I translation visual quality, versatility, and
controllability. The code is publicly available at:
https://github.com/XiangGao1102/FBSDiff.