## Few-shot Defect Image Generation based on Consistency Modeling

Qingfeng Shi 1 2 , , Jing Wei 1 2 , , Fei Shen 1 2 3 , , ( B ) , and Zhengtao Zhang 1 2 3 , ,

1 Institute of Automation, Chinese Academy of Sciences 2 School of Artificial Intelligence, University of Chinese Academy of Sciences 3 CASI Vision Technology CO., LTD., Luoyang, China {shiqingfeng2022, weijing2020, fei.shen, zhengtao.zhang}@ia.ac.cn

Abstract. Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zeroshot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at https://github.com/FFDDdiffusion/DefectDiffu.

Keywords: Diffusion Model · Image Generation · Defect Detection

## 1 Introduction

Detection algorithms based on deep learning are crucial to industrial applications such as manufacturing [28], transportation [18], and power systems [1]. Efficient defect detection methods facilitate the identification of anomalies from normal products [27]. However, the performance of detection algorithms is often positively correlated with the number of annotated samples. With the improvement

Fig. 1: Example of intra-product background and inter-product defect consistency

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-ac2cf450-259c-47ce-959c-00d88a9c345d.jpg)

of production processes, defects become rarer in production lines and occupy only a small portion of the object [34], posing challenges in data collection and annotation. Recently, image synthesis utilizing Generative Adversarial Networks (GANs) [8] and Denoising Diffusion Probabilistic Model (DDPM) [9] has attracted a surge of research [16], leading to impressive progress in generating photo-realistic images [29] [3] [15]. Recent works [35] [19] [20] [21] [5] [11] have demonstrated the feasibility of generating photo-realistic defect images to address the scarcity of defect images. However, existing methods face challenges when dealing with limited defect training samples, such as overfitting, poor diversity, and inaccurate annotations.

These challenges stem from the fact that current methods primarily focus on tailored designs for specific scenarios, neglecting to address two key consistencies in industrial defect images. Firstly, various defects within a single product all originate from normal regions. As depicted in the first row of Fig. 1, normal regions of defect images and normal images are consistent, highlighting intra-product background consistency. Secondly, as shown in the second row of Fig. 1, defects of the same category across various products exhibit morphological similarities, illustrating inter-product defect consistency. Leveraging these consistencies, defect image generation can be divided into two steps: normal feature generation and defect feature transformation. Notably, the generation direction for the normal background should remain consistent within the same product, while the defect generation direction should be uniform across similar defects of different products, with the direction scale reflecting the strength of the defect. Compared with modeling the defect distribution of a single type of product, modeling the two consistencies across multiple products enriches available training data for generation. Therefore, modeling the two consistencies is crucial for generating high-quality and diversified defect images with limited training samples. Recently, the pre-trained vision-language model CLIP [23] has showcased remarkable zero-shot capability, displaying robust generalization. This prowess is largely attributed to that CLIP aligns text and images to effectively model the consistency of target objects. In downstream tasks, CLIP is often

utilized to model consistency information among images. In anomaly detection, WinCLIP [12] and AnomalyCLIP [40] leverage the pre-trained CLIP to model consistency in anomalies, achieving zero-shot anomaly detection. In image generation, text-to-image generation based on diffusion models has shown promising results. Text prompts are encoded by the text encoder of CLIP to provide consistent conditions for generation and guide diffusion models to generate images conforming to the text prompts, enabling diverse generation. DreamBooth [26] binds a unique identifier with the specific generation subject to achieve subjectdriven generation. Custom Diffusion [17] optimizes only a few parameters in the text-to-image conditioning mechanism by four images, enabling the model to learn new concepts and achieve combinations of multiple concepts.

- 1) We propose DefectDiffu to model intra-product background consistency and inter-product defect consistency across multiple products using text-guided disentangled integrated architecture, achieving controllable defect image generation with limited samples and obtaining masks during the generation.

Based on the aforementioned analysis, we propose a novel text-guided diffusion method DefectDiffu, to model the intra-product background consistency and inter-product defect consistency across multiple products, achieving controllable diversified defect image generation with few-shot training samples. Firstly, we introduce a disentangled integrated architecture, comprising three key parts: background, defect, and fusion. Then multiple single-object text conditions are inputted into each part through a text encoder to facilitate the disentanglement and integration of background and defect, achieving the modeling of the two consistencies. Additionally, we extract cross-attention maps from the defect part to derive accurate binary mask annotations. Secondly, we propose the double-free strategy to concretize the two consistency directions. Double-free allows for flexible control over product types and defect strengths by adjusting the consistency perturbation directions. Finally, addressing the issues of poor generation of small defects and inaccurate masks, we introduce the adaptive attention-enhanced loss to boost the attention on defect regions, improving the generation quality of small defects and masks. Furthermore, we explore the ability of DefectDiffu for zero-shot defect generation by transferring defect perturbation directions. The main contributions of this paper are as follows:

- 2) We propose the double-free strategy to concretize background and defect perturbation directions, allowing for adjustment of perturbation scales to control product types and defect strength, and further exploring zero-shot defect generation capabilities.
- 3) We introduce the adaptive attention-enhanced loss to improve the generation quality of small defects and the accuracy of masks.

## 2 Related Work

## 2.1 Controllable Image Generation

Diffusion-based generation. Diffusion-based generation. In recent years, the diffusion models have experienced rapid development in image generation due

to their high generation quality and training stability, surpassing GANs and VAEs (Variational Autoencoders) [14] in terms of the realism and diversity of generated images, becoming the most prominent image generation model, particularly in controllable image generation. [4] focuses on pre-training a classifier on noisy images, using the classification gradients of noisy images to guide category-controllable generation. [10] proposes classifier-free guidance, simultaneously training both conditional and unconditional models, using their differences as a substitute for an explicit classifier. DiT [22] replaces the commonly used UNet architecture in diffusion models with transformers to learn parameters of adaptive normalization layers from conditions, achieving state-of-the-art performance on the ImageNet.

Text-guided Image Generation. Recently, large-scale models based on diffusion models, such as DALLE 2 [24], Imagen [37], and Stable diffusion [25], have demonstrated impressive text-guided image generation capabilities, enabling precise control over the generated semantic contents [39] [38]. Cross-attention mechanisms are the most commonly used method for introducing textual guidance into generation models, where cross-attention maps reflect positions corresponding to textual tokens in generated images. To ensure the alignment between generated images and text prompts, Attend-and-Excite [2], MaskDiffusion [41], and StructureDiffusion [6] modify cross-attention to enhance the focus of models on each token. Some works achieve label generation and image editing by processing cross-attention maps. DiffuMask [31] extracts cross-attention maps and obtains pseudo-masks for generated images through affinity networks. Attn2Mask [33] uses post-processed cross-attention maps as pseudo-masks for training segmentation models. Prompt-to-Prompt dynamically changes cross-attention maps during the generation to enable flexible image editing. Text-guided models demonstrate the ability to learn high-level semantic information from images [26] [17] [7], allowing for customization generation by fine-tuning pre-trained models with a few images and textual descriptions.

## 2.2 Defect Image Generation

Defect image generation is widely used to address insufficient data issues in detection tasks. Most current methods employ GANs to generate defect images with binary masks (e.g., DefectGAN [36], SDGAN [19], AnomalyGAN [18]). MDGAN [30] synthesizes defects based on masks constructed from Perlin noise and defect annotations on real normal samples. [20] and [19] encode images into feature space to establish transformation directions between normal and defect images, and adjust transformation factors to achieve strength controllable defect image generation. DFMGAN [5] pre-trains StyleGAN v2 [13] on normal samples and fine-tunes with a few defect samples to update the proposed mask and defect generation modules. Currently, some works have explored few-shot defect image generation based on diffusion models. To improve generation quality, Defect-Gen [32] trains two diffusion models, the large model to determine the

generated target structure and the small model for detail generation. However, these methods require training separate models for each product. Anomalydiffusion [11] generates masks by text flipping, adopting normal samples and masks as inputs to generate defect images for multiple products. However, predefined masks and normal samples limit the diversity of defect shapes and normal backgrounds. Moreover, the aforementioned methods fail to model the relationships between similar defects of multiple products, resulting in poor generalization. In contrast, DefectDiffu models image consistencies across multiple products and obtains masks based on cross-attention maps, thereby integrating features from multiple products to generate diversified defect images and avoid restrictions from predefined masks.

## 3 Method

As illustrated in Fig. 2, DefectDiffu adopts pre-trained DiT as backbones, consisting of the Condition Module, Generation Module, and the Mask Generation Module. The condition module employs the CLIP Text Encoder to introduce consistent text prompts into corresponding parts of the Generation Module. The generation module adopts a disentangled integrated architecture to disentangle the generation of normal backgrounds and defects, and then integrate them to generate realistic images under the guidance of text prompts. The mask generation module integrates cross-attention maps of the Defect Blocks to obtain mask annotations corresponding to generated images. Furthermore, the Adaptive Loss Ratio is designed to adjust the loss weights of the defect regions, enhancing the attention of DefectDiffu to defects. Next, we propose a double-free strategy to concretize the modeled consistencies, thereby achieving controllable generation of the product type and defect strength by modulating denoising results.

## 3.1 Preliminaries

Image generation based on DDPM consists of the forward noise addition process and the reverse denoising process. In the forward process, Gaussian noise is added into the input image x 0 for T times to obtain the pure noisy image x T ,

$$q ( x _ { 1 \colon T } | x _ { 0 } ) \coloneqq \prod _ { t = 1 } ^ { T } q ( x _ { t } | x _ { t - 1 } ), q ( x _ { t } | x _ { t - 1 } ) \coloneqq N ( x _ { t } ; \sqrt { 1 - \beta _ { t } } x _ { t - 1 }, \beta _ { t } I ) \quad ( 1 )$$

where β t is the variance at timestep t . In the reverse denoising process, the noise prediction model ϵ θ recovers x 0 from the noisy image x t , where θ represents the learnable model parameters. ϵ θ takes x t and the current time step t as inputs to predict the removed noise at this time step. Furthermore, when implementing controllable image generation with text, images, or other information as condition c , ϵ θ takes x t , the time step t , and the condition c as inputs to predict the noise. The training objective of the model is:

$$\min _ { \theta } L = \mathbb { E } _ { x _ { t }, \epsilon } \left [ \| \epsilon - \epsilon _ { \theta } ( x _ { t }, t, c ) \| _ { 2 } ^ { 2 } \right ]$$

Fig. 2: Overall framework of DefectDiffu. During training, real images and masks are encoded by Image Encoder into feature spaces, which are then used for training the Generation Module and the Mask Generation Module . The real masks are resized for calculating the Adaptive Loss Ratio . During testing, generated image features and mask features are decoded by Image Decoder to obtain image-mask pairs.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-83cab17c-b00f-4a7b-8ab3-8b1165209c71.jpg)

## 3.2 Text-guided Disentangled Integrated Architecture

To model the intra-product background consistency and inter-product defect consistency, it is necessary to provide conditions that include consistency information for DefectDiffu and design architecture to achieve disentangled generation guided by consistency information. Firstly, as shown in Fig. 2, we adopt product names and defect names as intra-product consistency and inter-product defect consistency text prompts respectively, employing the CLIP text encoder to construct consistent guidance. Previous text-to-image generation methods used A photo of {object 1} {object 2} {object n} to control the generation of multiple objects, which is not suitable for our work. The reason is that the text encoder encodes multiple objects in one text prompt into a fused output. Although this fused output can guide the model to generate high-quality images, it is difficult to disentangle multiple objects in the generation and may cause catastrophic neglect [2]. However, disentangling defects and backgrounds is the key point in modeling two consistencies of multiple products. Therefore, we use two single-object prompts as conditions to model consistencies, achieving the disentangled generation of backgrounds and defects. In addition, compared to the UNet-based diffusion architecture, DiT has a scalable hierarchical stacking architecture based on transformers. It allows us to introduce different conditions into each part of the model to learn the disentanglement generation of multiple objects.

Therefore, as shown in Fig. 2, combining the advantages of single-object guidance and the flexibility of DiT, we propose the Text-guided Disentangled Integrated Architecture (TDIA). TDIA divides DiT into three parts, background, defect, and fusion. Background text embedding for modeling background consistency is introduced into the background part, defect text embedding for modeling

defect consistency is introduced into the defect part, and the fusion part adopts a defect-background fusion text embedding to generate realistic defect images. Designed text prompt templates for each part are as follows:

Consistency condition c p for Background Block:

## A photo of {product name}

Consistency condition c d for Defect Block:

## A photo of {defect name}

Fusion condition c f for Fusion Block:

## A photo of {defect name} {product name}

## 3.3 Double-free

Although DefectDiffu can generate diversified defect image ϵ θ ( x , t, c t d , c p ) conditioned on c p and c d after modeling two consistency by TDIA, the strength of generated defects can not be controlled and is limited to the training set distribution, which affects the diversity of generation. We hypothesize that there should be a perturbation direction along which normal images gradually become defect images, and the perturbation scale reflects the defect strength. Therefore, we perceive controllable defect image generation as a two-stage perturbation process involving both background and defect, with the defect strength adjustable by varying the perturbation scale. We set F t p and F t d as perturbation directions concerning two consistent conditions c p and c d at time step t . Then controllable generation can be achieved by adjusting the perturbation scale w d and w p to update the denoised results

$$\epsilon _ { \theta } ( x _ { t }, t, c _ { d }, c _ { p } ) = \epsilon _ { \theta } ( x _ { t }, t, \varnothing, \varnothing ) + w _ { p } F _ { d } ^ { t } + w _ { d } F _ { p } ^ { t }$$

In order to calculate the two perturbation directions, we propose the doublefree strategy based on the classifier-free guidance, achieving background and defect strength controllable generation beyond the diversity of the training set. First, considering ϵ θ ( x , t, c t g , c p ) as conditional generation of normal background, where c g is good , and unconditional background generation is realized by setting c p to null to get ϵ θ ( x , t, c t g , ∅ ) , and the difference F t p = ϵ θ ( x , t, c t g , c p ) -ϵ θ ( x , t, c t g , ∅ ) is the perturbation direction for intra-product consistency, realizing controllable background generation. On this basis, inter-product defect consistency guidance is regarded as conditional generation of defect ϵ θ ( x , t, c t d , ∅ ) , corresponding unconditional generation is ϵ θ ( x , t, c t g , ∅ ) , and the difference F t d = ϵ θ ( x , t, c t d , ∅ ) -ϵ θ ( x , t, c t g , ∅ ) is perturbation direction for inter-product defect consistency. Then we set the perturbation direction scale w d of defects relative to normal images to achieve the controllable generation of defect strength. Finally, the Eq. (3) can be simplified as:

$$\epsilon _ { \theta } ( x _ { t }, t, c _ { d }, c _ { p } ) & = \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) + w _ { p } ( \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, c _ { p } ) - \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) ) \\ & + w _ { d } ( \epsilon _ { \theta } ( x _ { t }, t, c _ { d }, \varnothing ) - \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) )$$

## Algorithm 1 Double-free guidance.

Input: Training data ( x, c r d , c r p ) from the dataset, p 1 : probability of unconditional prod- uct, p 2 : probability of unconditional defect, c d = c r d , c p = c r p

Output: Condition modified data, ( x, c d , c p ) ;

1: with probability p 1 , c p → ∅

2: If c r d is not good :

3: with probability p 2 , c d → good ;

4: with probability p 1 , c p → ∅ ;

5: return ( x, c d , c p ) ;

where w p and w d control the representation of the background and the strength of the defect respectively. The algorithm of double-free is shown in Algorithm 1.

## 3.4 Mask Generation

When training segmentation models with generated images, obtaining pixellevel mask annotations manually is laborious and time-consuming. Therefore, as shown in Fig. 2, we employ cross-attention mechanisms into the defect blocks and combine them with the mask generation module to simultaneously generate corresponding masks while generating defect images. Cross-attention is applicable for fusing the embeddings of the visual and textual features and producing spatial attention maps for each textual token. The noisy feature φ x ( t ) ∈ R H × W C × is projected to query vector Q ∈ R H × W d × , where H W , , and C is the length, width, number of channels of the tensor respectively, and d is the dimension of Q . The text embedding p ∈ R l × C with length l is projected to get key vector K ∈ R l × d and value vector V ∈ R l × d . Then cross-attention is calculated as,

$$C r o s s A t t e n t i o n = S o f t m a x ( \frac { Q K ^ { T } } { \sqrt { d } } )$$

Next, we concatenate the cross-attention maps of multiple defect blocks along the channel dimension, followed by the mask generation module to obtain mask features M f , and then decode M f to obtain pixel-level masks M pre with values between [0, 255] to locate generated defect positions. The real mask is encoded to feather Mask f and the mask prediction loss is:

$$l o s s _ { m } = \| M _ { f } - M a s k _ { f } \| _ { 2 } ^ { 2 }$$

With increasing time steps, the high-level semantic structure of the image will gradually emerge. Therefore, for accurate defect localization, we compute the average mask M pre = 1 100 Σ 99 t =0 M t pre obtained from the last 100 time steps during testing and use an iterative method to binarize M pre to obtain the final mask.

## 3.5 Adaptive Attention-enhanced Loss

During training, we found that DefectDiffu tends to overlook defect features and generate normal images easily. Additionally, the generated masks are inaccurate

for cases involving small defects or multiple defects within a single image. The reason lies in the fact that compared to normal regions, the features of defect regions with small proportions are more difficult to capture. Moreover, as shown in Eq. (2), the training objective of the diffusion model is to minimize the global reconstruction loss, where the reconstruction loss of defects contributes less to the overall loss. Consequently, the model has insufficient attention to defect regions, leading to poor reconstruction effects and inaccurate annotations. To address these issues, we propose the adaptive attention-enhanced loss based on crossattention maps to increase the loss weight of defect regions, thereby enhancing the attention of DefectDiffu on defects. Firstly, we calculate the average crossattention Map from extracted cross-attention maps of the defect block:

$$M a p = \frac { 1 } { M } \sum _ { i = 0 } ^ { M } S o f t m a x ( \frac { Q _ { i } K ^ { T } } { \sqrt { d } } )$$

where M is the number of extracted cross-attention maps. The real Mask is scaled to mask with the same size of Map to extract the attention for defect regions and normal regions, and then calculate the sum of each to obtain the adaptive attention ratio:

$$R a t = \frac { \sum _ { i, j } ( 1 - m a s k ) _ { i, j } \cdot M a p _ { i, j } } { \sum _ { i, j } m a s k _ { i, j } \cdot M a p _ { i, j } + \alpha }$$

where i and j are height and width of Map α , is a constant used to prevent division by zero. The lower the attention allocated to the defect regions, the higher the Rat . To prevent the imbalance between defect and background regions caused by excessively high weights on defect regions, we further modify the magnitude of the attention ratio Rat to get R ,

$$R ( R a t ) = \begin{cases} 8, \quad R a t \geq 8 \\ R a t, \, 2 < R a t < 8 \\ 2, \quad R a t \leq 2 \end{cases}$$

then the improved defect region loss is:

$$l o s s _ { d } = R \| m a s k \cdot \epsilon _ { \theta } ( z _ { t }, t, c _ { d }, c _ { p } ) - m a s k \cdot \epsilon \| _ { 2 } ^ { 2 } \quad \quad ( 1 0 )$$

Finally, the training objective of DefectDiffu is:

$$\min _ { \theta } l o s s = \| \epsilon _ { \theta } ( z _ { t }, t, c _ { d }, c _ { p } ) - \epsilon \| _ { 2 } ^ { 2 } + l o s s _ { d } + \lambda l o s s _ { m } \quad \quad ( 1 1 )$$

where λ is hyperparameter.

## 3.6 Zero-shot Generation

Based on the TDIA and double-free strategy, DefectDiffu can transfer defect features across normal backgrounds of different products, enabling zero-shot

Fig. 3: Comparison of the generation results.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-57257509-26d9-4498-b217-216187ef4f5c.jpg)

generation of defects. Only normal samples of the target product T are used in training, during testing, the perturbation direction of defect c s d learned from source products is added into the generation of normal images of T , resulting in generating defect samples of T with defect c s d . Specifically, during testing, the model is simultaneously fed with product prompt c T and defect prompt c s d . Then the perturbation direction of c s d modeled on other products is computed to update the denoised results at time step t ,

$$\epsilon _ { \theta } ( x _ { t }, t, c _ { d } ^ { s }, c _ { T } ) & = \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) + w _ { p } ( \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, c _ { T } ) - \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) ) \\ & + w _ { d } ( \epsilon _ { \theta } ( x _ { t }, t, c _ { d } ^ { s }, \varnothing ) - \epsilon _ { \theta } ( x _ { t }, t, c _ { g }, \varnothing ) )$$

Hence, DefectDiffu can generate defect images of T without requiring corresponding real defect images for training, which is attributed to that we successfully model the perturbation direction of the inter-product defect consistency.

## 4 Experiment

## 4.1 Experiment Settings

Dataset and Evaluation Metrics. Weconduct experiments on the MVTecAD dataset to validate the effectiveness of DefectDiffu. We select half of the images from the MVTecAD-test as the training set for image generation and baseline

Table 1: Evaluation results of quality and diversity for comparison and ablation experiments. Our DefectDiffu achieves optimal results on the MVTec dataset.

| Category   | DefectDiffu FID LPIPS   | DefectDiffu FID LPIPS   | Defect-Gen   | Defect-Gen   | DFMGAN   | DFMGAN   | StyleGAN2   | StyleGAN2   | w/o fusion   | w/o fusion   | w/o AAL   | w/o AAL   | OPT FID   | single LPIPS   |
|------------|-------------------------|-------------------------|--------------|--------------|----------|----------|-------------|-------------|--------------|--------------|-----------|-----------|-----------|----------------|
| Category   |                         |                         | FID          | LPIPS        | FID      | LPIPS    | FID         | LPIPS       | FID          | LPIPS        | FID       | LPIPS     |           |                |
| bottle     | 73.48                   | 0.16                    | 124.86       | 0.10         | 130.69   | 0.13     | 112.79      | 0.08        | 132.20       | 0.08         | 134.76    | 0.08      | 127.12    | 0.09           |
| cable      | 101.69                  | 0.29                    | 158.82       | 0.24         | 111.38   | 0.27     | 144.33      | 0.17        | 112.48       | 0.10         | 108.30    | 0.10      | 185.85    | 0.28           |
| capsule    | 26.85                   | 0.08                    | 70.36        | 0.02         | 50.54    | 0.05     | 25.62       | 0.03        | 57.68        | 0.05         | 60.35     | 0.03      | 70.43     | 0.07           |
| carpet     | 40.31                   | 0.17                    | 51.35        | 0.13         | 45.09    | 0.16     | 40.34       | 0.14        | 44.56        | 0.10         | 43.85     | 0.10      | 77.25     | 0.16           |
| grid       | 62.55                   | 0.35                    | 136.62       | 0.30         | 84.58    | 0.34     | 90.91       | 0.33        | 75.91        | 0.24         | 81.45     | 0.24      | 161.24    | 0.35           |
| hazelnut   | 70.18                   | 0.17                    | 434.55       | 0.13         | 378.17   | 0.20     | 435.11      | 0.20        | 420.76       | 0.15         | 464.62    | 0.13      | 185.90    | 0.17           |
| leather    | 67.19                   | 0.15                    | 470.15       | 0.10         | 95.84    | 0.31     | 369.94      | 0.14        | 96.94        | 0.10         | 88.45     | 0.09      | 435.32    | 0.14           |
| metal_nut  | 59.46                   | 0.29                    | 168.21       | 0.47         | 151.68   | 0.33     | 87.56       | 0.18        | 158.56       | 0.25         | 172.42    | 0.19      | 232.26    | 0.29           |
| pill       | 46.23                   | 0.15                    | 258.37       | 0.42         | 133.07   | 0.14     | 227.78      | 0.02        | 162.59       | 0.06         | 152.54    | 0.07      | 186.61    | 0.14           |
| screw      | 26.11                   | 0.27                    | 217.84       | 0.42         | 71.06    | 0.26     | 32.59       | 0.26        | 70.54        | 0.17         | 71.36     | 0.18      | 120.84    | 0.26           |
| tile       | 120.39                  | 0.36                    | 333.24       | 0.24         | 162.14   | 0.18     | 136.40      | 0.26        | 152.25       | 0.22         | 158.48    | 0.22      | 162.18    | 0.34           |
| toothbrush | 110.59                  | 0.14                    | 196.56       | 0.13         | 224.22   | 0.24     | 48.08       | 0.12        | 220.34       | 0.19         | 212.30    | 0.16      | 418.58    | 0.13           |
| transistor | 69.25                   | 0.24                    | 173.35       | 0.21         | 142.69   | 0.28     | 67.07       | 0.26        | 127.27       | 0.19         | 123.21    | 0.11      | 204.25    | 0.20           |
| wood       | 128.26                  | 0.27                    | 333.03       | 0.19         | 315.69   | 0.23     | 188.20      | 0.34        | 331.29       | 0.19         | 325.66    | 0.16      | 305.98    | 0.23           |
| zipper     | 37.74                   | 0.12                    | 378.24       | 0.48         | 80.97    | 0.07     | 78.18       | 0.16        | 89.36        | 0.05         | 93.80     | 0.05      | 89.48     | 0.11           |
| Average    | 69.35                   | 0.21                    | 233.70       | 0.24         | 145.19   | 0.21     | 138.99      | 0.18        | 149.83       | 0.14         | 152.77    | 0.13      | 197.55    | 0.20           |

in segmentation experiments, with the remaining images used as the test set to evaluate the effectiveness of generated images in defect detection. All images are resized to 512×512. FID (Fréchet Inception Distance) and LPIPS (Learned Perceptual Image Patch Similarity) are calculated for quality and diversity evaluation respectively. A lower FID indicates higher generation quality. A higher LPIPS indicates better diversity. For the downstream application, we calculated mIOU (mean Intersection over Union), F1 score, mAP (mean Average Precision), and AUROC (Area Under the Receiver Operating Characteristic Curve) to evaluate the effectiveness of generated image-mask pairs, higher values indicate better performance.

Implementation. We set the N, M, and K in Fig. 2 to 10, 10, and 8 respectively. The parameters of the corresponding modules are initialized with pre-trained DiT in ImageNet with the resolution of 512 × 512 . We set λ = 0 2 . in Eq. (11), p 1 = 0 2 . , p 2 = 0 2 . , α = 0 001 . in Eq. (8). We use an iterative thresholding method for the binarization of generated masks. See the appendix for more details.

Comparison Methods. We employ StyleGAN2, DFMGAN, and Defect-Gen as comparison methods. StyleGAN2 has shown promising results in natural image generation and is commonly used as a foundational framework in industrial image generation. DFMGAN focuses on few-shot defect generation based on StyleGAN2. Defect-Gen utilizes two diffusion models to generate defects and their corresponding masks. They are the current state-of-the-art methods in industrial image generation with annotated masks. It is worth noting that these models need to be trained separately for each product. See the appendix for implementation details of the comparison methods.

Fig. 4: Generation results under different strengths, with defect strength gradually increasing from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-af7c0f24-1996-4503-af86-e775307dc9b8.jpg)

## 4.2 Comparison of Generation Results

Fig. 3 shows the generated results of different methods, see the appendix for more results. It can be seen that Defect-Gen fails to model realistic high-level semantic features and the generated defects do not match real defect characteristics. DFMGAN generates unrealistic defects and contains numerous artifacts. StyleGAN2 fails to converge on leather and exhibits artifacts in products like cable. In contrast, DefectDiffu achieves good convergence across all 15 products. The generated results conform to real features, without mode collapse or artifacts, and the masks are accurate, validating the effectiveness of the proposed multi-product consistency modeling for few-shot image generation. Additionally, Fig. 4 demonstrates the control of defect strength utilizing w d . As w d increases, the area of generated defects expands, and the number of defect regions increases, indicating a gradual increase in the severity of product damage.

Tab. 1 shows that our method achieves the optimal generation quality and good diversity on the MVTec dataset. Although the generation quality of toothbrush is slightly inferior to StyleGAN2, DefectDiffu simultaneously generates defect masks, which is more beneficial for training segmentation methods. Despite DFMGAN and Defect-Gen achieving higher LPIPS on some products, this is due to the introduction of unrealistic artifacts and the loss of real semantic features. It can be seen in Fig. 3 and Tab. 1 that the generation quality of these products is poor, corresponding to high FID. Clearly, diversity without adher-

Table 2: Comparison on detection performance.

| Category   | mIoU   | Baseline   | Baseline   | Baseline   | DefectDiffu   | DefectDiffu   | DefectDiffu   | DefectDiffu   | Defect-Gen   | Defect-Gen   | Defect-Gen   | Defect-Gen   | DFMGAN   | DFMGAN   | DFMGAN   | DFMGAN   |
|------------|--------|------------|------------|------------|---------------|---------------|---------------|---------------|--------------|--------------|--------------|--------------|----------|----------|----------|----------|
| Category   | mIoU   | F1         | mAP        | AUC        | mIoU          | F1            | mAP           | AUC           | mIoU         | F1           | mAP          | AUC          | mIoU     | F1       | mAP      | AUC      |
| bottle     | 66.1   | 79.6       | 89.7       | 98.7       | 75.7          | 86.2          | 93.8          | 99.2          | 76.0         | 86.3         | 93.2         | 99.0         | 71.3     | 83.2     | 92.3     | 98.8     |
| cable      | 55.0   | 71.0       | 76.0       | 92.2       | 62.5          | 76.9          | 81.8          | 95.1          | 60.5         | 75.4         | 80.6         | 95.1         | 58.4     | 73.7     | 78.0     | 91.8     |
| capsule    | 28.2   | 44.0       | 42.2       | 94.1       | 34.2          | 51.0          | 54.9          | 97.2          | 31.4         | 47.8         | 47.5         | 91.2         | 33.1     | 49.8     | 56.4     | 96.8     |
| carpet     | 64.8   | 78.6       | 88.8       | 99.6       | 69.8          | 82.2          | 90.9          | 99.6          | 67.4         | 80.5         | 89.8         | 99.3         | 66.1     | 79.6     | 88.8     | 99.5     |
| grid       | 32.7   | 49.3       | 46.6       | 95.8       | 47.3          | 64.2          | 69.4          | 99.3          | 35.6         | 52.5         | 49.6         | 97.4         | 37.4     | 54.4     | 63.8     | 98.8     |
| hazelnut   | 85.0   | 91.9       | 96.9       | 99.6       | 87.2          | 93.2          | 97.5          | 99.7          | 87.4         | 93.3         | 97.9         | 99.8         | 83.1     | 90.8     | 96.4     | 99.5     |
| leather    | 67.4   | 80.5       | 89.6       | 99.8       | 67.9          | 80.9          | 88.1          | 99.7          | 68.3         | 81.1         | 89.9         | 99.8         | 65.3     | 79.0     | 88.4     | 99.8     |
| metal_nut  | 86.9   | 93.0       | 97.6       | 99.3       | 91.5          | 95.6          | 99.0          | 99.7          | 89.5         | 94.5         | 98.8         | 99.8         | 73.0     | 84.4     | 92.4     | 97.9     |
| pill       | 76.1   | 86.4       | 95.2       | 99.7       | 83.4          | 90.9          | 97.4          | 99.8          | 69.5         | 82.0         | 91.8         | 99.0         | 76.8     | 86.9     | 94.3     | 99.3     |
| screw      | 26.1   | 41.4       | 47.0       | 98.3       | 48.9          | 65.7          | 76.6          | 99.7          | 43.0         | 60.1         | 69.9         | 98.5         | 32.7     | 49.3     | 57.9     | 98.1     |
| tile       | 86.7   | 92.9       | 98.2       | 99.8       | 89.7          | 94.6          | 98.8          | 99.9          | 89.1         | 94.2         | 98.9         | 99.9         | 87.7     | 93.4     | 98.6     | 99.8     |
| toothbrush | 34.5   | 51.3       | 56.6       | 95.8       | 44.0          | 61.1          | 60.2          | 96.2          | 44.8         | 61.8         | 60.7         | 97.0         | 34.2     | 31.4     | 47.8     | 58.3     |
| transistor | 58.2   | 73.6       | 78.9       | 96.0       | 64.3          | 78.3          | 79.2          | 97.1          | 52.8         | 69.1         | 84.6         | 96.7         | 48.7     | 65.5     | 74.7     | 94.4     |
| wood       | 68.3   | 81.1       | 89.5       | 97.7       | 69.4          | 81.9          | 89.8          | 98.3          | 69.3         | 81.9         | 89.5         | 98.0         | 66.5     | 79.9     | 88.1     | 97.8     |
| zipper     | 67.9   | 80.8       | 87.7       | 99.3       | 70.7          | 82.9          | 90.2          | 99.6          | 68.6         | 81.4         | 88.9         | 99.4         | 70.8     | 82.9     | 89.2     | 99.6     |
| Average    | 60.3   | 73.0       | 78.7       | 97.7       | 67.1          | 79.0          | 84.5          | 98.7          | 63.5         | 76.1         | 82.1         | 98.0         | 60.3     | 72.3     | 80.5     | 95.4     |

Fig. 5: Zero-shot generation results. A-B means to transfer source defect A to target product B.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-58d47087-a979-41d2-832f-bb121072f0ff.jpg)

ence to real features is meaningless. In contrast, DefectDiffu achieves optimal generation quality on most products while maintaining good diversity.

## 4.3 Comparison of Detection Results.

To validate the effectiveness of the generated results in industrial detection, we employ the real training set to train a UNet as a baseline and generate synthetic samples twice the size of real images by those methods capable of generating masks to augment the real dataset. Segmentation results are shown in Tab. 2. Compared to other methods, DefectDiffu can improve the performance of all products and achieve optimal segmentation results on most products. More results are provided in the appendix. In summary, DefectDiffu can efficiently improve the performance of detection methods and have high application value.

## 4.4 Zero-shot Generation

As outlined in Sec. 3.6, we validate the zero-shot generation capability of DefectDiffu, with results shown in Fig. 5. Although DefectDiffu only learns the features of normal images from the target product during training, the defect consistency perturbation directions, modeled from other products, effectively guide the generation of defects for the target product. This enables DefectDiffu to achieve

zero-shot defect generation without the need for real defect images of the target product during training. See the appendix for more results.

## 4.5 Ablation Experiments

Fusion part . We remove the fusion part in the generation module, and the results are shown in Tab. 1 w/o fusion. Without the fusion part, the semantic coherence between generated defects and backgrounds is disrupted, leading to a reduction in generation quality and diversity.

AAL (Adaptive Attention-enhanced Loss) . We set R to 3 in Eq. (9) to train DefectDiffu, with results shown in Tab. 1 w/o AAL. It indicates that without AAL, both the quality and diversity of the generated results decrease, indicating that AAL contributes to preventing the loss of defect features and enhancing the quality of generated defect images.

Multiple single-object prompts . To validate the role of multiple singleobject prompts in TDIA for improving the generated quality of defect images, we only use the text prompt from the fusion part as the condition for DefectDiffu, and the quality of generated results are shown in Tab. 1 OTP. It can be seen that employing a single text prompt to control the generation of multiple objects adversely affects generation quality.

Defect strength scale w d . We set w d =1 and generate the same number of defect images at multiple strength scales to validate the effect of multi-strength generation on diversity. The LPIPS results are shown in Tab. 1 single. It can be observed that under a single scale, the diversity is reduced, but it still outperforms the comparison methods in most products. This indicates that DefectDiffu effectively extends the diversity of the dataset through the double-free strategy.

## 5 Conclusion

This paper proposes DefectDiffu to achieve controllable generation of defect image-mask pairs across multiple products with limited samples. DefectDiffu leverages a disentangled integrated architecture and multiple single-object text prompts to disentangle defects and normal backgrounds, which facilitates the modeling of intra-product background consistency and inter-product defect consistency, thereby enhancing the diversity of normal backgrounds and defects. Furthermore, the double-free strategy is proposed to concretize the perturbation directions of the modeled background consistency and defect consistency, enabling multi-attribute controllable generation. Experimental results demonstrate that DefectDiffu outperforms state-of-the-art methods in both generation quality and diversity, effectively enhancing detection performance. Moreover, DefectDiffu achieves zero-shot defect generation based on inter-product defect consistency, reducing the costs associated with data collection and annotation.

## Acknowledgements

This work was supported by the National Key Research and Development Program of China (2022YFB3303800), the National Natural Science Foundation of China (U21A20482), and Beijing Municipal Natural Science Foundation, China, under Grant L243018.

## References

- 1. Ahmed, C.M., M R, G.R., Mathur, A.P.: Challenges in machine learning based approaches for real-time anomaly detection in industrial control systems. In: Proceedings of the 6th ACM on Cyber-Physical System Security Workshop (Oct 2020). https://doi.org/10.1145/3384941.3409588 , http://dx.doi.org/10. 1145/3384941.3409588
- 2. Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models (Jan 2023)
- 4. Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 , 8780-8794 (2021)
- 3. Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., Raff, E.: Vqgan-clip: Open domain image generation and editing with natural language guidance. In: European Conference on Computer Vision. pp. 88-105. Springer (2022)
- 5. Duan, Y., Hong, Y., Niu, L., Zhang, L.: Few-shot defect image generation via defect-aware feature manipulation (Mar 2023)
- 7. Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022)
- 6. Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032 (2022)
- 8. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. p. 2672-2680. NIPS'14, MIT Press, Cambridge, MA, USA (2014)
- formation Processing Systems,Neural Information Processing Systems (Jan 2020) 10. Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
- 9. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Neural In-
- 11. Hu, T., Zhang, J., Yi, R., Du, Y., Chen, X., Liu, L., Wang, Y., Wang, C.: Anomalydiffusion: Few-shot anomaly image generation with diffusion model (Dec 2023)
- 13. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8107-8116 (2020). https: //doi.org/10.1109/CVPR42600.2020.00813
- 12. Jeong, J., Zou, Y., Kim, T., Zhang, D., Ravichandran, A., Dabeer, O.: Winclip: Zero-/few-shot anomaly classification and segmentation (Mar 2023)
- 14. Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 1416, 2014, Conference Track Proceedings (2014)

- 15. Kowalski, M., Garbin, S.J., Estellers, V., Baltrušaitis, T., Johnson, M., Shotton, J.: Config: Controllable neural face image generation. In: European Conference on Computer Vision (ECCV) (2020)
- 16. Kulikov, V., Yadin, S., Kleiner, M., Michaeli, T.: Sinddm: A single image denoising diffusion model. In: International Conference on Machine Learning. pp. 1792017930. PMLR (2023)
- 17. Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1931-1941 (2023)
- 18. Liu, R., Liu, W., Zheng, Z., Wang, L., Mao, L., Qiu, Q., Ling, G.: Anomaly-gan: A data augmentation method for train surface anomaly detection. Expert Systems with Applications p. 120284 (Oct 2023). https://doi.org/10.1016/j.eswa.2023. 120284 , http://dx.doi.org/10.1016/j.eswa.2023.120284
- 19. Niu, S., Li, B., Wang, X., Lin, H.: Defect image sample generation with gan for improving defect recognition. IEEE Transactions on Automation Science and Engineering p. 1-12 (Jan 2020). https://doi.org/10.1109/tase.2020.2967415 , http://dx.doi.org/10.1109/tase.2020.2967415
- 20. Niu, S., Li, B., Wang, X., Peng, Y.: Region- and strength-controllable gan for defect generation and segmentation in industrial images. IEEE Transactions on Industrial Informatics p. 4531-4541 (Jul 2022). https://doi.org/10.1109/tii. 2021.3127188 , http://dx.doi.org/10.1109/tii.2021.3127188
- 21. Niu, S., Peng, Y., Li, B., Wang, X.: A transformed-feature-space data augmentation method for defect segmentation
- 22. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4195-4205 (2023)
- 23. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
- 24. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical textconditional image generation with clip latents
- 25. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2022). https://doi. org/10.1109/cvpr52688.2022.01042 , http://dx.doi.org/10.1109/cvpr52688. 2022.01042
- 26. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation (Aug 2022)
- 27. Schlüter, H.M., Tan, J., Hou, B., Kainz, B.: Natural synthetic anomalies for selfsupervised anomaly detection and localization. In: European Conference on Computer Vision. pp. 474-489. Springer (2022)
- 28. Singh, S.A., Desai, K.A.: Automated surface defect detection framework using machine vision and convolutional neural networks. Journal of Intelligent Manufacturing p. 1995-2011 (Apr 2023). https://doi.org/10.1007/s10845-021-01878-w , http://dx.doi.org/10.1007/s10845-021-01878-w
- 29. Tang, H., Bai, S., Zhang, L., Torr, P.H., Sebe, N.: Xinggan for person image generation. In: ECCV (2020)

- 30. Wei, J., Zhang, Z., Shen, F., Lv, C.: Mask-guided generation method for industrial defect images with non-uniform structures. Machines 10 (12) (2022). https:// doi.org/10.3390/machines10121239 , https://www.mdpi.com/2075-1702/10/ 12/1239
- 31. Wu, W., Zhao, Y., Shou, M., Zhou, H., Shen, C.: Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models (Mar 2023)
- 33. Yoshihashi, R., Otsuka, Y., Doi, K., Tanaka, T.: Attention as annotation: Generating images and pseudo-masks for weakly supervised semantic segmentation with diffusion (Sep 2023)
- 32. Yang, S., Chen, Z., Chen, P., Fang, X., Liu, S., Chen, Y.: Defect spectrum: A granular look of large-scale defect datasets with rich semantics. arXiv preprint arXiv:2310.17316 (2023)
- 34. Zavrtanik, V., Kristan, M., Skočaj, D.: Dsr -a dual subspace re-projection network for surface anomaly detection
- 36. Zhang, G., Cui, K., Hung, T.Y., Lu, S.: Defect-gan: High-fidelity defect synthesis for automated defect inspection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2524-2534 (2021)
- 35. Zhang, G., Cui, K., Hung, T.Y., Lu, S.: Defect-gan: High-fidelity defect synthesis for automated defect inspection. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (Jan 2021). https://doi.org/10.1109/wacv48630. 2021.00257 , http://dx.doi.org/10.1109/wacv48630.2021.00257
- 37. Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models
- 39. Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y., Li, X.: Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2249022499 (June 2023)
- 38. Zhao, S., Chen, D., Chen, Y.C., Bao, J., Hao, S., Yuan, L., Wong, K.Y.K.: Unicontrolnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems (2023)
- 40. Zhou, Q., Pang, G., Tian, Y., He, S., Chen, J.: Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection (Oct 2023)
- 41. Zhou, Y., Zhou, D., Zhu, Z.L., Wang, Y., Hou, Q., Feng, J.: Maskdiffusion: Boosting text-to-image consistency with conditional mask. arXiv preprint arXiv:2309.04399 (2023)

## Few-shot Defect Image Generation based on Consistency Modeling

Supplementary Material

## 1 Supplementary Experiment

## 1.1 Implementation

To model the inter-product defect consistency direction, we modify the defect names of similar defects in the MVTecAD dataset. The defect names before and after modification are shown in Tab. 2.

## 1.2 Implementation for Comparison Methods

StyleGAN2: We conduct training for 3000 iterations on StyleGAN2 for each defect category, with a batch size of 4. Other parameters remained consistent with the specifications outlined in the official documentation.

DFMGAN: Firstly, StyleGAN2 is trained on defect-free images of each product for 3000 iterations using a batch size of 4. Subsequently, the pre-trained StyleGAN2 is finetuned to defect images of each defect category, employing a batch size of 4. All other parameters adhered to the official documentation.

Defect-Gen: Both models undergo training for 200,000 iterations, with all other settings aligned with the official documentation.

## 1.3 Comparison of Classification

To verify the role of generated defect images in classification tasks, we employ the same dataset as in the segmentation experiments to train the ResNet50 for binary classification. Each product is trained for 2000 epochs with a learning rate of 0.0002 and a batch size of 12. The average classification accuracy on MVTec is shown in Tab. 1, where our DefectDiffu achieves the optimal classification performance.

Table 1: Comparison of Classification Experiments. AA represents Average Accuracy.

| Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   | Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   | Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   | Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   | Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   | Method Baseline DefectDiffu(Ours) DFMGAN Defect-Gen StyleGAN   |
|----------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|
| AA                                                             | 81.13%                                                         | 91.74%                                                         | 88.15%                                                         | 89.75%                                                         | 87.26%                                                         |

Table 2: MVTecAD Dataset Defect Names: defect refers to the original defect name, uni\_defect refers to the modified defect name used in training, and product refers to the product name

| product    | defect                                                                                | uni_defect                                      | product    | defect                                                                         | uni_defect                                                |
|------------|---------------------------------------------------------------------------------------|-------------------------------------------------|------------|--------------------------------------------------------------------------------|-----------------------------------------------------------|
| metal_nut  | scratch flip color bent                                                               | scratch flip color bent                         | hazelnut   | print hole cut crack                                                           | print hole cut crack                                      |
| carpet     | hole metal_contamination cut thread color                                             | hole metal_contamination cut thread color       | wood       | scratch combined hole liquid color                                             | scratch combined hole liquid color                        |
| screw      | scratch_neck thread_top scratch_head manipulated_front thread_side                    | scratch thread scratch manipulated_front thread |            | scratch faulty_imprint squeeze crack                                           | scratch faulty_imprint squeeze crack hole                 |
|            | glue                                                                                  | liquid                                          | capsule    | poke                                                                           |                                                           |
| leather    | cut color poke                                                                        | cut color hole                                  |            | broken glue                                                                    | broken liquid metal_contamination                         |
| pill       | fold scratch                                                                          | fold scratch faulty_imprint                     | grid       | metal_contamination thread bent rough                                          | thread bent rough                                         |
|            | faulty_imprint combined contamination pill_type crack color                           | combined contamination pill_type crack color    | zipper     | fabric_border combined fabric_interior squeezed_teeth broken_teeth split_teeth | cut combined cut squeeze broken split                     |
| cable      | missing_cable missing_wire cable_swap                                                 | missing missing cable_swap                      | tile       | rough gray_stroke glue_strip oil                                               | rough                                                     |
|            | bent_wire poke_insulation combined cut_inner_insulation cut_outer_insulation cut_lead | bent hole combined cut cut cut                  | bottle     | crack contamination broken_small broken_large                                  | color glue_strip liquid crack contamination broken broken |
|            | damaged_case misplaced                                                                | damaged_case misplaced                          |            | defective                                                                      | bent                                                      |
| transistor | bent_lead                                                                             | bent                                            | toothbrush |                                                                                |                                                           |

## 1.4 Experiments on the VISA and KSDD datasets

We tested our method on the KSDD and VISA datasets under the same settings. The experimental results are presented in Table A and Table B.

Table A: Comparison of average quantitative results.

| Dataset   | Index     | Baseline   |   DefectDiffu |   DefectGen |   DFMGAN |
|-----------|-----------|------------|---------------|-------------|----------|
| VisA      | FID       | \          |         39.24 |      185.91 |   180.15 |
| VisA      | LPIPS (%) | \          |         10.84 |       10.57 |     7.15 |
| VisA      | IoU (%)   | 44.44      |         53.34 |       47.88 |    46.78 |
| VisA      | F1 (%)    | 57.75      |         66.65 |       61.72 |    60.49 |
| VisA      | AP (%)    | 63.96      |         71.26 |       68.15 |    67.19 |
| VisA      | AUC (%)   | 98.16      |         98.81 |       98.25 |    98.15 |
|           | FID       | \          |         47.92 |      320.6  |   365.56 |
|           | LPIPS (%) | \          |          2.89 |        4.32 |     1.75 |
| KSDD2     | IoU (%)   | 63.96      |         66.08 |       65.17 |    63.32 |
| KSDD2     | F1 (%)    | 78.02      |         79.58 |       78.91 |    77.54 |
| KSDD2     | AP (%)    | 85.33      |         87.52 |       87.32 |    85.5  |
| KSDD2     | AUC (%)   | 98.66      |         99.53 |       99.45 |    99.36 |

Table B: Comparison of mIoU across each product.

| VisA        |   candle capsules cashew chewinggum fryum macaroni1 |   candle capsules cashew chewinggum fryum macaroni1 |   candle capsules cashew chewinggum fryum macaroni1 |   candle capsules cashew chewinggum fryum macaroni1 |   candle capsules cashew chewinggum fryum macaroni1 | candle capsules cashew chewinggum fryum macaroni1   | candle capsules cashew chewinggum fryum macaroni1   |   macaroni2 pcb1 |   macaroni2 pcb1 |   pcb2 pcb3 |   pcb2 pcb3 | pcb4   | pcb4   |   pipe_fryum |    |    |
|-------------|-----------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|------------------|------------------|-------------|-------------|--------|--------|--------------|----|----|
| Baseline    |                                                8.06 |                                               45.58 |                                               81.32 |                                               54.01 |                                               69.12 | 25.49                                               |                                                     |            15.52 |            66.46 |       27.26 |       26.43 | 39.03  |        |        75.04 |    |    |
| DefectDiffu |                                               29.18 |                                               62.12 |                                               86.89 |                                               61.81 |                                               79.83 | 28.08                                               |                                                     |            24.19 |            70.89 |       32.51 |       27.22 |        | 49.84  |        87.57 |    |    |
| DefectGen   |                                               24.9  |                                               60.64 |                                               82.08 |                                               59.85 |                                               62.85 |                                                     | 24.04                                               |            23.23 |            65.13 |       32.33 |       18.62 |        | 42.10  |        78.83 |    |    |
| DFMGAN      |                                               19.95 |                                               49.54 |                                               79.84 |                                               57.34 |                                               78.88 |                                                     | 27.56                                               |            22.04 |            71.11 |       23.95 |       19.28 |        | 49.64  |        62.21 |    |    |

## 1.5 Additional Zero-shot Generation Results

We validate the effectiveness of zero-shot generation on wood, tile, and leather, and the results are depicted in Fig. 1.

## 1.6 Controllable Generation Results for All Types of Defects Strengths

We perform controllable defect image generation for all defect products in the MVTecAD dataset. The generated results for various defect types are shown in Fig. 2-Fig. 16, demonstrating the strong controllability of our model over defect strengths.

Fig. 1: Zero-shot defect generation results of tile, leather, and wood.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-2d6558ff-a66d-48e5-904e-a559bc6ec7e2.jpg)

Fig. 2: Results of varying defect strengths for different types of defects under the category bottle. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-5e9b4e65-0b0d-43f8-b09a-8e61745be2a9.jpg)

Wd

Fig. 3: Result of varying defect strengths for different types of defects under the category cable. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-3663bd1a-c6d9-4be1-a237-ecbacdb3d1bc.jpg)

Fig. 4: Result of varying defect strengths for different types of defects under the category capsule. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-a10e3550-de84-47dd-9a48-ee8843241d74.jpg)

Fig. 5: Result of varying defect strengths for different types of defects under the category carpet. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-9cd4a9ab-7c07-46e6-a517-578a008d5cba.jpg)

Wd

Fig. 6: Result of varying defect strengths for different types of defects under the category grid. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-9927ba4d-ff9a-4079-8df9-d62c54d22e33.jpg)

Fig. 7: Result of varying defect strengths for different types of defects under the category hazelnut. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-fe4bc40d-1cd1-47a9-98c4-9bc1c6776533.jpg)

Fig. 8: Result of varying defect strengths for different types of defects under the category leather. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-60f691d6-e3f2-4729-b9d7-2b4293e21e80.jpg)

Wd

Fig. 9: Result of varying defect strengths for different types of defects under the category metal\_nut. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-eb12bb28-2dbc-4bcf-b1f2-85bf9fc62f23.jpg)

Fig. 10: Result of varying defect strengths for different types of defects under the category pill. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-9da92782-872b-42c0-9a81-638d716a8017.jpg)

Wd

Fig. 11: Result of varying defect strengths for different types of defects under the category screw. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-1df89809-4207-48e4-a611-5be106040237.jpg)

Wd

Fig. 12: Result of varying defect strengths for different types of defects under the category tile. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-2cc45a43-2c4d-4dcd-a6a3-5ee76ef0f167.jpg)

Wd

Fig. 13: Result of varying defect strengths for different types of defects under the category toothbrush. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-fe8338ca-8ca2-443e-8dd4-4c121f53df3e.jpg)

Fig. 14: Result of varying defect strengths for different types of defects under the category transistor. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-442ef66a-fdb2-40fc-bc82-b5dfb23d4758.jpg)

Wd

Fig. 15: Result of varying defect strengths for different types of defects under the category wood. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-df1a0be7-d9d4-4ada-92e2-314ee0a16d68.jpg)

Wd

Fig. 16: Result of varying defect strengths for different types of defects under the category zipper. MVTecAD represents real data, while w d represents increasing defect strengths from left to right.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00372v1-figure-ea22f249-66f8-4eba-876c-f9ec32a335ed.jpg)
Qingfeng Shi Jing Wei Fei Shen Zhengtao Zhang
Few-shot Defect Image Generation based on Consistency Modeling
Image generation can solve insufficient labeled data issues in defect
detection. Most defect generation methods are only trained on a single product
without considering the consistencies among multiple products, leading to poor
quality and diversity of generated results. To address these issues, we propose
DefectDiffu, a novel text-guided diffusion method to model both intra-product
background consistency and inter-product defect consistency across multiple
products and modulate the consistency perturbation directions to control
product type and defect strength, achieving diversified defect image
generation. Firstly, we leverage a text encoder to separately provide
consistency prompts for background, defect, and fusion parts of the
disentangled integrated architecture, thereby disentangling defects and normal
backgrounds. Secondly, we propose the double-free strategy to generate defect
images through two-stage perturbation of consistency direction, thereby
controlling product type and defect strength by adjusting the perturbation
scale. Besides, DefectDiffu can generate defect mask annotations utilizing
cross-attention maps from the defect part. Finally, to improve the generation
quality of small defects and masks, we propose the adaptive attention-enhance
loss to increase the attention to defects. Experimental results demonstrate
that DefectDiffu surpasses state-of-the-art methods in terms of generation
quality and diversity, thus effectively improving downstream defection
performance. Moreover, defect perturbation directions can be transferred among
various products to achieve zero-shot defect generation, which is highly
beneficial for addressing insufficient data issues. The code are available at
https://github.com/FFDD-diffusion/DefectDiffu.