## CIResDiff: A Clinically-Informed Residual Diffusion Model for Predicting Idiopathic Pulmonary Fibrosis Progression

Caiwen Jiang 1 2 , , 2 1 4 2

Xiaodan Xing , Zaixin Ou , Mianxin Liu , Simon Walsh , GLYPH&lt;12&gt; GLYPH&lt;12&gt; )

Guang Yang 2 5 7 8( , , , ) , and Dinggang Shen 1 3 6( , ,

- 1 School of Biomedical Engineering &amp; State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, China

{jiangcw,dgshen}@shanghaitech.edu.cn

- 2 Bioengineering Department and Imperial-X, Imperial College London, London, UK g.yang@imperial.ac.uk
- 3 Shanghai Clinical Research and Trial Center, Shanghai, 201210, China
- 4 Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
- 5 6
- 7 Cardiovascular Research Centre, Royal Brompton Hospital, London, UK
- National Heart and Lung Institute, Imperial College London, London, UK Shanghai United Imaging Intelligence Co., Ltd., Shanghai, China
- 8 School of Biomedical Engineering &amp; Imaging Sciences, King's College London, London, UK

Abstract. The progression of Idiopathic Pulmonary Fibrosis (IPF) significantly correlates with higher patient mortality rates. Early detection of IPF progression is critical for initiating timely treatment, which can effectively slow down the advancement of the disease. However, the current clinical criteria define disease progression requiring two CT scans with a one-year interval, presenting a dilemma: a disease progression is identified only after the disease has already progressed . To this end, in this paper, we develop a novel diffusion model to accurately predict the progression of IPF by generating patient's follow-up CT scan from the initial CT scan. Specifically, from the clinical prior knowledge, we tailor improvements to the traditional diffusion model and propose a Clinically-Informed Residual Diffusion model, called CIResDiff. The key innovations of CIResDiff include 1) performing the target region preregistration to align the lung regions of two CT scans at different time points for reducing the generation difficulty, 2) adopting the residual diffusion instead of traditional diffusion to enable the model focus more on differences (i.e., lesions) between the two CT scans rather than the largely identical anatomical content, and 3) designing the clinicallyinformed process based on CLIP technology to integrate lung function information which is highly relevant to diagnosis into the reverse process for assisting generation. Extensive experiments on clinical data demonstrate that our approach can outperform state-of-the-art methods and effectively predict the progression of IPF.

GLYPH&lt;12&gt; Guang Yang and Dinggang Shen are co-senior last authors.

Keywords: Prediction of pulmonary fibrosis progression · Residual diffusion model · Clinically Informed · CLIP-based text processing.

## 1 Introduction

Idiopathic Pulmonary Fibrosis (IPF) is a severe and irreversible lung disease that scars and thickens lung tissues, leading to respiratory difficulties [15,21]. Timely treatment of IPF can effectively slow down its process and improve patients' quality of life [4, 22]. The progression of IPF can either remain stable or exacerbate over time. Consequently, to save healthcare expenses, in some countries (especially those with universal healthcare like the UK), antifibrotic treatment is initiated only if the IPF is confirmed to exacerbate over time. This approach presents a challenge: treatment is delayed until the fibrosis has advanced, losing early intervention opportunities for patients at high risk but not yet showing significant progression. In this context, predicting the progression of IPF in advance is extremely important for enabling timely treatment and reducing healthcare costs.

A feasible approach for predicting the progression of IPF is to generate the follow-up CT scan from the initial CT scan. This method of disease prediction or diagnosis through generation has already achieved success in numerous studies [5,6,10,11,14]. For instance, Han et al adopt the regularized generative adversarial networks to generate images of future time points for predicting the risk of osteoarthritis [6]. Jiang et al employ a transformer-based generative adversarial network to generate dual-energy CT images from single-energy CT for diagnosing postoperative cerebral hemorrhage [10]. Moreover, we choose to generate the follow-up CT scan rather than directly predicting disease progression from the initial CT scan considering the following facts: 1) The use of follow-up CT scan is more clinically natural in terms of diagnosis and thus more subjectively convincing. 2) The generation of follow-up CT scan can better exploit in the information entailed initial CT scan, thus more likely to produce a correct diagnosis. 3) The generated follow-up CT scan can provide additional information, such as the location of the lesion area.

Among the existing image generation techniques, the diffusion model has shown large potential and obtained great success [7, 20, 25]. They accomplish this by converting complex generation tasks into a series of simpler denoising tasks, enabling more stable and detailed generation. Meanwhile, clinical observations reveal the following prior information. First , due to patient posture and physiological movement, there is a significant spatial difference between the two CT scans. Second , as both scans are from the same individual, the majority of their content (i.e., the anatomical structure) is identical, while the lesion areas we focus on only constitute a minor part. Lastly , patients typically undergo lung function tests during their initial CT scan, and the information from these tests is highly relevant to the progression of IPF.

Taking all into consideration, in this paper we propose a Clinically-Informed Residual Diffusion model (CIResDiff) based on the traditional diffusion model to

Fig. 1: Overview of proposed CIResDiff. (a) and (c) provide the framework of CIResDiff as well as depict its implementation during both the training and testing phases, and (b) illustrates the details of clinically-informed process.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00938v2-figure-93ce3baa-8a31-4f57-b0af-5cd900cb3516.jpg)

predict the progression of IPF. Specifically, we improve the traditional diffusion model by 1) performing target region pre-registration for the two CT scans to enable voxel-to-voxel generation for reducing the complexity, 2) adopting residual diffusion to make the model's generation process focus more on the differences (i.e., the lesion areas) between the two CT scans for more precise generation of target areas and fast model inference, and 3) designing a clinically-informed process based on CLIP [18] technology to capture the correlation between lung function test information and corresponding CT scans, and utilizing this correlation to guide the generation for producing images with higher diagnostic value.

The main contributions of our work include: i) the first attempt to predict the progression of IPF by generating follow-up CT scans from initial scans, ii) developing a novel diffusion model (CIResDiff) tailored for this task based on clinical prior knowledge, and iii) demonstrating the effectiveness of our CIResDiff on collected datasets by extensive experiments.

## 2 Method

Our method is illustrated in Fig. 1. First, we extract and align the lung region from the two CT scans by the target region pre-registration. Then, in the residual diffusion process, the follow-up lung image x 0 is converted to the noisy initial lung image x T by incrementally incorporating the differences of two CT scans. Subsequently, the noisy initial lung image x T undergoes a series of reconstruction steps guided by the clinically-informed process until it's converted back to the follow-up lung image x 0 . Note that only the reverse process is involved during

the testing phase. In the following, we introduce the details of target region pre-registration, residual diffusion, and CLIP-based text processing module.

## 2.1 Target Region Pre-registration

Voxel-to-voxel supervised generation is less challenging compared to direct unsupervised generation. Hence, we employ a target area pre-registration to obtain spatially-aligned lung image pairs from the initial and follow-up CT scans for model training.

Specifically, for the input of two CT scans, to eliminate background interference, we first use the TotalSegmentator [23], an open-access tool based on nnU-Net and trained with more than one thousand samples, to segment the left and right lung regions from both CT scans. Then, we perform a dilation operation on the segmented results to preserve lung-surrounding tissues relevant to diagnosis. Subsequently, we apply an affine registration method [1] to align the segmented left and right lung regions individually. Finally, we merge the aligned left and right lung regions to obtain spatially-aligned lung region image pairs for model training.

## 2.2 Residual Diffusion

In the process of generating follow-up scans from initial scans, the main learning objective for the model is the lesion areas that are crucial for diagnosis but occupy a small proportion. Thus, we integrate residual learning with the diffusion model to enhance the model's ability to learn changes in such lesion areas, proposing a residual diffusion strategy. This strategy enables the diffusion model to exclusively learn the residuals between the initial and follow-up scans.

Unlike the traditional diffusion process, which disrupts the input initial lung image x 0 into pure Gaussian noise, our proposed residual diffusion converts the input initial lung image x 0 into noisy follow-up lung image x T . Specifically, we first calculate the difference e 0 = y 0 -x 0 between the initial lung image y 0 and the corresponding follow-up lung image x 0 , and then apply a shifting sequence { η t } T t =1 to incrementally add this difference e 0 to x 0 . In this way, the residual diffusion process can be formulated as follows:

$$q ( x _ { t } | x _ { 0 }, y _ { 0 } ) = \mathcal { N } ( x _ { t } ; x _ { 0 } + \eta _ { t } e _ { 0 }, k ^ { 2 } \eta _ { t } I ), \ \ t = 1, 2, \dots, T, \quad \quad ( 1 )$$

where t is the timestep which satisfies η 1 → 0, η T → 1, k is a hyper-parameter controlling the noise variance, and I is the identity matrix.

In this context, the reverse process, corresponding to the residual diffusion process, involves recovering the follow-up lung image from the noisy initial lung image rather than from pure noise. Such reverse process can estimate the posterior distribution p x ( 0 | y 0 ) via the following formulation:

$$p ( x _ { 0 } | y _ { 0 } ) = \int p ( x _ { T } | y _ { 0 } ) \prod _ { t = 1 } ^ { T } p _ { \theta } ( x _ { t - 1 } | x _ { t }, y _ { 0 } ) \text{d} x _ { 1 ; T }, \\ n _ { a } ( x _ { a }, \cdot | x _ { a }, u _ { a } ) = \mathcal { N } ( x _ { a }, \cdot \colon u _ { a } ( x _ { a }, u _ { a }, t ) ).$$

p θ ( x t -1 | x ,y t 0 ) = N ( x t -1 ; µ θ ( x , y t 0 , t ) , Σ θ ( x , y t 0 , t )) ,

$$p ( x _ { T } | y _ { 0 } ) \approx \mathcal { N } ( x _ { T } ; y _ { 0 }, k ^ { 2 } I ),$$

where Σ θ ( x , y t 0 , t ) is a fixed variance and µ θ ( x , y t 0 , t ) can be reparameterized as follows:

$$\colon \\ \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
 \quad \colon \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \ \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \ 0. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \.. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \.
 \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \. \ { \mathrm c }$$

Here, f θ is a deep neural network with parameter θ , aiming to ˆ x 0 , In our implementation, we utilize the attention U-Net [16] as f θ . The reverse process can be trained through the following reconstruction loss:

$$\mathcal { L } _ { r e c } = \min _ { \theta } \sum _ { t } \| f _ { \theta } ( x _ { t }, y _ { 0 }, t ) - x _ { 0 } \| _ { 2 } ^ { 2 }$$

## 2.3 Clinically-informed Process

During the diagnosis of IPF, lung function tests are typically conducted in addition to two CT scans. The information derived from these tests is highly relevant to the lung anatomical structure. Therefore, we believe that capturing the correlation between function test information and the corresponding CT scan, and using it to guide the generation process, can aid in producing CT scans with higher diagnostic value. To achieve this, we design a clinically-informed process based on Contrastive Language-Image Pretraining (CLIP) [18] technology.

Details of the clinically-informed process are shown in Fig. 1 (b), which includes pre-training and feature fusion stages. In the pre-training stage, we extract textual and image features from function text information and the corresponding initial scan using text and image encoders, respectively. Then, we constrain these features to be aligned using the contrastive loss. In this way, we can obtain the pre-trained text encoder for incorporating function text information into the reverse process. Note that the samples used for pre-training are exclusively from the training set and do not involve any test samples.

In the feature fusion stage, for a particular reconstruction in the reverse process, the pre-trained text encoder first extracts textual features z text from function test information. Then z text is fed into a denoising attention U-Net [16] at each step for calculation of cross-attention, where the query Q and key K are calculated from z text while the value V is still calculated from the output of the previous layer because our final goal is CT estimation. Denoting the output of previous layer as z CT , the CT-guided cross-attention can be formulated as follows:

$$O u t p u t = s o f t m a x ( \frac { Q _ { t e x t } K _ { t e x t } ^ { T } } { \sqrt { A } } + B ) \cdot V _ { P E T },$$

$$\mathcal { U } u p u = \mathcal { S O J } \, \i m u x \, & \sqrt { d } \, + \, D \, \jmath \cdot \, \nu \, P E T \,, \\ Q _ { t e x t } = C o n v _ { Q } ( z _ { t e x t } ), \quad K _ { t e x t } = C o n v _ { K } ( z _ { t e x t } ), \quad V _ { C T } = C o n v _ { V } ( z _ { C T } ),$$

6 Caiwen Jiang et al.

| Method   | PSNR [dB] ↑      | SSIM [%] ↑       |
|----------|------------------|------------------|
| DM       | 25 . 84 ± 0 . 93 | 98 . 19 ± 1 . 24 |
| DM-CIP   | 26 . 12 ± 1 . 07 | 98 . 75 ± 1 . 44 |
| DM-R     | 26 . 84 ± 0 . 92 | 99 . 03 ± 1 . 16 |
| DM-R-CIP | 27 . 52 ± 0 . 83 | 99 . 25 ± 0 . 92 |

Table 1: Quantitative results of ablation analysis, in terms of PSNR and SSIM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00938v2-figure-9f5b025d-ceea-4407-b532-6f4f5a9f6b54.jpg)

Fig. 2: Diagnostic evaluation.

where d is the number of channels, B is the position bias, and Conv ( ) · denotes the 1 × 1 × 1 convolution with stride of 1.

## 3 Experiments

## 3.1 Dataset and Implementation

Our dataset originates from OSIC , an openly accessible global database con9 taining numerous CT scans related to Idiopathic Pulmonary Fibrosis (IPF). We collect a total of 200 samples, each comprising of two CT scans with a 46 3 . ± 7 8 . weeks interval, along with corresponding lung function test information (including physiological indicators such as vital capacity, peak expiratory flow, etc.) and diagnostic labels. Among these 200 samples, 160 samples are used for training and the remaining 40 samples are used for testing. During the evaluation, we conduct five-fold cross-validation to exclude randomness.

In our implementation, experiments were conducted on the PyTorch platform using two NVIDIA Tesla A100 GPUs and an Adam optimizer with initial learning rate of 0.001. All images are resampled to voxel spacing of 1 × × 1 1 mm 3 and resolution of 512 × 512 × 256 , while their intensity range is normalized to [ 0 1 , ] by min-max normalization. For increasing the training samples and reducing the dependence on GPU memory, we extract the overlapped patches of size 96 × 96 × 96 from every whole CT scan. We evaluate the quantitative results by two commonly used quantitative metrics, including Peak Signal to Noise Ratio (PSNR) [9] and Structural Similarity Index (SSIM).

## 3.2 Ablation Analysis

To verify the effectiveness of our proposed strategies, i.e., residual diffusion and clinically-informed process , we design another four variant diffusion models (DMs) including: 1) DM: standard DM; 2) DM-CIP: DM with clinically-informed process; 3) DM-R: DM with residual diffusion; 4) DM-R-CIP: DM with residual diffusion and clinically-informed process. All methods use the same experimental settings, and their quantitative results are given in Table 1.

9 https://www.osicild.org

Table 2: Quantitative comparison of our CIResDiff with several state-of-the-art generation methods, in terms of PSNR and SSIM.

| Method           | PSNR [dB] ↑   | PSNR [dB] ↑   | PSNR [dB] ↑   | SSIM [%] ↑   | SSIM [%] ↑   | SSIM [%] ↑   |
|------------------|---------------|---------------|---------------|--------------|--------------|--------------|
| Pix2Pix-GAN [8]  | 22 . 53       | ±             | 2 . 13        | 94 . 16      | ±            | 1 . 59       |
| SAGAN [13]       | 22 . 93       | ±             | 1 . 98        | 94 . 84      | ±            | 1 . 44       |
| TransUNet [2]    | 23 . 35       | ±             | 1 . 72        | 95 . 49      | ±            | 1 . 24       |
| ResViT [3]       | 23 . 62       | ±             | 1 . 65        | 95 . 96      | ±            | 1 . 32       |
| DiffusionCT [19] | 25 . 21       | ±             | 0 . 92        | 97 . 82      | ±            | 1 . 21       |
| cDiff [17]       | 25 . 84       | ±             | 0 . 93        | 98 . 19      | ±            | 1 . 24       |
| CIResDiff        | 27 . 52       | ±             | 0 . 83        | 99 . 25      | ±            | 0 . 92       |

Fig. 3: Visual comparison of follow-up lung images produced by six different methods. From left to right are the input (initial scan), results of five other comparison methods (2nd-6th columns) and our CIResDiff (7th column), and the ground truth (follow-up scan). The corresponding difference maps between the generated results and GT are shown in the 2nd and 4th rows, where darker colors indicate larger differences. Red boxes show the lesion areas for detailed comparison.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00938v2-figure-9cafccf7-768c-46b1-9b3c-0fe9ddadee3e.jpg)

The quantitative results are provided in Table 1, from which, we can find the following observations. (1) DM-CIP with the clinically-informed process achieves better performance than DM. This proves that integrating highly relevant clinical information (i.e., function test information) into the reverse process is beneficial for the generation of follow-up CT scans. (2) DM-R, employing residual diffusion, achieves better performance than DM employing the traditional diffusion process, proving that residual diffusion is more appropriate for learning the mapping between highly similar initial and follow-up CT scans. (3) DM-R-CIP achieves better results than all other variants on both PSNR and SSIM, which shows both of our proposed strategies contribute to the final performance. These three comparisons conjointly verify the effective design of our proposed CIResDiff, where the residual diffusion and clinically-informed process both benefit our generation task.

## 3.3 Comparison with State-of-the-art Methods

We further compare our CIResDiff with six state-of-the-art generation methods, which can be divided into three classes: 1) GAN-based methods, including Pix2Pix-GAN [8] and SAGAN [13]; 2) transformer-based methods, including TransUNet [2] and ResViT [3]; 3) diffusion model-based methods, including DiffusionCT [19] and conditional Diffusion (cDiff) [17]. The quantitative and qualitative results are provided in Table 2 and Fig. 3, respectively.

Quantitative Comparison: The quantitative results are provided in Table 2. From the table, it is evident that diffusion model-based methods generally outperform GAN-based and transformer-based methods, validating our selection of the diffusion model as the baseline. Moreover, among all diffusion modelbased methods, our CIResDiff achieves the optimal results, with improvements in PSNR and SSIM over the sub-optimal cDiff by 1 68 . dB and 1 06% . , respectively. This demonstrates the effectiveness of our targeted improvements to the traditional diffusion model, including residual diffusion and clinically-informed process.

Qualitative Comparison: We provide a visual comparison of follow-up lung images generated by six different methods in Fig. 3. First, compared to other methods, our CIResDiff can generate the overall optimal images, characterized by the least noise, fewest artifacts but clearest structure. Second, in terms of detail, our CIResDiff can also most accurately generate the lesion areas (i.e., areas marked by red boxes) that are crucial for predicting the progression of IPF. Finally, the lightest color in the difference map demonstrates our CIResDiff can generate lung images with the smallest difference from the ground truth. Such key observations demonstrate that our CIResDiff is superior to those state-ofthe-art methods.

## 3.4 Diagnostic Evaluation

Our ultimate goal is to predict the progression of IPF using generated follow-up lung images. Therefore, we design relevant downstream diagnostic tasks to assess the diagnostic value of follow-up lung images generated by different methods. Specifically, we first train a ResNet-based classifier using real image pairs (i.e., real follow-up lung image and real initial lung image) from the training set. During the training process, follow-up and initial lung images are concatenated together as input to predict diagnostic labels. Then, we use the pre-trained classifier to assess the diagnostic value of images generated by different methods. During evaluation, the input is the fake image pairs (i.e., generated follow-up lung image and real initial lung image). The prediction results are provided in Fig. 2.

Fig. 2 shows that the follow-up lung images generated by our CIResDiff yield the best results for predicting IPF compared to other methods. Specifically, compared to Pix2Pix-GAN, which obtains the worst results, our CIResDiff shows a significant improvement in accuracy by 18% . This may be due to our method's incorporation of highly relevant lung function information into the generation process, thereby producing images more beneficial for diagnosis. These results

further confirm the effectiveness of using generation for IPF prediction and highlight the clinical application potential of our CIResDiff.

## 4 Conclusion

In this paper, to achieve early prediction of IPF progression, we develop a novel diffusion model named CIResDiff to predict patients's follow-up CT scan from the initial CT scan. To facilitate more precise and effective generation, our CIResDiff employs three strategies: 1) pre-aligning lung regions in both CT scans to reduce the complexity of generation; 2) learning residuals between initial and follow-up CT scans to focus more on lesion areas critical for prediction; 3) integrating clinical lung function test information to help generate results with greater diagnostic value. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches and can generate images with higher diagnostic value.

Acknowledgments. This work was supported in part by National Natural Science Foundation of China (grant numbers U23A20295, 62131015, 62250710165), the STI 2030-Major Projects (No. 2022ZD0209000), Shanghai Municipal Central Guided Local Science and Technology Development Fund (grant number YDZX20233100001001), the China Ministry of Science and Technology (STI2030Major Projects-2022ZD0213100), The Key R&amp;D Program of Guangdong Province, China (grant numbers 2023B0303040001, 2021B0101420006), the ERC IMI (10100 5122), the H2020 (952172), the MRC (MC/PC/21013), the Royal Society (IEC\NS FC\211235), the NVIDIA Academic Hardware Grant Program, the SABER project supported by Boehringer Ingelheim Ltd, NIHR Imperial Biomedical Research Centre (RDA01), Wellcome Leap Dynamic Resilience, UKRI guarantee funding for Horizon Europe MSCA Postdoctoral Fellowships (EP/Z002206/1), and the UKRI Future Leaders Fellowship (MR/V023799/1). This work was completed under the close collaboration between Caiwen Jiang and Xiaodan Xing, and they contributed equally to this work.

Declaration of Competing Interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## References

- 1. Avants, B., Tustison, N., Song, G., et al.: Advanced normalization tools (ANTS). Insight j 2 (365), 1-35 (2009)
- 2. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)

- 3. Dalmaz, O., Yurt, M., Çukur, T.: Resvit: Residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging 41 (10), 2598-2614 (2022)
- 5. Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., Greenspan, H.: GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 321 , 321-331 (2018)
- 4. Finnerty, J., Ponnuswamy, A., Dutta, P., Abdelaziz, A., Kamil, H.: Efficacy of antifibrotic drugs, nintedanib and pirfenidone, in treatment of progressive pulmonary fibrosis in both idiopathic pulmonary fibrosis (IPF) and non-IPF: a systematic review and meta-analysis. BMC Pulmonary Medicine 21 (1), 411 (2021)
- 6. Han, T., Kather, J., Pedersoli, F., Zimmermann, M., Keil, S., Schulze-Hagen, M., Terwoelbeck, M., Isfort, P., Haarburger, C., Kiessling, F., et al.: Image prediction of disease progression for osteoarthritis by style-based manifold extrapolation. Nature Machine Intelligence 4 (11), 1029-1039 (2022)
- 8. Isola, P., Zhu, J., Zhou, T., Efros, A.: Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition pp. 1125-1134 (2017)
- 7. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 , 6840-6851 (2020)
- 9. Jiang, C., Pan, Y., Cui, Z., Nie, D., Shen, D.: Semi-supervised standard-dose PET image generation via region-adaptive normalization and structural consistency constraint. IEEE Transactions on Medical Imaging 42 (10), 2974-2987 (2023)
- 11. Jiang, C., Wang, T., Pan, Y., Ding, Z., Shen, D.: Real-time diagnosis of intracerebral hemorrhage by generating dual-energy CT from single-energy CT. Medical Image Analysis 95 , 103194 (2024)
- 10. Jiang, C., Pan, Y., Wang, T., Chen, Q., Yang, J., Ding, L., Liu, J., Ding, Z., Shen, D.: S2DGAN: Generating dual-energy CT from single-energy CT for real-time determination of intracerebral hemorrhage. International Conference on Information Processing in Medical Imaging pp. 375-387 (2023)
- 12. Kreuter, M., Polke, M., Walsh, S., Krisam, J., Collard, H., Chaudhuri, N., Avdeev, S., Behr, J., Calligaro, G., Corte, T., et al.: Acute exacerbation of idiopathic pulmonary fibrosis: international survey and call for harmonisation. European Respiratory Journal 55 (4) (2020)
- 14. Ma, L., Shuai, R., Ran, X., Liu, W., Ye, C.: Combining DC-GAN with ResNet for blood cell image classification. Medical &amp; Biological Engineering &amp; Vomputing 58 , 1251-1264 (2020)
- 13. Lan, H., D., A., Toga, A., Sepehrband, F.: Three-dimensional self-attention conditional GAN with spectral normalization for multimodal neuroimaging synthesis. Magnetic Resonance in Medicine 86 (3), 1718-1733 (2021)
- 15. Maher, T., Bendstrup, E., Dron, L., Langley, J., Smith, G., Khalid, J., Patel, H., Kreuter, M.: Global incidence and prevalence of idiopathic pulmonary fibrosis. Respiratory Research 22 (1), 1-10 (2021)
- 17. Peng, J., Qiu, R., Wynne, J., Chang, C., Pan, S., Wang, T., Roper, J., Liu, T., Patel, P., Yu, D., et al.: CBCT-based synthetic CT image generation using conditional denoising diffusion probabilistic model. arXiv preprint arXiv:2303.02649 (2023)
- 16. Oktay, O., Schlemper, J., Folgoc, L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N., Kainz, B., et al.: Attention U-Net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)
- 18. Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from

natural language supervision. International Conference on Machine Learning pp. 8748-8763 (2021)

- 20. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning pp. 2256-2265 (2015)
- 19. Selim, M., Zhang, J., Brooks, M., Wang, G., Chen, J.: DiffusionCT: Latent diffusion model for CT image standardization. arXiv preprint arXiv:2301.08815 (2023)
- 21. Spagnolo, P., Kropski, J., Jones, M., Lee, J., Rossi, G., Karampitsakos, T., Maher, T., Tzouvelekis, A., Ryerson, C.: Idiopathic pulmonary fibrosis: Disease mechanisms and drug development. Pharmacology Therapeutics 222 , 107798 (2021)
- 23. Wasserthal, J., Breit, H., Meyer, M., Pradella, M., Hinck, D., Sauter, A., Heye, T., Boll, D., Cyriac, J., Yang, S., et al.: Totalsegmentator: Robust segmentation of 104 anatomic structures in CT images. Radiology: Artificial Intelligence 5 (5) (2023)
- 22. Torrisi, S., Kahn, N., Vancheri, C., Kreuter, M.: Evolution and treatment of idiopathic pulmonary fibrosis. La Presse Médicale 49 (2), 104025 (2020)
- 24. Wasserthal, J., Meyer, M., Breit, H., Cyriac, J., Yang, S., Segeroth, M.: Totalsegmentator: robust segmentation of 104 anatomical structures in CT images. arXiv preprint arXiv:2208.05868 (2022)
- 26. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision pp. 2223-2232 (2017)
- 25. Yue, Z., Wang, J., Loy, C.: Resshift: Efficient diffusion model for image superresolution by residual shifting. Advances in Neural Information Processing Systems 36 (2024)
Caiwen Jiang Xiaodan Xing Zaixin Ou Mianxin Liu Walsh Simon Guang Yang Dinggang Shen
CIResDiff: A Clinically-Informed Residual Diffusion Model for Predicting Idiopathic Pulmonary Fibrosis Progression
The progression of Idiopathic Pulmonary Fibrosis (IPF) significantly
correlates with higher patient mortality rates. Early detection of IPF
progression is critical for initiating timely treatment, which can effectively
slow down the advancement of the disease. However, the current clinical
criteria define disease progression requiring two CT scans with a one-year
interval, presenting a dilemma: a disease progression is identified only after
the disease has already progressed. To this end, in this paper, we develop a
novel diffusion model to accurately predict the progression of IPF by
generating patient's follow-up CT scan from the initial CT scan. Specifically,
from the clinical prior knowledge, we tailor improvements to the traditional
diffusion model and propose a Clinically-Informed Residual Diffusion model,
called CIResDiff. The key innovations of CIResDiff include 1) performing the
target region pre-registration to align the lung regions of two CT scans at
different time points for reducing the generation difficulty, 2) adopting the
residual diffusion instead of traditional diffusion to enable the model focus
more on differences (i.e., lesions) between the two CT scans rather than the
largely identical anatomical content, and 3) designing the clinically-informed
process based on CLIP technology to integrate lung function information which
is highly relevant to diagnosis into the reverse process for assisting
generation. Extensive experiments on clinical data demonstrate that our
approach can outperform state-of-the-art methods and effectively predict the
progression of IPF.