## Chest CT Super-resolution and Domain-adaptation using Memory-efficient 3D Reversible GANs

Tycho F. A. van der Ouderaa 1 2 ,

tychovdo@gmail.com

Daniel E. Worrall 1 Bram van Ginneken 2

d.e.worrall@uva.nl

bram.vanginneken@radboudumc.nl

- 1 University of Amsterdam, Amsterdam, The Netherlands
- 2 Diagnostic Image Analysis Group, Radboud UMC, Nijmegen, The Netherlands

## 1. Introduction

Recently, paired (e.g. Pix2pix (Isola et al., 2017)) and unpaired (e.g. CycleGAN (Zhu et al., 2017)) image-to-image translation methods have shown effective in medical imaging tasks (Yi et al., 2018) (Wolterink et al., 2017). In practice, however, it can be difficult to apply these deep models on medical data volumes, such as from MR and CT scans, since such data volumes tend to be 3-dimensional and of a high spatial resolution pushing the limits of the memory constraints of GPU hardware that is typically used to train these models.

Recent studies in the field of invertible neural networks have shown that reversible neural networks do not require to store intermediate activations for training (Gomez et al., 2017). We use the RevGAN model (van der Ouderaa and Worrall, 2019) that makes use of this property to perform memory-efficient partially-reversible image-to-image translation. We demonstrate this by performing a 3D super-resolution and a 3D domain-adaptation task on chest CT data volumes.

## 2. Problem Statement

The tasks addressed in this paper relates to CT imaging of the chest. Recent scanners allow for high-resolution scans with isotropic voxel sizes around or below 0.5 mm 3 and this requires the reconstruction of image matrices consisting of slices of 1024 × 1024 pixels, contrary to the traditionally used slices of 512 × 512 pixels. In these high-resolution scans, fine parenchymal details such as small airway walls, vasculature and lesion texture, are better visible. At the same time, the vast majority of scans available today for training networks consist of 512 matrices and often thicker slices, around 2mm. In addition, a wide variety of CT reconstruction kernels are used in practice, from more noisy high frequency kernels to smoother (soft kernels), and both traditional filtered backprojection as well as iterative reconstruction methods, varying between scanner vendors, are used (Mets et al., 2012). To produce robust and accurate image processing results, it is therefore desirable to pre-process chest CT scans to a standardized high resolution, and to remove the structural differences resulting from the use of different reconstruction algorithms. We aim to provide a proof-of-principle that we can use reversible networks to create a model capable of preprocessing CT scans to high resolution with a standardized appearance.

1

Figure 1: Illustration of the model: Encoders Enc X (green) and Enc Y (blue) lift/encode from the image space features spaces ˜ X and ˜ . Y Decoders Dec X (red) and Dec Y (purple) project/decode back to X and Y . The invertible mapping C transforms between ˜ and X ˜ . Y

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-61834cf0-ebca-4904-b459-5760b9310040.jpg)

## 3. Method

To perform memory-efficient image-to-image translation in 3D, we adapt the RevGAN model (van der Ouderaa and Worrall, 2019) that combines the Pix2pix loss (Isola et al., 2017) in Equation 1 for paired training or the CycleGAN loss (Zhu et al., 2017) in Equation 2 for unpaired training with a 3D architectures (Figure 1) and a reversible core to lower memory usage during training. For a more extensive description of the model losses, including a description of L GAN , L L1 and L cyc , we refer to the original RevGAN paper.

Paired Loss For paired training, we combine the L GAN loss with the L 1 loss:

$$L _ { \text{paired} } ( F, G ) & = L _ { \text{GAN} } ( G, D _ { X } ) + L _ { \text{GAN} } ( F, D _ { Y } ) \\ & + \lambda ( L _ { \text{L1} } ( F, X, Y ) + L _ { \text{L1} } ( G, X, Y ) )$$

Unpaired Loss For unpaired training, we combine the L GAN loss with the L cyc loss:

$$L _ { \text{unpaired} } ( F, G ) & = L _ { \text{GAN} } ( G, D _ { X } ) + L _ { \text{GAN} } ( F, D _ { Y } ) ) \\ & + \lambda L _ { \text{cyc} } ( G, F )$$

The λ parameter determines the relative importance of the terms in the losses.

## 4. Data

Super-resolution The Super-resolution dataset contains 18 train volumes and 5 test volumes each consisting of 671 ( ± 49) slices of size 1024 × 1024 obtained with a high-end Canon CT scanner (Aquilion ONE). We train for 125 epochs on 128 × 128 × 128 patches that were normalized in the range [-1, 1], uniformly redistributed from [-1150, 350] Hounsfield Units (HU). The source domain was generated by aggressively down-sampling 4 times in the z-dimension and 2 times in the x and y dimensions (corresponding to a typically used resolution of 512 by 512).

Domain-adaptation The Domain-adaptation dataset contains 17 CT scans for training and 3 CT scans for testing from The National Lung Screening Trial (NLST) (Team, 2011). All scans were obtained by a Siemens scanner and both a smooth (B30f) and a sharper (B50f) reconstruction kernel were available. Each scan contains an average of 169 ( ± 14 7) . axial 512 × 512 slices. We train for 125 epochs on 64 × 64 × 64 patches that were normalized in a range of [-1, 1], uniformly redistributed from [-1150, 350] HU.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-92413706-f01d-477c-817e-f3c7b5904f35.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-139cac6a-b2cc-4430-ad1d-e757ea27f347.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-f58e0a3b-2511-4583-a199-37b72af7b97a.jpg)

Paired 3D-RevGAN

Figure 2: Visualization of a full and patched coronal slice of the first volume in the Superresolution test set. From left to right: low-resolution input, trilinear up-sampling, paired 3D-RevGAN, unpaired 3D-RevGAN and high-resolution ground-truth.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-2daf94f1-9560-49d0-b917-bf7d8c7a6df4.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-578fa013-7c62-4bd9-a0cd-18742d47ab2e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00295v1-figure-85b5a94a-bc87-4764-8ddc-f62327c80047.jpg)

## 5. Results

Qualitative Results In Figure 2, we present a comparison between trilinear up-sampling, paired RevGAN and unpaired RevGAN. Upon closer inspection of the more detailed structures in the CT volumes, we find that the model produces visually compelling results and often is able to correctly fill-in parenchymal details such as small airway walls, vasculature and tissue texture. The output, however, is still far from perfect, which we account to the fact that the input has been down-sampled heavily. We expect even better results when the input source images are of a higher resolution.

Quantitative Results As we can see from Table 1, we are able to increase performance by adding (+R) reversible blocks, without increasing memory cost. In Table 2, we show memory savings resulting from reversible layers.

Table 1: Mean and variance 2 of MAE, PSNR and SSIM scores. We can improve performance, using deeper models at the same level of memory complexity as shallow models.

| Model       | Super-resolution (LR → HR)   | Super-resolution (LR → HR)   | Super-resolution (LR → HR)   | Domain-adaptation (B50f → B30f)   | Domain-adaptation (B50f → B30f)   | Domain-adaptation (B50f → B30f)   |
|-------------|------------------------------|------------------------------|------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|
| Model       | MAE                          | PSNR                         | SSIM                         | MAE                               | PSNR                              | SSIM                              |
| Unpaired    | 0.24 ± 0.007                 | 15.43 ± 0.39                 | 0.44 ± 0.008                 | 0.14 ± 0.003                      | 14.56 ± 0.22                      | 0.28 ± 0.004                      |
| Unpaired+2R | 0.23 ± 0.014                 | 16.43 ± 0.44                 | 0.50 ± 0.024                 | 0.13 ± 0.003                      | 17.44 ± 0.20                      | 0.29 ± 0.003                      |
| Paired      | 0.18 ± 0.001                 | 15.89 ± 0.16                 | 0.46 ± 0.008                 | 0.14 ± 0.001                      | 18.29 ± 0.15                      | 0.33 ± 0.007                      |
| Paired+2R   | 0.15 ± 0.000                 | 18.19 ± 0.08                 | 0.48 ± 0.008                 | 0.10 ± 0.001                      | 23.24 ± 0.16                      | 0.46 ± 0.009                      |

Table 2: Memory usage (MiB) of 3D-RevGAN model on GPU on a Nvidia Tesla K40m GPU. Note that the memory cost to store activations stays constant for deeper models.

|   Depth |   Model | Activations (Ours)   | Activations (Naive)   |
|---------|---------|----------------------|-----------------------|
|       0 |     630 | + 2312.4             | (+ 2316.6)            |
|       1 |     644 | + 2312.4             | (+ 2719.2)            |
|       2 |     652 | + 2312.4             | (+ 3919.3)            |
|       4 |     700 | + 2312.4             | (+ 4319.2)            |
|       8 |     748 | + 2312.4             | (+ 5519.5)            |

## 6. Conclusion

This study provides a proof-of-principle for learned 3D pre-processing of CT scans to high resolution with a standardized appearance. We use the RevGAN model to successfully perform memory-intensive 3D domain-adaptation and 3D super-resolution tasks, using paired and unpaired data. Thereby, this study provides additional evidence that reversibility can be useful in practical settings to lower memory requirements of deep models. Future research should determine how well the model generalizes beyond the data used in this study.

## Acknowledgements

We are grateful to the Diagnostic Image Analysis Group (DIAG) of the Radboud University Medical Center and the Netherlands Organisation for Scientific Research (NWO) for supporting this research and providing computational resources.

## References

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems , pages 2214-2224, 2017.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint , 2017.

OM Mets, PA De Jong, B Van Ginneken, HA Gietema, and JWJ Lammers. Quantitative computed tomography in copd: possibilities and limitations. Lung , 190(2):133-145, 2012.

National Lung Screening Trial Research Team. The national lung screening trial: overview and study design. Radiology , 258(1):243-253, 2011.

Tycho FA van der Ouderaa and Daniel E Worrall. Reversible gans for memory-efficient image-to-image translation. arXiv preprint arXiv:1902.02729 , 2019.

Jelmer M Wolterink, Anna M Dinkla, Mark HF Savenije, Peter R Seevinck, Cornelis AT van den Berg, and Ivana Iˇgum. Deep mr to ct synthesis using unpaired data. In s International Workshop on Simulation and Synthesis in Medical Imaging , pages 14-23. Springer, 2017.

Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. arXiv preprint arXiv:1809.07294 , 2018.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint , 2017.
Tycho F. A. van der Ouderaa Daniel E. Worrall Bram van Ginneken
Chest CT Super-resolution and Domain-adaptation using Memory-efficient 3D Reversible GANs
Recently, paired (e.g. Pix2pix) and unpaired (e.g. CycleGAN) image-to-image
translation methods have shown effective in medical imaging tasks. In practice,
however, it can be difficult to apply these deep models on medical data
volumes, such as from MR and CT scans, since such data volumes tend to be
3-dimensional and of a high spatial resolution pushing the limits of the memory
constraints of GPU hardware that is typically used to train these models.
Recent studies in the field of invertible neural networks have shown that
reversible neural networks do not require to store intermediate activations for
training. We use the RevGAN model that makes use of this property to perform
memory-efficient partially-reversible image-to-image translation. We
demonstrate this by performing a 3D super-resolution and a 3D domain-adaptation
task on chest CT data volumes.