summaries: This paper introduces the "clustering effect" of linear components of adversarial robust models. To be specific, the authors propose to remove the non-linear functions in networks and further integrate all linear components into a single weight matrix W in order to directly study the effect of linearity in adversarial models. By conducting experiments with W, the authors find that adversarially trained models tend to show clustering effects in both weight and feature dimensions. This effect further benefits some downstream tasks such as adversarial attack and transfer learning and can be enhanced with additional supervision. In all, the phenomenon showed in this paper is interesting and deserves more and deeper research.

summaries: Adversarial robustness has received a lot of attention along with the study of adversarial data. So far, almost works have shown that robust classifier not only outperform under comprehensive adversarial attack evaluations but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear.  In this paper, they investigate adversarial robustness from the perspective of linear components. From this very novel perspective, they find that there are some statistical properties for comprehensively robust classifiers. Specifically, robust classifiers show an obvious hierarchical clustering effect on their linear sub-networks. Based on the above investigation, this paper applies this on more tasks, such as robustness boosting and domain adaption.

summaries: The paper analyses adversarially robust models in terms of their clustering behavior with respect to the learned weights as well as with respect to the class-wise feature representations. For this purpose, the CNNs are linearized, i.e. non-linearities (ReLUs) are removed from the trained network such that it can be represented by a single weight matrix W, where each column corresponds to one class. The paper comes to the conclusion that hierarchical clustering of these representations is an indicator for adversarial robustness. Consequently, it formulates a regularization term which encourages such clustering behavior. The resulting models are evaluated in terms of adversarial robustness and in the context of domain adaptation.

summaries: This paper explores adversarial robustness via focusing on linear sub-networks. Through back propagation on linear sub-networks of a pre-trained network, an implicit weight expression which represents approximated class-wise responses can be obtained. With this matrix, the correlation among classes can be computed for exploring the hierarchy of classes. Similarly, the feature distance for different classes aligns well with class hierarchy through empirical observations. Based on these observations, the authors propose to impose feature distance regularization based on the computed class hierarchy.

summaries: The paper proposed a novel method of linear model exploration by extracting their implicit linear matrix expression and find the weight clustering effect. The paper also claimed that the adversarial robustness shows close relationship with such clustering effect, which enables the further understanding and improvement of various tasks in the experimental parts. Overall, I like the analysis in the paper, it provides an interesting and reasonable way to understand adversarial robustness via linearity of weights.

abstractText: Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, we interpret adversarial robustness from the perspective of linear components, and find that there exist some statistical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing or replacing all non-linear components (e.g., batch normalization, maximum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy. Our code is available at https://github.com/bymavis/Adv_Weight_NeurIPS2021.

authors:
Motasem Alfarra
Juan C Pérez
Adel Bibi
Ali Thabet
Pablo Arbeláez
Bernard Ghanem

title: Clustr: Clustering training for robustness

venue: arXiv e-prints,

authors:
Anish Athalye
Nicholas Carlini
David Wagner

title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples

venue: In ICML,

authors:
Yang Bai
Yuyuan Zeng
Yong Jiang
Yisen Wang
Shu-Tao Xia
Weiwei Guo

title: Improving query efficiency of black-box adversarial attack

venue: In ECCV,

authors:
Yang Bai
Yuyuan Zeng
Yong Jiang
Shu-Tao Xia
Xingjun Ma
Yisen Wang

title: Improving adversarial robustness via channel-wise activation suppressing

venue: In ICLR,

authors:
Nitin Bansal
Xiaohan Chen
Zhangyang Wang

title: Can we gain more from orthogonality regularizations in training deep networks? 2018

authors:
Nicholas Carlini
David Wagner

title: Defensive distillation is not robust to adversarial examples

venue: arXiv preprint arXiv:1607.04311,

authors:
Moustapha Cissé
Piotr Bojanowski
Edouard Grave
Yann Dauphin
Nicolas Usunier

title: Parseval networks: Improving robustness to adversarial examples

authors:
Adam Coates
Andrew Ng
Honglak Lee

title: An analysis of single-layer networks in unsupervised feature learning

venue: In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,

authors:
Francesco Croce
Matthias Hein

title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

authors:
Nilaksh Das
Madhuri Shanbhogue
Shang-Tse Chen
Fred Hohman
Siwei Li
Li Chen
Michael E Kounavis
Duen Horng Chau

title: Compression to the rescue: Defending from adversarial attacks across modalities

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL,

authors:
Angus Galloway
Anna Golubeva
Thomas Tanay
Medhat Moussa
Graham W Taylor

title: Batch normalization is a cause of adversarial vulnerability

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

authors:
Yiwen Guo
Qizhang Li
Hao Chen

title: Backpropagating linearly improves transferability of adversarial examples

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: arXiv preprint arXiv:

authors:
Gao Huang
Zhuang Liu
Laurens Van Der Maaten
Kilian Q Weinberger

title: Densely connected convolutional networks

authors:
Daniel Jakubovitz
Raja Giryes

title: Improving dnn robustness to adversarial attacks using jacobian regularization

authors:
Xiaojun Jia
Xingxing Wei
Xiaochun Cao
Hassan. Foroosh

title: Comdefend: An efficient image compression model to defend adversarial examples

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

venue: In NeurIPS,

authors:
Yandong Li
Lijun Li
Liqiang Wang
Tong Zhang
Boqing Gong

title: Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. 2019

authors:
Ziquan Liu
Yufei Cui
Antoni B Chan

title: Improve generalization and robustness of neural networks via weight scale shifting invariant regularizations

venue: arXiv preprint arXiv:2008.02965,

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Nicolas Papernot
Patrick McDaniel
Xi Wu
Somesh Jha
Ananthram Swami

title: Distillation as a defense to adversarial perturbations against deep neural networks

venue: In S&P,

title: On the momentum term in gradient descent learning algorithms

venue: Neural Networks,

authors:
Kevin Roth
Yannic Kilcher
Thomas Hofmann

title: Adversarial training generalizes datadependent spectral norm regularization

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael S. Bernstein
Alexander C. Berg
Fei-Fei Li

title: Imagenet large scale visual recognition challenge

venue: arXiv preprint arXiv: 1409.0575,

authors:
Hadi Salman
Andrew Ilyas
Logan Engstrom
Ashish Kapoor
Aleksander Madry

title: Do adversarially robust imagenet models transfer better

venue: arXiv preprint arXiv:2007.08489,

authors:
Shibani Santurkar
Dimitris Tsipras
Aleksander Madry

title: Breeds: Benchmarks for subpopulation shift

venue: arXiv preprint arXiv:2008.04859,

authors:
Qi Tian
Kun Kuang
Kelu Jiang
Fei Wu
Yisen Wang

title: Analysis and applications of class-wise robustness in adversarial training

venue: In KDD,

authors:
Yisen Wang
Xuejiao Deng
Songbai Pu
Zhiheng Huang

title: Residual convolutional ctc networks for automatic speech recognition

venue: arXiv preprint arXiv:1702.07793,

authors:
Yisen Wang
Xingjun Ma
James Bailey
Jinfeng Yi
Bowen Zhou
Quanquan Gu

title: On the convergence and robustness of adversarial training

authors:
Yisen Wang
Difan Zou
Jinfeng Yi
James Bailey
Xingjun Ma
Quanquan Gu

title: Improving adversarial robustness requires revisiting misclassified examples

authors:
Dongxian Wu
Yisen Wang
Shu-Tao Xia
James Bailey
Xingjun Ma

title: Skip connections matter: On the transferability of adversarial examples generated with resnets

authors:
Dongxian Wu
Shu-Tao Xia
Yisen Wang

title: Adversarial weight perturbation helps robust generalization

authors:
Jia Xu
Yiming Li
Yong Jiang
Shu-Tao Xia

title: Adversarial defense via local flatness regularization

venue: In ICIP,

authors:
Ziang Yan
Yiwen Guo
Changshui Zhang

title: Deep defense: Training dnns with improved adversarial robustness

authors:
Hongyang Zhang
Yaodong Yu
Jiantao Jiao
Eric P Xing
Laurent El Ghaoui
Michael I Jordan

title: Theoretically principled trade-off between robustness and accuracy. 2019

heading: 1 Introduction

text:
Nowadays, deep neural networks (DNNs) have shown a strong learning capacity through a huge number of parameters and diverse structures [16, 31, 11]. Meanwhile, adversary has raised increasing security concerns on DNNs due to the observation of adversarial examples (i.e., the examples mislead a classifier when crafted with human imperceptible but carefully designed perturbations) [13]. Adversarial robustness and defense techniques have thus become crucial for deep learning, yet many adversarial defense techniques are found with deficiencies such as obfuscated gradient [2]. Up to now, the widely accepted techniques to improve adversarial robustness are adversarial training variants [23, 32, 38, 33, 35, 36], by training a DNN after data augmentation with the worst-case adversarial examples. Recent study has found that not only do robust models show moderate robustness under vast attacks, but also they can surprisingly boost the downstream tasks (e.g., domain adaption task with subpopulation shift) [29].
However, the underlying mechanism of adversarial robustness is still not clear. Existing studies have explored adversary on some specific components of DNNs, such as batch normalization [12], skip connection [34], or activation layers [14, 4], which yield some enlightening understanding. On the other hand, despite variation in DNNs’ structure, the components can be roughly divided into linear and non-linear ones. The non-linear components tend to be instance-wise. For example, adversarial
*Equal contribution. †Correspondence to: Yisen Wang (yisen.wang@pku.edu.cn) and Shu-Tao Xia (xiast@sz.tsinghua.edu.cn).
35th Conference on Neural Information Processing Systems (NeurIPS 2021) .
and benign examples tend to present different patterns on activation [4]. In contrast, for those linear components, once the optimization is done and model evaluation mode is activated, they are shared and fixed independently from inputs. Nowadays, the intrinsic relationship of adversary with linearity has been studied. Some ascribes the adversary to linear accumulation of perturbations from inputs to final outputs, dubbed ‘accidental steganography’ [13]. Some show that linear backward propagation enhances adversarial transferability [34, 14]. Nevertheless, such mentioned linearity analyses are all on the original non-linear models, which hinders a deeper understanding on linearity directly. Further essential exploration is encouraged.
In this paper, we thus study adversarial robustness on the statistical regularity of linear components. We provide a novel insight on adversarial robustness by showing the clustering effect well aligned with class hierarchy. As shown in Fig. 1(d), given a data set, superclasses are defined by grouping semantically similar subclasses together to form a hierarchy, e.g., in CIFAR-10, ‘Cat’ and ‘Dog’ represent different subclasses/fine labels, despite both being the same superclasses/coarse labels ‘Animal’. Based on this, we conduct backward propagation on linearized sub-networks of robust and non-robust DNNs (through adversarial/standard training respectively) to extract the corresponding implicit linear expression. In this fashion, we could obtain a Dinput⇥Doutput linear matrix W , which is extracted from inputs to predictive outputs. Dinput represents for the dimension of one input data, specifically defined as width ⇥ height ⇥ channel. Doutput represents for the dimension of one output vector, which is also the number of classes. We further study the correlation of vectors in W by matrix C, whose shape is Doutput ⇥Doutput. The details of W and C are introduced in Sec. 3.2. Then we find that robust models tend to show hierarchical clustering effect in correlation matrix C, which is surprisingly consistent with class hierarchy. For example, in Fig. 1 on CIFAR-10, the fine labels (0,1,8,9) belong to ‘Non-animal’, others belong to ‘Animal’. The matrix C of robust model shows block clustering: values are close to ‘+1’ in the two superclasses (0,1,8,9) and (2,3,4,5,6,7), and values are close to ‘-1’ across these two superclasses. Such clustering effect is enhanced by our clustering regularization penalty in Fig. 1(d). This phenomenon could be semantically explained as subclasses from the same superclass share more feature similarities. Such hierarchical clustering effect can give a novel and insightful explanation on the superiority of robust models in some observed downstream tasks, as they can extract more semantic and representative features. To further confirm our findings, we enhance the hierarchical clustering effect in both adversarial robustness and downstream tasks, e.g., domain adaption proposed in [29, 28]. The improvements of experimental results demonstrate that the superiority of robust models is closely related to their hierarchical classification capability. Our contributions are summarized as follows:
• To the best of our knowledge, we are the first to systematically analyze the statistical regularity of adversarially robust models (through adversarial training) compared to nonrobust models (through standard training) on their linearized sub-networks.
• We present an intriguing phenomenon of hierarchical clustering effect in robust models, and provide a novel yet insightful understanding of adversarial robustness. The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models, which benefits a lot in various tasks.
• Based on the observations, we propose a plugged-in hierarchical clustering training strategy to generally enhance adversarial robustness and investigate some intriguing adversarial attack findings. Besides adversarial-related study, we further explore some downstream tasks with
the understanding of hierarchical clustering, e.g., domain adaption with subpopulation shift. Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well.

heading: 2 Related Work

heading: 2.1 Adversarial Attack and Defense

text:
Attack Given a clean example x, its true label y, a DNN model F with parameters ✓ and its loss function L, the goal of adversarial attack is to find an adversarial example x0 that is close to clean example on pixel level but can fool F to give an incorrect prediction. Fast Gradient Sign Method (FGSM) [13] adds perturbations on clean example x by one step with step size ✏:
x0 = x+ ✏sign(rxL(F(x, ✓), y)). (1) Projected Gradient Descent (PGD) [23] adds perturbations on clean example x by K steps with smaller step size ↵. After every step (k-th) attack, adversarial example is projected into the ✏-ball B✏(x) around clean example (function ⇧):
xk = ⇧(xk 1 + ↵sign(rxL(F(xk 1, ✓), y))). (2) There are other attack techniques, such as white-box CW attack [6], black-box attacks [21, 3] and adaptive AutoAttack [9].
Defense Many adversarial defense techniques have been proposed since then, such as input preprocessing [19], defensive distillation [24], model compression [10], and adversarial training [23]. Among them, adversarial training variants are assumed to be most effective in comprehensive attack evaluations. They solve a min-max problem as:
min ✓ max x02B✏(x)
L(F(x0, ✓), y). (3)
The inner maximum problem often generates adversarial examples by FGSM or PGD, while the outer minimum problem optimizes the worst-case loss. Some defense technique improves robustness through class-wise feature clustering [1, 30], which is different from our instance-wise feature clustering.

heading: 2.2 Linearity Exploration in Adversary

text:
Beside attack or defense techniques, the cause and understanding of adversary are studied. The linearity is one essential perspective with no doubt. Previous studies attribute the existing of adversarial examples to the linearity of DNNs [13] and observe the adversarial transferability enhancement on more linear models [34, 14]. Some works study linearity from another perspective by focusing on the properties of weight layers, e.g., their norms, variances and orthogonality. Lp norm regularization works by pulling examples far away from the decision boundary, which comes as the side-effect of adversarial robustness [37, 18]. Spectral norm [7, 26] is useful in adversarial defense, which is defined as i =
kwixk2 kxk2 and is used to constrain the Lipschiz constant of a DNN F on x (assuming
Lipschiz continuous) kF(x+ ) F(x)k2  kxk2  Q
ikxk2. The weight scale shifting issue is also discussed in adversary [22], that is, scale of weights can be shifted between layers without changing the input-output function specified by the network, which could affect the capacity to regularize models. Then one weight scale shift invariant regularization is proposed and improves adversarial robustness. Moreover, the orthogonality could help improve generalization and adversarial robustness [5, 7] by inducing uncorrelated features. However, these experiments are all conducted on the original non-linear models instead of the linear components directly, which hinders a deeper understanding on linearity.

heading: 3 Observations on Linear Components

text: The robust models perform completely differently from non-robust ones especially under comprehensive adversarial attacks. However, the statistical regularity on linear components of these robust models is yet under little exploration. Different from related works, we explore linear components directly. To be specific, we extract a weight matrix expression of linearized sub-networks, estimating the linear propagation from inputs to outputs.

heading: 3.1 Extracting a Linear Weight Matrix

text:
Given a DNN F composed of L linear layers and non-linear components, the output y = F(x) could be expressed as
y = gL(wL ⇥ gL 1(wL 1 ⇥ . . . g1(w1 ⇥ x+ b1) · · ·+ bL 1) + bL), (4)
where gi is a series of non-linear components (e.g., activation function ReLUs), wi and bi are the expressions of linear ones. Then a corresponding linear sub-network of this original one could be extracted with the same weights and architectures, yet it removes activation function (e.g., ReLU layers) and batch normalization layers, and further replaces maximum pooling layers with average pooling layers. This output of linear network ylinear = Flinear(x) is expressed as
ylinear = wL ⇥ (wL 1 ⇥ ...(w1 ⇥ x+ b1)...+ bL 1) + bL. (5)
Thus in Flinear, each input x is multiplied by an instance-agnostic total weight
W = wL ⇥ wL 1 ⇥ ...w1 + X
i,j
bi...⇥ wj .... (6)
However, such total weight expression W in DNNs is difficult to compute directly, because the multiplication of convolution layers is infeasible. Instead, we propose to compute W by applying backward propagation on the linear sub-network. To be specific, given a random input x and a pre-trained DNN F , W is computed as the gradient propagated on linear sub-network Flinear following Algorithm 1. For a non-linear model, as the activation part is example-wise, it is difficult to analyse the original network with non-linear components. In contrast, the linear component is generally applied on all examples and example-agnostic. So we extract a linear expression from the original non-linear model to approximate and analyse the model performance. Intuitively, although the linearized network has limited capacity, we think that the class-wise directions (the extracted linearized weight vectors) should still represent some class-wise directions, which holds a connection with the feature space of the non-linear models.
Algorithm 1 Extracting the Linear Weight Matrix W Input: A random input x, a pre-trained DNN model F with Doutput classes Output: Weight matrix W
Get the corresponding linear sub-network Flinear from F Conduct forward propagation as: ylinear = Flinear(x) for i in range(Doutput) do ylinear[i].backward() W [:, i] = x.grad
end for
return W

heading: 3.2 Observation on Weight Clustering Effect

text:
As indicated, though instance-wise performance varies on one specific DNN with non-linear components, it shares the same linear sub-network (i.e. parameters and linear structures) and thus the same implicit weight matrix W . As the implicit matrix W could estimate linear output scores ylinear of any input x when forward propagating on the linear sub-network, it hints some class-wise linear amplifications. After extracting W with shape Dinput ⇥Doutput, we further explore the correlation across classes by normalizing weight vectors in W (to avoid the class-wise scale variance) and computing a correlation matrix C following
Ci,j = WTi kWTi k2 ⇥ WjkWjk2 . (7)
In such fashion, the matrix C, whose shape is Doutput ⇥Doutput, has all elements in [-1, 1]. C(i, j) represents for cosine value of two weight vectors corresponding to class i and j on linear sub-networks, which also represents for the correlation of class-wise weight vectors. If the element C(i, j) is close to 1, it means that i-th and j-th class-wise weight vectors are strongly positive correlated in the linear weight space, which also demonstrates that class i and j are more likely to be positively related with
each other. In contrast, for the element C(i, j) close to -1, it means that i-th and j-th class-wise weight vectors are strongly negative correlated in the linear weight space, which also demonstrates that class i and j are more likely to be negatively related with each other. Datasets and Setups Beside Fig. 1, we show more clustering effect across data sets using STL-10 [8] and two reconstructed data sets CIFAR-20 and ImageNet-10, composed of 20/10 subclasses from 4/2 superclasses in CIFAR-100 and TinyImageNet [27]. The details of constructed data sets are in Appendix A. For robust models, we adversarially train ResNet-18 [15] using PGD-10 (✏ = 8/255 and step size 2/255) with random start. The non-robust models are standard trained. Both models are trained for 200 epochs using SGD with momentum 0.9, weight decay 2e-4, and initial learning rate 0.1 which is divided by 10 at the 75-th and 90-th epoch. Result Analysis We plot the correlation matrix C in Fig. 2. Similar to Fig. 1 on CIFAR-10, the similar block clustering effect is observed on robust models and aligns well with class hierarchy. To be specific, in Fig. 2(a) and 2(e) on STL-10, fine labels (0,2,8,9) belong to ‘Non-animal’, others belong to ‘Animal’, and the matrix C of robust model shows block clustering: values close to ‘+1’ in the two superclasses (0,2,8,9) and (1,3,4,5,6,7), values close to ‘-1’ across these two superclasses. Such phenomena hint at the fundamental connection of adversarial robustness with weight clustering effect. That is, subclasses in the same superclass are positively related to each other, which is in contrast with those in different superclasses. This hierarchical classification property is more semantic with data sets, demonstrating that robust model has a better representative feature extraction capacity and thus outperforms standard model in various tasks.
Discussion In order to analyse the connection of weight clustering effect on linearized models with their original models, we specifically take the linear model as an example whose linearized model is itself. We additionally train two linear models with initial learning rate 0.01 and 100 epochs by standard training or PGD-10 adversarial training. The detailed architecture of the model is shown in Appendix F. The final accuracies are listed in Table 1. Though the model capacity restricts their accuracies, the robust model still shows a clustering effect. Fig. 3 indicates that the clustering effect of linear weight matrix generally holds even on a simplified linear model.

heading: 4 Explorations with Clustering Effect

text:
A simple and direct idea is to apply the linear weight clustering effect as a penalty. However, the time cost and computational cost are too expensive. Based on our understanding that robust models learn better class hierarchy of one data set and thus better semantic features, we further add the penalty in feature space by evaluating a class-wise feature distance matrix D of original non-linear DNNs on CIFAR-10. Given data pairs (xi, yi), their features zi, we compute the class-wise feature centers Zj for class j following Zj = mean zi · 1(yi = j) (8) F (a, b) represents the distance of class-wise feature centers Za and Zb, which is normalized with the largest class-wise distances for a better visualization.
F (a, b) = kZa Zbk2
max c,c 6=a
kZa Zck2 . (9)
Result Analysis We adopt the same basic setting as Sec. 3.2, and find a similar clustering effect on CIFAR-10 in Fig. 4. The models are the same with those in Fig. 2. Robust models consistently show the better clustering effect compared with non-robust models. That is, the centers of subclasses in the same superclass lie closer to each other in feature space. The feature clustering effect also aligns well with class hierarchy. Such effect further confirms our understanding that robust models could extract more semantic and hierarchical features even in their original non-linear feature space. Further Clustering Enhancement Given any unknown data set, we first extract its class hierarchy from a pre-trained robust model, e.g., trained by adversarial training. To be specific, we compute matrices W and C following Sec. 3.2. We then formulate C to an approximate matrix Cop with all items set to ‘-1’ or ‘+1’, i.e., in Cop, subclasses in the same superclass block have values ‘+1’, else have values ‘-1’. With the extracted class hierarchy, a regularization loss is designed to minimize outputs within the same superclass. Take the data set CIFAR-10 for example, subclasses are divided into (0,1,8,9) and (2,3,4,5,6,7) in Cop. The regularization loss is defined in Eq. 10. The hierarchical instance-wise clustering training strategy is proposed following Algorithm 2.
F0 = ExF(x)i · 1(i = 0, 1, 8, 9) F1 = ExF(x)j · 1(j = 2, 3, 4, 5, 6, 7) Lreg(x) = X
i=0,1,8,9
kF(x)i F0k2 + X
j=2,3,4,5,6,7
kF(x)j F1k2. (10)
Algorithm 2 Enhancing the Hierarchical Clustering Effect Input: A random input x, a pre-trained robust DNN model F (e.g., on CIFAR-10) Output: Robustness Enhanced Model F
Compute the weight correlation matrix C following Algorithm 1 and Eq. 7 Compute an approximate matrix Cop with all items ‘+1/-1’ based on C and extract class hierarchy Compute regularization loss Lreg(x) with feature clusters following Eq. 10 Retrain F from scratch using the original robust loss L(x, y) (e.g., adversarial training loss) added with Lreg return F

heading: 4.1 Domain Adaption Case

text:
Recent study shows that robust models tend to perform better in downstream tasks, especially domain adaption with subpopulation shift [29]. However, further analyses are insufficient. We study from two perspectives, 1) verifying how robust models outperform in domain adaption with different settings and 2) exploring a further improvement by enhancing the clustering effect. Specifically, we randomly divide a hierarchical data set into source data and target data according to their subclasses. That is, source and target data share the same superclass yet different subclasses, dubbed subpopulation shift. In the training phase, we use fine labels of source data and map fine labels to coarse ones. In such a fashion, the model returns fine and coarse labels. Only coarse label is evaluated in the test phase as the target test data has different fine labels from source data. Both coarse and fine accuracies are
evaluated on source domain. We enhance the feature clustering effect by a penalty loss defined in Eq. 10. To better utilize hierarchical features extracted by pre-trained models, we also finetune parameters on target data to achieve a further improvement. Different models, different constructed data sets and different adversarial training settings (i.e. adversarial robustness levels) are evaluated and analyzed.

heading: 4.1.1 Different Models

text: Datasets and Setups The data set CIFAR-100 is composed of 20 superclasses, and each superclass is composed of 5 subclasses. Based on that, we construct our own data sets by randomly sampling from CIFAR-100, e.g., our default CIFAR-20 is composed of 20 subclasses from 4 superclasses. The details of all constructed data sets are shown in Appendix A. We adversarially train ResNet-18, DenseNet-121 [17], and AlexNet [20] with PGD. If not specifically mentioned, PGD-5 is adopted (5 steps, step size 1/255, ✏=2/255). The robust/non-robust ResNet-18 and DenseNet-121 are trained with SGD [25] and a initial learning rate 0.1. The robust/non-robust AlexNet are trained with SGD and a initial learning rate 0.01. The learning rates decay with a factor of 0.1 at the 75 and 90 epoch. Other hyper-parameters are the same with Sec. 3.2. Moreover, we finetune the pre-trained models on target data for 20 epochs with a learning rate 0.01. Results Analysis We evaluate the models on our default CIFAR-20. As shown in Table 2, when comparing the coarse accuracies on target and source data, robust models tend to decline less than non-robust ones. Take pre-trained ResNet-18 for example, the coarse accuracy on robust model directly declines by 5.44% (from 87.94% to 82.50%) but declines by 10.18% (from 90.93% to 80.75%) on non-robust model. When considering finetuned models, the superiority of robust models outstands. For example, the coarse accuracy on finetuned robust ResNet-18 increases by 6.46% (from 87.94% to 94.40%) while increases by 3.32% (from 90.93% to 94.25%) on finetuned non-robust ResNet-18. When applying the clustering penalty defined in Eq. 10, coarse accuracy is improved by 2.25% on robust ResNet-18 (from 82.50% to 84.75%) even without finetuning. Experimental results demonstrate that features extracted by robust models have a non-trivial impact on domain adaptation. Such superiority keeps consistent when finetuning with the feature centers of target training data. Thus our understanding is confirmed that a better hierarchical classification learned by robust models benefits a lot in downstream domain adaption task.

heading: 4.1.2 Different Data Sets

text: Datasets and Setups For CIFAR-100, we randomly select 4 to 6 superclasses, select 4 subclasses from each superclass for training and apply the left one for test. We also select 5 superclasses, select 3 subclasses from each superclass for training and apply the left 2 subclasses for test. When selecting 4 superclasses and 4 subclasses, we typically design two extreme data set construction methods, i.e. the most different and similar data set. The ‘different’ 4 superclasses are selected as index 1, 2, 5, 14 (fish, flowers, household electrical devices, people). The ‘similar’ 4 superclasses are selected as index 8, 11, 12, 16, which are all animals. The reconstructed data set ImageNet-20 is composed of

heading: 4.1.3 Different Robustness Settings

text:
Datasets and Setups We evaluate ResNet-18 with different robustness settings (i.e. different PGDs in adversarial training) on random CIFAR-4?4 defined in Appendix A. The PGD details are given in Appendix B. Results Analysis As shown in Fig. 6, we could observe a trade-off between adversarial robustness and accuracies on target data. For example, in the polylines of both two subfigures, purple points show higher accuracies on target data than brown ones. However, brown points are of better robustness as their accuracies on source data are much lower. Royalblue points show the relatively bad performance
on target data yet the best performance on source data. The phenomena demonstrate a trade-off between robustness and domain adaption performances. The better performance on target domain could only be achieved with both robustness and moderate accuracies on source data. Furthermore, robust model with an enhanced clustering effect has significantly results than the original robust model. We show the specific numerical results in Appendix C. In addition, the time costs results with different robustness settings are shown in Appendix D.

heading: 4.2 Adversarial Case

text: We further conduct extensive experiments to explore such an intriguing clustering effect along with other adversarial phenomena. The experiments are designed from two perspectives, 1) adversarial robustness enhancement and 2) attack confusion matrix study. The results show that adversarial robustness could be improved by enhancing the hierarchical clustering effect, and attack success rates are highly related to class hierarchy as well.

heading: 4.2.1 Adversarial Robustness Enhancement

text: Datasets and Setups We adversarially train a robust ResNet-18 on CIFAR-10. The basic training setting is the same with Sec. 3.2. The PGD-20 attack is applied with ✏ = 8/255 and step size 0.003. Results Analysis Although the main exploration of clustering effect is studied not only in robustness improvement, we conduct comprehensive robustness evaluations following the state-of-the-art attack strategy with an adaptive AutoAttack [9]. We find that the clustering enhancement actually boosts adversarial robustness in Table 3 for both best and last checkpoints. For black-box attacks, the surrogate model is a standard trained VGG-16. NAttack is conducted on 1,000 random test data and with a maximum query 20,000. Note that, the improvement against AutoAttack is 1.31%, which shows that the clustering alignment with coarse labels learned by robust models could indeed boost adversarial robustness. An interesting phenomenon is that when enhancing clustering effect, both natural and robust accuracies are improved because the clustering effect improves feature representation. The natural accuracy by standard training improves as well (from 92.75% to 93.54%) with a clustering effect. Therefore, the trade-off between adversarial robustness and natural accuracy still exists. In addition, we count the training time of two models (AT and AT+C). The AT model costs 191.20s per epoch, while the AT+C model costs 193.66s, using one GPU 1080X with batch size 128 of ResNet-18 on CIFAR-10. The time cost slightly increases with the clustering regularization, which is negligible. More results are shown in Appendix E.

heading: 4.2.2 Attack Confusion Matrix

text: To further analyze how class hierarchy matters in adversary, we evaluate the attack confusion matrix M of adversarial examples in Fig. 7, whose shape is Doutput ⇥Doutput. M(i, j) is defined as attack success rate (ASR) of adversarial examples being attacked from the ground-truth label i to the target label j. Datasets and Setups We train robust and non-robust ResNet-18 on CIFAR-10. The basic setting is the same with Sec. 3.2. The PGD-20 attack is applied with ✏ = 8/255 and step size 0.003. For the target attack, target labels are set as the other 9 classes except for the ground-truth one. Results Analysis Experimental results show that the attack confusion matrix M performs in another blocking pattern especially on robust models, which is well-aligned with class hierarchy as well as the weight correlation matrix C and the feature distance matrix F . That is, the difficulty in attacking robust models shows distinguishing performance for subclasses in or out of the same superclass. For example, against untarget PGD-20 attack in Fig. 7(a), the ASRs on adversarially trained ResNet-18 show that classes in (2,3,4,5,6,7) are easier to be attacked from one to other. However, the ASRs could decline to almost zero from one superclass (0,1,8,9) to the other (2,3,4,5,6,7). Different superclasses are harder to attack especially on robust models, demonstrating that the hierarchical classification capacity is enhanced by adversarial robustness. Such phenomenon is also consistent with our hierarchical clustering observations on robust models.

heading: 5 Conclusions

text: In this paper, we provide a novel view of linear model exploration by extracting an implicit linear matrix expression. Then we surprisingly find an intriguing clustering effect on adversarially robust models, which is well-aligned with class hierarchy. Based on such observations, we first give an insightful explanation and understanding of robust models, which have a better capacity in extracting more representative features and semantic information. Extensive experiments show that enhancing the hierarchical clustering effect not only improves model robustness, but also benefits other downstream domain adaption task. Overall, our observations and findings motivate a deeper understanding on adversarial robustness.

heading: Acknowledgement

text: Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund. Shu-Tao Xia is supported in part by the National Key Research and Development Program of China under Grant 2018YFB1800204, the National Natural Science Foundation of China under Grant 62171248, the R&D Program of Shenzhen under Grant JCYJ20180508152204044.

title: Clustering Effect of (Linearized) Adversarial Robust Models


summaries: This paper develops a learning based approach for end-to-end reconstruction of operators for ill-posed problems. In particular it examines the problem of image reconstruction. The main contribution is to combine traditional variational approach with an optimal transport based regularizer.

summaries: This paper proposes a method for adversarial regularization, to be used in image reconstruction. This method relies on a joint training of the regularizing part of the system and the reconstruction. Several properties regarding convergence are proven, and the method is shown to outperform similar regularization methods for this task.

summaries: This paper proposes a deep learning based reconstruction for ill-posed inverse problems that combines ideas from end-to-end supervised training with learned regularization in a variational setting. The method adversarially trains the weights of a deep neural network (used as a prior for regularization) while also learning an end-to-end, unrolled deep network. The unrolled network takes the form of a learned primal-dual algorithm, and the regularizer aims to discern between ground-truth images and images generated by the unrolled network. After training, the end-to-end network can be used for reconstruction alone, or as an initialization for a separate reconstruction that strictly uses the learned regularizer in a variational setting. The authors claim that this combines the best of both worlds, as an end-to-end model can be used for fast reconstruction, while the variational model can be used for refinement (while also being more amenable to analysis). The authors provide theoretical and empirical evidence that their approach is successful and outperforms other state-of-the-art methods.

summaries: This paper proposes a framework for training an adversarial network, where the discriminator is a regularization functional, for image reconstruction.  A similar work mentioned in the paper is [14]. The key difference between [14] and the proposed method, UAR, is that the former is based on the variational inference while the latter replaces it with an unfolding network, making UAR a pure end-to-end training deep learning model. Since UAR incorporates the forward model in the loss function, it is self-supervised requiring only $y$ to impose supervision.  The author also addresses the theoretical properties of UAR, presenting interesting results on the optimality, influence of $\lambda$, and justification of end-to-end learning & regularization.

summaries: This paper proposes a framework, named UAR, for jointly training an iterative deep unfolding neural network (DUNN) and an adversarial discriminator based on Wasserstein distance for solving image inverse problems, inspired by adversarial regularization (AR) method (Lunz’2018). Unlike original AR method that takes pseudo-inverse reconstruction as input images, the proposed method assimilates the reconstruction from DUNN that unrolls the iterative primal-dual algorithm in each training step. The authors also provide theoretical analysis about when such adversarial training with unrolling networks would be successful. The theoretical results are well established by with thorough proofs. Finally, the performance of the proposed method is evaluated for solving CT inverse problem with satisfactory results compared to both supervised and unsupervised deep learning methods such as U-Net, LPD, and AR.

abstractText: We propose a new approach for learning end-to-end reconstruction operators based on unpaired training data for ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling and essentially seeks to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction and the ground-truth. More specifically, the regularizer in the variational setting is parametrized by a deep neural network and learned simultaneously with the unrolled reconstruction operator. The variational problem is then initialized with the output of the reconstruction network and solved iteratively till convergence. Notably, it takes significantly fewer iterations to converge as compared to variational methods, thanks to the excellent initialization obtained via the unrolled operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. Moreover, we demonstrate with the example of image reconstruction in X-ray computed tomography (CT) that our approach outperforms state-of-the-art unsupervised methods and that it outperforms or is at least on par with state-of-the-art supervised data-driven reconstruction approaches.

authors:
J. Adler
O. Öktem

title: Solving ill-posed inverse problems using iterative deep neural networks

venue: Inverse Problems,

title: Learned primal-dual reconstruction

venue: IEEE transactions on medical imaging,

authors:
J. Adler
H. Kohr
O. Öktem

title: Operator discretization library (ODL)

venue: Software available from https://github.com/odlgroup/odl,

authors:
M. Arjovsky
S. Chintala
L. Bottou

title: Wasserstein generative adversarial networks

venue: In Proceedings of the 34th International Conference on Machine Learning,

authors:
S. Arridge
P. Maass
O. Öktem
C.-B. Schönlieb

title: Solving inverse problems using datadriven models

venue: Acta Numerica,

authors:
A. Chambolle
T. Pock

title: A first-order primal-dual algorithm for convex problems with applications to imaging

venue: J. Math. Imaging and Vision,

authors:
S.H. Chan
X. Wang
O.A. Elgendy

title: Plug-and-play ADMM for image restoration: Fixedpoint convergence and applications

venue: IEEE Transactions on Computational Imaging,

authors:
E.K. Cole
J.M. Pauly
S.S. Vasanawala
F. Ong

title: Unsupervised MRI reconstruction with generative adversarial networks

venue: arXiv preprint arXiv:2008.13065,

authors:
M. Cuturi
G. Peyré

title: Computational Optimal Transport

venue: Arxiv preprint arXiv:1803.00567,

authors:
K. Gregor
Y. LeCun

title: Learning fast approximations of sparse coding

venue: In Intl. Conf. on Machine Learning,

authors:
K.H. Jin
M.T. McCann
E. Froustey
M. Unser

title: Deep convolutional neural network for inverse problems in imaging

venue: IEEE Transactions on Image Processing,

authors:
E. Kobler
T. Klatzer
K. Hammernik
T. Pock

title: Variational networks: connecting variational methods and deep learning

venue: In German conference on pattern recognition,

authors:
E. Kobler
A. Effland
K. Kunisch
T. Pock

title: Total deep variation for linear inverse problems

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
K. Lei
M. Mardani
J.M. Pauly
S.S. Vasanawala

title: Wasserstein GANs for MR Imaging: From Paired to Unpaired Training

venue: IEEE Transactions on Medical Imaging,

authors:
H. Li
J. Schwab
S. Antholzer
M. Haltmeier

title: NETT: solving inverse problems with deep neural networks

authors:
J. Liu
Y. Sun
C. Eldeniz
W. Gan
H. An
U.S. Kamilov

title: Rare: Image reconstruction using deep priors learned without groundtruth

venue: IEEE Journal of Selected Topics in Signal Processing,

authors:
S. Lunz
O. Öktem
C.-B. Schönlieb

title: Adversarial regularizers in inverse problems

venue: In Advances in Neural Information Processing Systems,

title: TU-FG-207A-04: Overview of the Low Dose CT Grand Challenge

venue: Medical Physics,

authors:
T. Meinhardt
M. Moller
C. Hazirbas
D. Cremers

title: Learning proximal operators: Using denoising networks for regularizing inverse imaging problems

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
V. Monga
Y. Li
Y. Eldar

title: Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing

venue: arXiv preprint arXiv:1912.10557v3,

authors:
S. Mukherjee
S. Dittmer
Z. Shumaylov
S. Lunz
O. Öktem
C.-B

title: Schönlieb. Learned convex regularizers for inverse problems

venue: arXiv preprint arXiv:2008.02839v2,

authors:
C. Oh
D. Kim
J.-Y. Chung
Y. Han
H. Park

title: ETER-net: End to End MR image reconstruction using recurrent neural network

authors:
J.-C. Pesquet
A. Repetti
M. Terris
Y. Wiaux

title: Learning maximally monotone operators for image recovery

venue: arXiv preprint arXiv:2012.13247v2,

authors:
E.T. Reehorst
P. Schniter

title: Regularization by denoising: clarifications and new interpretations

authors:
Y. Romano
M. Elad
P. Milanfar

title: The little engine that could: Regularization by denoising (RED)

venue: SIAM Journal on Imaging Sciences,

authors:
E. Ryu
J. Liu
S. Wang
X. Chen
Z. Wang
W. Yin

title: Plug-and-play methods provably converge with properly trained denoisers

venue: In International Conference on Machine Learning,

title: Optimal Transport for Applied Mathematicians

venue: Birkhäuser Basel,

authors:
O. Scherzer
M. Grasmair
H. Grossauer
M. Haltmeier
F. Lenzen

title: Variational methods in imaging

authors:
O. Senouf
S. Vedula
T. Weiss
A.M. Bronstein
O.V. Michailovich
M. Zibulevsky

title: Self-supervised learning of inverse problem solvers in medical imaging. In Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data

venue: First MICCAI Workshop, DART 2019,

authors:
B. Sim
G. Oh
J. Kim
C. Jung
J.C. Ye

title: Optimal transport driven CycleGAN for unsupervised learning in inverse problems

authors:
Y. Sun
B. Wohlberg
U.S. Kamilov

title: An online plug-and-play algorithm for regularized image reconstruction

authors:
J.I. Tamir
S.X. Yu
M. Lustig

title: Unsupervised deep basis pursuit: Learning inverse problems without ground-truth data

authors:
D. Ulyanov
A. Vedaldi
V. Lempitsky

title: Deep image prior

title: Optimal transport — Old and new, volume 338 of Grundlehren der mathematischen Wissenschaften

authors:
Z. Wang
A.C. Bovik
H.R. Sheikh
E.P. Simoncelli

title: Image quality assessment: From error visibility to structural similarity

authors:
Z. Wu
Y. Sun
A. Matlock
J. Liu
L. Tian
U.S. Kamilov

title: Simba: Scalable inversion in optical tomography using deep denoising priors

authors:
B. Yaman
S.A.H. Hosseini
S. Moeller
J. Ellermann
K. Uğurbil
M. Akçakaya

title: Selfsupervised learning of physics-guided reconstruction neural networks without fully sampled reference data

venue: Magnetic resonance in medicine,

authors:
Y. Yang
J. Sun
H. Li
Z. Xu

title: Deep ADMM-Net for compressive sensing MRI

authors:
K. Zhang
L.V. Gool
R. Timofte

title: Deep unfolding network for image super-resolution

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
B. Zhu
J.Z. Liu
S.F. Cauley
B.R. Rosen
M.S. Rosen

title: Image reconstruction by domain-transform manifold learning

heading: 1 Introduction

text:
Inverse problems are ubiquitous in imaging applications, wherein one seeks to recover an unknown model parameter x ∈ X from its incomplete and noisy measurement, given by
yδ = A(x) + e ∈ Y. Here, the forward operator A : X→ Y models the measurement process in the absence of noise, and e, with ‖e‖2 ≤ δ, denotes the measurement noise. For example, in computed tomography (CT), the forward operator computes line integrals of x over a predetermined set of lines in R3 and the goal is to reconstruct x from its projections along these lines. Without any further information about x, inverse problems are typically ill-posed, meaning that there could be several possible reconstructions that are consistent with the measured data, even without any noise.
The variational framework circumvents ill-posedness by encoding prior knowledge about x via a regularization functionalR : X→ R. In the variational setting, one solves
min x∈X LY(yδ,A(x)) + λR(x), (1)
where LY : Y× Y→ R+ measures data-fidelity andR penalizes undesirable or unlikely solutions. The penalty λ > 0 balances the regularization strength with the fidelity of the reconstruction. The variational problem (1) is said to be well-posed if it has a unique solution varying continuously in yδ .
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
The success of deep learning in recent years has led to a surge of data-driven approaches for solving inverse problems [5], especially in imaging applications. These methods come broadly in two flavors: (i) end-to-end trained models that aim to directly map the measurement or a model-based noisy reconstruction to the corresponding true parameter [11, 2] and (ii) learned regularization methods that seek to find a data-adaptive regularizer instead of handcrafting it [15, 17, 13]. Techniques in both categories have their relative advantages and demerits. Specifically, end-to-end approaches offer fast reconstruction of astounding quality, but lack in terms of theoretical guarantees and need supervised data (i.e., pairs of input and target images) for training. On the contrary, learned regularization methods inherit the provable well-posedness properties of the variational setting and can be trained on unpaired training data, however the reconstruction entails solving a high-dimensional optimization problem, which is often slow and computationally demanding.
Our work derives ideas from learned optimization and adversarial machine learning, and makes an attempt to combine the best features of both aforementioned paradigms. In particular, the proposed method offers the flexibility of training on unpaired samples, produces fast reconstructions comparable to end-to-end supervised methods in quality, while enjoying the well-posedness and stability guarantees of the learned regularization framework. We first provide an overview of the recent literature on data-driven solutions for inverse problems before detailing our contributions.

heading: 1.1 Related works

text:
End-to-end learned methods for inverse problems either map the measurement directly to the clean image [40, 22], or learn to eliminate artifacts from a reconstruction produced by a model-based technique [11]. Such approaches are data-intensive and may generalize poorly when trained on limited data. Iterative unrolling [20, 38, 1, 19, 12], with its origin in the seminal work by Gregor and LeCun on data-driven sparse coding [10], employs reconstruction networks that are inspired by optimization-based approaches and hence are interpretable. The unrolling paradigm enables one to encode the knowledge about the acquisition physics into the network architecture [2], thereby achieving data-efficiency. Nevertheless, end-to-end trained methods are supervised, and it is often challenging to obtain a large ensemble of paired data, especially in medical imaging applications.
Data-driven regularization methods aim to learn a regularizer in the variational setting instead of handcrafting it. Some notable approaches in this paradigm include adversarial regularization (AR) [17] and its convex counterpart [21], network Tikhonov (NETT) [15], total deep variation (TDV) [13], etc., wherein one explicitly parametrizes the regularization functional using a neural network. The regularization by denoising (RED) and more general Plug-and-play (PnP) approaches aim to solve inverse problems by using a pre-trained denoiser inside an algorithm for minimizing the variational objective [24, 25, 39, 36, 26, 16, 7, 31]. Such methods achieve remarkable performances for various inverse problems when equipped with sophisticated pre-trained denoisers [39, 36, 26, 16, 7], and are essentially equivalent to data-driven regularization, subject to additional constraints on the denoiser [23]. The deep image prior technique [33] is training-free, but it regularizes the solution by restricting it to be in the range of a generator and can thus be interpreted as a deep learning-based regularization scheme. It is relatively easier to analyze learned regularization schemes using classical functional analysis [28], but they tend to perform worse than end-to-end supervised methods. Moreover, these methods require solving a high-dimensional, potentially non-convex, variational optimization, leading to slow reconstruction with no provably convergent optimization algorithm in general.
Data-driven methods for inverse problems can be classified into different categories based on the training protocol. Supervised approaches need data in the form of pairs of noisy measurements (or some model-based reconstructions) and target images (or ground-truths) [2, 40, 11], while fully unsupervised methods require access to only the noisy measurements for training [29, 32, 8, 37]. Our approach lies in between these two extremes, as it requires samples from the marginal distributions of the measurements and the ground-truth images, but not from their joint distribution. Recently, such unpaired training approaches have been developed in [30, 14] based on the optimal transport theory. The training objective in [14] only seeks to match the ground-truth distribution via a Wasserstein generative adversarial network (GAN) without any data-consistency, whereas [30] shows that the cycle-GAN loss between the image and the measurement spaces is an upper bound on the optimal transport loss with a penalized least squares objective as the transportation cost. Unlike [30] and [14], our approach minimizes a fidelity term in the data space alone and not in the image space (together with an optimal transport-based loss), and allows for a more straightforward theoretical analysis using the standard variational framework.

heading: 1.2 Specific contributions

text: Our work seeks to combine iterative unrolling with data-adaptive regularization via an adversarial learning framework, and hence is referred to as unrolled adversarial regularization (UAR). The proposed approach learns a data-adaptive regularizer parametrized by a deep neural network, along with an iteratively unrolled reconstruction network that minimizes the corresponding expected variational loss in an adversarial setting. Unlike AR [17] where the undesirable images are taken as the pseudo-inverse reconstruction and kept fixed throughout the training, we update them with the output of the unrolled reconstruction network in each training step, and, in turn, use them to further improve the regularizer. Thanks to the Kantorovich-Rubinstein (KR) duality [4], the alternating learning strategy of the reconstruction and the regularizer networks is equivalent to minimizing the expected data-fidelity over the distribution of the measurement, penalized by the Wasserstein-1 distance between the distribution of the reconstruction and the ground-truth. Once trained, the iteratively unrolled operator produces a fast end-to-end reconstruction. We show that this efficient reconstruction can be improved further by a refinement step that involves minimizing the variational loss with the corresponding regularizer, starting from this initial estimate. The refinement step not only produces reconstructions that are superior to state-of-the-art unsupervised methods and are competitive with supervised methods, but also facilitates well-posedness and stability analyses akin to classical variational approaches [28]. Our theoretical results on the learned unrolled operator and the regularizer are corroborated by strong experimental evidence for the CT inverse problem, and the illustrative inpainting and denoising examples (see Sec. B in the supplementary document). The unpaired training framework for UAR is particularly useful for medical imaging applications, where it is in general challenging to collect a large corpus of paired noisy input and reference images.

heading: 2 The proposed unrolled adversarial regularization (UAR) approach

text: In this section, we give a brief mathematical background on optimal transport, followed by a detailed description of the UAR framework, including the training protocol and the network architectures.

heading: 2.1 Background on Optimal transport

text:
Optimal transport theory [9, 34] has recently gained prominence in the context of measuring the distance between two probability measures. In particular, given two probability measures π1 and π2 on Rn, the Wasserstein-1 distance between them is defined as
W1(π1, π2) := inf µ∈Π(π1,π2)
∫ ‖x1 − x2‖2 dµ(x1,x2), (2)
where Π(π1, π2) denotes all transport plans having π1 and π2 as marginals. The Wasserstein distance has proven to be suitable for deep learning tasks, when the data is assumed to be concentrated on low-dimensional manifolds in Rn. It has been shown that in such cases, the Wasserstein distance provides a usable gradient during training [4], as opposed to other popular divergence metrics.
By the KR duality, the Wasserstein-1 distance can be computed equivalently by solving a maximization problem over the space of 1-Lipschitz functions (denoted by L1) as
W1(π1, π2) = sup R∈L1
∫ R(x1) dπ1(x1)− ∫ R(x2) dπ2(x2), (3)
provided that π1 and π2 have compact support [27]. Finally, we recall the definition of push-forward of probability measures, which is used extensively in our theoretical exposition. Given a probability measure π on A and a map T : A → B, the push-forward of π by T (denoted as T#π) is defined as a probability measure on B such that T#π(B) = π(T−1(B)), for all measurable B ⊂ B.

heading: 2.2 Training strategy and network parametrization for UAR

text:
The principal idea behind UAR is to learn an unrolled deep network Gφ : Y→ X for reconstruction, together with a regularization functionalRθ : X→ R parametrized by another convolutional neural network (CNN). The role of Rθ is to discern ground-truth images from images produced by Gφ, while Gφ learns to minimize the variational loss withRθ as the regularizer. As the images produced
Algorithm 1 Learning unrolled adversarial regularization (UAR). 1. Input: Training data-set {xi}Ni=1 ∼ πx and { yj }N j=1 ∼ πyδ , initial reconstruction network
parameter φ and regularizer parameter θ, batch-size nb = 1, penalty λ = 0.1, gradient penalty λgp = 10.0, Adam optimizer parameters (β1, β2) = (0.50, 0.99). 2. Learn a baseline regularizer: • for training steps k = 1, 2, · · · , 10 Nnb , do: (i.e., # epochs = 10, # mini-batches = N nb
) – Sample xj ∼ πx, yj ∼ πyδ , and j ∼ uniform [0, 1]; for 1 ≤ j ≤ nb. Compute uj = A†(yj) and x ( ) j = jxj + (1− j)uj . – θ ← Adamη,β1,β2(θ,∇ J̃1(θ)), where η = 10−4, and
J̃1(θ) = 1
nb nb∑ j=1 [ Rθ (xj)−Rθ (uj) + λgp (∥∥∥∇Rθ (x( )j )∥∥∥ 2 − 1 )2] .
3. Learn a baseline reconstruction operator: (with 5 epochs, Nnb batches per epoch) • for training steps k = 1, 2, · · · , 5 Nnb , do:
– Sample yj ∼ πyδ , and compute J̃2(φ) = 1nb ∑nb j=1 ∥∥yj −A (Gφ(yj))∥∥22 +λRθ (Gφ(yj)). – φ← Adamη,β1,β2(φ,∇ J̃2(φ)), with step-size η = 10−4.
4. Jointly trainRθ and Gφ adversarially: (over 25 epochs, with Nnb batches in each epoch) • for k = 1, 2, · · · , 25 Nnb , do:
– Sample xj , yj , and j ∼ uniform [0, 1]; for 1 ≤ j ≤ nb. Compute uj = Gφ(yj) and x
( ) j = jxj + (1− j)uj .
– θ ← Adamη,β1,β2(θ,∇ J̃1(θ)), where J̃1(θ) is as in Step 2, with η = 2× 10−5. – Update φ← Adamη,β1,β2(φ,∇ J̃2(φ)) twice, with J̃2(φ) as in Step 3, and η = 2× 10−5.
5. Output: The trained networks Gφ andRθ.
by Gφ get better, Rθ faces a progressively harder task of telling them apart from the ground-truth images, thus leading to an improved regularizer. On the other hand, as the regularizer improves, the quality of reconstructions obtained using Gφ improves as a consequence. Thus, Gφ andRθ help each other improve as the training progresses via an alternating update scheme.

heading: 2.2.1 Adversarial training

text:
Let us denote by πx the ground-truth distribution and by πyδ the distribution of the noisy measurement. The UAR approach seeks to learn Gφ andRθ simultaneously starting from an appropriate initialization. At the kth iteration of training, the parameters φ of the reconstruction network are updated as
φk ∈ arg min φ J (1) k (φ), where J (1) k (φ) := Eπyδ [∥∥A(Gφ(yδ))− yδ∥∥22 + λRθk(Gφ(yδ))] , (4) for a fixed regularizer parameter θk. Subsequently, the regularizer parameters are updated as
θk+1 ∈ arg max θ:Rθ∈L1 J (2) k (θ), where J (2) k (θ) := Eπyδ
[ Rθ ( Gφk(yδ) )] − Eπx [Rθ (x)] . (5)
The learning protocol for UAR only requires unpaired samples of x and yδ , since the loss functionals J
(1) k (φ) and J (2) k (θ) can be computed based solely on the marginals πx and πyδ . The alternating update algorithm in (4) and (5) essentially seeks to solve the min-max variational problem given by
min φ max θ:Rθ∈L1 Eπ yδ ∥∥A(Gφ(yδ))− yδ∥∥22 + λ(Eπyδ [Rθ (Gφ(yδ))]− Eπx [Rθ (x)]) . (6) Thanks to KR duality in (3) and the definition of push-forward, (6) can be reformulated as
min φ Eπ yδ ∥∥A(Gφ(yδ))− yδ∥∥22 + λW1 ((Gφ)#πyδ , πx) . (7)
We refer the reader to Section 3 for a mathematically rigorous statement of this equivalence as well as for a well-posedness theory of the problem in (7). Note that the equivalence of the alternating minimization procedure and the variational problem in (7) holds only if the regularizer is fully optimized in every iteration. Nevertheless, in practice, the reconstruction and regularizer networks are not fully optimized in every iteration. Instead, one refines the parameters by performing one (or a few) Adam updates on the corresponding loss functionals. Notably, if W1 ( (Gφk)#πyδ , πx ) = 0, i.e., the parameters of G are such that the reconstructed images match the ground-truth in distribution, the loss functional J (2)k (θ) and its gradient vanish, leading to no further update of θ. Thus, both networks stop updating when the outputs of Gφ are indistinguishable from the ground-truth images. The concrete training steps are listed in Algorithm 11.

heading: 2.2.2 Variational regularization as a refinement step

text:
The unrolled operator Gφ∗ trained by solving the min-max problem in (6) provides reasonably good reconstruction when evaluated on X-ray CT, and already outperforms state-of-the-art unsupervised methods (c.f. Section 4). We demonstrate that the regularizer Rθ∗ obtained together with Gφ∗ by solving (6) can be used in the variational framework to further improve the quality of the end-to-end reconstruction Gφ∗(yδ) for a given yδ ∈ Y. Specifically, we solve the variational problem
min x∈X ‖A(x)− yδ‖2 + λ′
( Rθ∗(x) + σ‖x‖22 ) , (8)
where λ′, σ ≥ 0, by applying gradient descent, initialized with Gφ∗(yδ). The additional Tikhonov term in (8) ensures coercivity of the overall regularizer, making it amenable to the standard wellposedness analysis [28]. Practically, it improves the stability of the gradient descent optimizer for (8). Nevertheless, one essentially gets the same reconstruction with σ = 0 subject to early stopping (100 iterations). Notably, the fidelity term in (8) is the `2 distance, instead of the squared-`2 fidelity. We have empirically observed that this choice of the fidelity term improves the quality of the reconstruction, possibly due to the higher gradient of the objective in the initial solution Gφ∗(yδ). Since the end-to-end reconstruction gives an excellent initial point, it takes significantly fewer iterations for gradient-descent to recover the optimal solution to (8), and therefore UAR retains its edge in reconstruction time over fully variational approaches with learned regularizers (e.g., AR [17] or its convex version [21]).

heading: 2.2.3 Architectures of the iteratively unrolled generator and the deep regularizer

text:
The objective of Gφ is to approximate the minimizer of the variational loss withRθ as the regularizer. Therefore, an iterative unrolling strategy akin to [2] is adopted for parametrizing Gφ. Iterative unrolling aims to approximate the variational minimizer via a primal-dual-style algorithm [6], with the proximal operators in the image and measurement spaces replaced with trainable CNNs. Although the variational loss in our case is non-convex, this parametrization for Gφ is chosen because of its expressive power over a generic network. Initialized with x(0) = A† yδ and h(0) = 0, Gφ produces a reconstruction x(L) by iteratively applying the CNNs Λ
φ (`) p and Γ φ (`) d in X and Y, respectively:
h(`+1) = Γ φ (`) d
( h(`), σ(`)A(x`),yδ ) , andx(`+1) = Λ
φ (`) p
( x(`), τ (`)A∗(h`+1) ) , 0 ≤ ` ≤ L−1.
The step-size parameters σ(`) and τ (`) are also made learnable and initialized as σ(`) = τ (`) = 0.01 for each layer `. All learnable parameters in the generator are denoted by the shorthand notation φ for brevity. The number of layers L is typically much smaller (we take L = 20) than the number of iterations needed by an iterative primal-dual scheme to converge, thus expediting the reconstruction by two orders of magnitude (as compared to variational methods) once trained.
The regularizerRθ is taken as a deep CNN with six convolutional layers, followed by one averagepooling and two dense layers in the end.

heading: 3 Theoretical results

text:
The theoretical properties of UAR are stated in this section and their proofs are provided in the supplementary document. Throughout this section, we assume that X = Rn and Y = Rk, and
1Codes at https://github.com/Subhadip-1/unrolling_meets_data_driven_regularization.
A1. πx is compactly supported and πyδ is supported on a compact set K ⊂ Rk for every δ ≥ 0.
We then consider the following problem:
inf φ sup R∈L1
J1 ( Gφ,R|λ, πyδ ) := Eπ
yδ ∥∥yδ −AGφ(yδ)∥∥22 + λ (Eπyδ [R(Gφ(yδ))]− Eπx [R(x)]) . (9)
Problem (9) is identical to the min-max variational problem defined in (6), with the only difference that the maximization in R is performed over the space of all 1-Lipschitz functions. Basically, we consider the theoretical limiting case where the neural networks Rθ are expressive enough to approximate all functions in L1 with arbitrary accuracy. We make the following assumptions on Gφ:
A2. Gφ is parametrized over a finite dimensional compact set K, i.e. φ ∈ K. A3. Gφn → Gφ pointwise whenever φn → φ. A4. supφ∈K ‖Gφ‖∞ <∞.
Assumptions A2-A4 are satisfied, for instance, when Gφ is parametrized by a neural network whose weights are kept bounded during training. These assumptions apply to all results in this section.

heading: 3.1 Well-posedness of the adversarial loss and noise stability

text:
Here, we prove well-posedness and stability to noise for the optimal reconstructions. As a consequence of the KR duality, (9) can be equivalently expressed as
inf φ J2 ( Gφ|λ, πyδ ) := Eπ yδ ∥∥yδ −AGφ(yδ)∥∥22 + λW1(πx, (Gφ)#πyδ) . (10) In the next theorem, we state this equivalence, showing the existence of an optimal Gφ andR for (9). Theorem 1. Problems (9) and (10) admit an optimal solution and
inf φ sup R∈L1
J1 ( Gφ,R|λ, πyδ ) = inf φ J2 ( Gφ|λ, πyδ ) . (11)
Moreover, if (Gφ∗ ,R∗) is optimal for (9), then Gφ∗ is optimal for (10). Conversely, if Gφ∗ is optimal for (10), then (Gφ∗ ,R∗) is optimal for (9), for allR∗ ∈ arg maxR∈L1 Eπyδ [ R(Gφ∗(yδ)) ] −Eπx [R(x)].
Next, we study the stability of the optimal reconstruction Gφ∗ to noise. We consider Gφn , where
φn ∈ arg inf φ
J2 ( Gφ|λ, πyδn ) , (12)
and show that Gφn → Gφ∗ as δn → δ, thus establishing noise-stability of the unrolled reconstruction. Theorem 2 (Stability to noise). Suppose, for given a sequence of noise levels δn → δ ∈ [0,∞), it holds that πyδn → πyδ in total variation. Then, with φn as in (12), Gφn → Gφ∗ up to sub-sequences.

heading: 3.2 Effect of λ on the end-to-end reconstruction

text:
In order to analyze the effect of the parameter λ in (10) on the resulting reconstruction Gφ∗ , it is convenient to introduce the following two sets:
ΦL := { φ : Eπ
yδ ∥∥yδ −AGφ(yδ)∥∥22 = 0} and ΦW := {φ : (Gφ)#πyδ = πx} . We assume that both ΦL and ΦW are non-empty, which is tantamount to asking that the parametrization of the end-to-end reconstruction operator is expressive enough to approximate a right inverse of A (ΦL 6= ∅) and a transport map from πyδ to πx (ΦW 6= ∅), and therefore is not very restrictive (keeping in view the excellent approximation power of unrolled deep architectures). Proposition 1. Let Gφ∗ be a minimizer for (10). Then, it holds that
• Eπy ∥∥yδ −AGφ∗(yδ)∥∥22 ≤ λW1(πx, (Gφ)#πyδ), for every φ ∈ ΦL.
• W1(πx, (Gφ∗)#πyδ) ≤ 1
λ Eπ yδ
∥∥yδ −AGφ(yδ)∥∥22 , for every φ ∈ ΦW.
The previous proposition shows in a quantitative way that for small λ, the optimal Gφ∗ has less expected distortion in the measurement space as the quantity Eπ
yδ ∥∥yδ −AGφ∗(yδ)∥∥22 is small. On the other hand, if λ is large, then the optimal Gφ∗ maps πyδ closer to the ground-truth distribution πx; as the quantity W(πx, (Gφ∗)#πyδ) is small. Therefore, the regularization is stronger in this case. We extend this analysis by studying the behavior of the unrolled reconstruction as λ converges to 0 and to +∞. Consider a sequence of parameters λn > 0 and the minimizer of the objective in (10) with parameter λn:
φ′n ∈ arg inf φ
J2 ( Gφ|λn, πyδ ) . (13)
Theorem 3. Let λn → 0. Then, there exists φ∗1 ∈ arg min φ∈ΦL W1(πx, (Gφ)#πyδ) such that Gφ′n → Gφ∗1 up to sub-sequences, and lim
n→∞ 1 λn inf φ J2 ( Gφ|λn, πyδ ) = W1(πx, (Gφ∗1 )#πyδ).
Theorem 4. Let λn → +∞. Then, there exists φ∗2 ∈ arg min φ∈ΦW Eπ yδ ∥∥yδ −AGφ(yδ)∥∥22 such that Gφ′n → Gφ∗2 up to sub-sequences, and limn→∞ infφ J2 ( Gφ|λn, πyδ ) = Eπ yδ
∥∥yδ −AGφ∗2 (yδ)∥∥22. Theorems 3 and 4 characterize the optimal end-to-end reconstruction Gφ∗ as λ → 0 and λ → ∞, respectively. Specifically, if λ→ 0, Gφ∗ minimizes the Wasserstein distance between reconstruction and ground-truth among all the reconstruction operators that achieve zero expected data-distortion. In particular, Gφ∗ is close to the right inverse of A that minimizes the Wasserstein distance. Therefore, when λ is very small, we expect to obtain a reconstruction that is close to the unregularized solution in quality. If λ → ∞ on the other hand, the operator Gφ∗ is close to a transport map between πx and πyδ , i.e., (Gφ∗)#πyδ = πx, which minimizes the expected data-distortion. Therefore the reconstruction produces realistic images, but they are not consistent with the measurement. These theoretical observations are corroborated by the numerical results (c.f. Section 4, Fig. 2). One has to thus select a λ that optimally trades-off data-distortion with the Wasserstein distance to achieve the best reconstruction performance.

heading: 3.3 End-to-end reconstruction vis-à-vis the variational solution

text:
The goal of this section is two-fold. Firstly, we theoretically justify the fact that the end-to-end reconstruction performs well, despite minimizing the expected loss over the distribution πyδ . Secondly, we analyze the role of the regularizer in the variational setting in refining the end-to-end reconstruction.
It is important to remark that the the end-to-end reconstruction is trained with respect to the average variational loss computed using samples from πyδ and πx. Therefore, the end-to-end reconstruction cannot learn a point-wise correspondence between measurement and model parameter, but only a distributional correspondence. Despite that, the end-to-end reconstruction achieves excellent performance for a given measurement yδ. A justification of this phenomenon is given formally by the next proposition.
Proposition 2. Let (Gφ∗ ,R∗) be an optimal pair for (9) such thatR∗ ≥ 0 almost everywhere under (Gφ)# πyδ . Define M1 := Eπyδ ∥∥yδ −AGφ∗(yδ)∥∥22 and M2 := W1(πx, (Gφ∗)#πyδ). Then, the following two upper bounds hold for every η > 0:
• Pπ yδ
{ yδ : ∥∥yδ −AGφ∗(yδ)∥∥22 ≥ η} ≤ M1η . • Suppose,R∗(x) = 0 for πx-almost every x. Then, P(Gφ∗ )#πyδ { x : R∗(x) ≥ η } ≤ M2η .
Proposition 2 provides an estimate in probability of the sets {yδ : ∥∥yδ −AGφ∗(yδ)∥∥22 ≥ η} and
{x : R∗(x) ≥ η}. In particular, if M1 is small, then ∥∥yδ −AGφ∗(yδ)∥∥22 is small in probability. If instead M2 is small, thenR∗(x) is small in probability on the support of (Gφ∗)#πyδ , implying that samples Gφ∗(yδ) are difficult to distinguish from the ground-truth. We remark that the assumption R∗(x) = 0 can be justified using a data manifold assumption as in Section 3.3. of [17]. We now analyze the role of the regularizer R∗ in the optimization of the variational problem (8) that refines the end-to-end reconstruction Gφ∗ . We rely on a similar distributional analysis as the one performed in [17]. For η > 0, consider the transformation by a gradient-descent step onR∗ given by gη(x) = x− η∇R∗(x). Using the shorthand πG∗ := (Gφ∗)#πyδ , and by denoting the distribution of gη(x) as πη := (gη)#πG∗ for x ∼ πG∗ , we have the following theorem. Theorem 5 ([17]). Suppose that η →W1(πη, πx) is differentiable at η = 0. Then, the derivative at η = 0 satisfies ddηW1(πη, πx) ∣∣∣ η=0 = −EπG∗ ‖∇R∗(x)‖22.
This theorem states that a gradient-descent step performed on R∗ at x = Gφ∗(yδ) decreases the Wasserstein distance with respect to the ground-truth distribution πx. Therefore, if the gradientdescent step to solve the variational problem (8) is initialized with the reconstruction Gφ∗(yδ), the next iterate gets pushed closer to the ground-truth distribution πx. We stress that this property holds because of the chosen initialization point, due to the relation between R∗ and Gφ∗ . For a different initialization, this property may not hold.

heading: 4 Numerical results

text:
On the application front, we consider the prototypical inverse problem of CT reconstruction from noisy sparse-view projections. The abdominal CT scans for 10 patients, made publicly available by the Mayo-Clinic for the low-dose CT grand challenge [18], were used in our numerical experiments. Specifically, 2250 2D slices of size 512 × 512 corresponding to 9 patients were used to train the models, while 128 slices from the remaining one patient were used for evaluation. The projections were simulated in ODL [3] using a parallel-beam geometry with 200 uniformly spaced angular positions of the source, with 400 lines per angle. Subsequently, Gaussian noise with standard deviation σe = 2.0 was added to the projection data to simulate noisy sinograms. Notably, the CT projection data are typically corrupted by Poisson noise, but the data-likelihood for Poisson noise is not the `22 distance and is therefore not amenable to the theoretical analysis presented in Sec. 3. Hence, we simulate the noisy projection data by adding Gaussian noise to demonstrate a proof-of-concept for the UAR method. Extending the theoretical results to deal with Poisson noise in the data remains to be undertaken as a future work. Additionally, since the UAR approach seeks to
minimize a variational loss, which includes a data-fidelity term arising from the noise distribution, one needs to systematically investigate the robustness of UAR to the change in noise statistics to make it fully applicable to CT reconstruction from real low-dose projection data.
The proposed UAR method is compared with two classical model-based approaches for CT, namely filtered back-projection (FBP) and total variation (TV). The LPD method [2] and U-net-based postprocessing [11] of FBP are chosen as two supervised approaches for comparison. The AR approach [17] and its convex variant [21], referred to as adversarial convex regularizer (ACR), are taken as the competing unpaired training approaches. For LPD and AR, we develop a PyTorch-based implementation based on their publicly available TensorFlow codes2 3, while for ACR, we use the publicly available PyTorch implementation4.
The unrolled network Gφ has 20 layers, with 5×5 filters in both primal and dual spaces to increase the overall receptive field for sparse-view measurements. The hyper-parameters involved in training the
2LPD: https://github.com/adler-j/learned_primal_dual. 3AR: https://github.com/lunz-s/DeepAdverserialRegulariser. 4ACR: https://github.com/Subhadip-1/data_driven_convex_regularization.
UAR are specified in Algorithm 1. We found that training a baseline regularizer and a corresponding baseline reconstruction operator stabilized the training process. Training UAR took approximately three hours per epoch on an NVIDIA Quadro RTX 6000 GPU (with 24 GB of memory).
The average performance on the test images in terms of PSNR and SSIM [35] indicates that UAR (with λ = 0.1 and no refinement) outperforms AR and ACR by approximately 0.5 dB and 2.8 dB, respectively (see Table 1). We would like to emphasize that this gain was found to be consistent across all test images and not just realized on average. With the refinement step, UAR surpasses AR by almost 1 dB and slightly surpasses the U-net-based post-processing. The end-to-end UAR reconstruction is a couple of orders of magnitude faster than AR, while the reduction in reconstruction time is by a factor of four with the refinement. Since the refinement step entails running a few gradient-descent iterations on a high-dimensional variational loss, it makes UAR slower than supervised end-to-end methods. However, thanks to a superior initialization provided by the generator, one requires significantly fewer gradient-descent iterations to refine as compared to a fully variational scheme such as AR. The reconstructions of a representative test image using the competing methods are shown in Fig. 1 for a visual comparison. The effect of λ on the reconstruction of UAR is demonstrated in Fig. 2, which confirms the theoretical results in Section 3.2. The refinement step also visibly improves the reconstruction quality of the end-to-end operator, as shown in Fig. 3.
Some additional numerical examples for the task of CT reconstruction, and some illustrative examples of the performance of UAR on two other important imaging inverse problems, namely inpainting and denoising, are provided in Sec. B of the supplementary document.

heading: 5 Conclusions and outlook

text: To the best of our knowledge, this work makes the first attempt to blend end-to-end reconstruction with data-driven regularization via an adversarial learning framework. The proposed UAR approach retains the fast reconstruction of the former together with provable guarantees of the latter. We rigorously analyze the proposed framework in terms of well-posedness, noise-stability, and the effect of the regularization penalty, and establish a link between the trained reconstruction operator and the corresponding variational objective. We show strong numerical evidence of the efficacy of the UAR approach for CT reconstruction, wherein it achieves the same performance as supervised data-driven post-processing and outperforms competing unsupervised techniques. Our work paves the way to better understand the role of adversarially learned regularizers in solving ill-posed inverse problems, although several important aspects need further investigation. Since the learned regularizer is non-convex, the performance of gradient-descent on the variational objective greatly depends on initialization. This problem is partly addressed by the unrolled reconstruction operator that efficiently computes a better initial point for gradient descent. However, the precise relationship between the end-to-end reconstruction and the variational minimizer for a given measurement vector remains elusive. Moreover, the quality of the reconstruction relies on the expressive power of neural networks and thus suffers from the curse of dimensionality. We believe that addressing such limitations will be important to better understand adversarial regularization methods.

heading: 6 Acknowledgments

text:
MC acknowledges support from the Royal Society (Newton International Fellowship NIF\R1\192048 Minimal partitions as a robustness boost for neural network classifiers). CBS acknowledges sup-
port from the Philip Leverhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC grants EP/S026045/1 and EP/T003553/1, EP/N014588/1, EP/T017961/1, the Wellcome Innovator Award RG98755, the Leverhulme Trust project Unveiling the invisible, the European Union Horizon 2020 research and innovation programme under the Marie Skodowska-Curie grant agreement No. 777826 NoMADS, the Cantab Capital Institute for the Mathematics of Information, and the Alan Turing Institute.

title: End-to-end reconstruction meets data-driven regularization for inverse problems