## Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration

## Chen Zhang

University College London chen.zhang.16@ucl.ac.uk

Bangti Jin b.jin@ucl.ac.uk

University College London

## Abstract

Aleatoric uncertainty is an intrinsic property of ill-posed inverse and imaging problems. Its quantification is vital for assessing the reliability of relevant point estimates. In this paper, we propose an efficient framework for quantifying aleatoric uncertainty for deep residual learning and showcase its significant potential on image restoration. In the framework, we divide the conditional probability modeling for the residual variable into a deterministic homo-dimensional level, a stochastic low-dimensional level and a merging level. The low-dimensionality is especially suitable for sparse correlation between image pixels, enables efficient sampling for high dimensional problems and acts as a regularizer for the distribution. Preliminary numerical experiments show that the proposed method can give not only state-ofthe-art point estimates of image restoration but also useful associated uncertainty information.

## 1 Introduction

Aleatoric uncertainty (data uncertainty), epistemic uncertainty (model uncertainty) and distributional uncertainty are three different but common types of uncertainty [10, 15, 26, 29, 33, 18]. Unlike the other two types of uncertainty, aleatoric uncertainty is irreducible due to the intrinsic complexity of the data or the model generating the data. It is an intrinsic attribute of ill-posed inverse problems [25, 48, 50], where one aims at recovering the unobservable variable giving rise to the observation and a single observation can be caused by potentially many unobservables.

For several image restoration tasks, recent deep learning based approaches [22, 6, 54, 7, 41, 31, 12, 34] have outperformed more conventional variational approaches, by leveraging powerful approximation ability of deep neural networks (DNNs). However, most works only provide a single restoration for one corrupted image observation. Thus, the possibility of restoring different plausible images (with different probabilities) is neglected, and downstream processing of the restored image can be biased and unreliable. Thus, it is highly desirable to recover the whole distribution rather than a single estimate, with which more samples are available as different references of the intended restoration and an uncertainty indicator, e.g., credible interval and variance, is also available for point estimates (e.g., mean or mode).

In this paper, we present Probabilistic Residual Learning (PRL), a novel framework leveraging the mathematical structure of image restoration problems and enabling efficient sampling from the modeled conditional distribution. It combines the idea of residual learning [54, 30, 55, 49, 14, 35, 36] with Conditional Variational Auto-encoders (CVAEs) [47] and is flexible with network architecture for residual learning. It enables quantifying aleatoric uncertainty by learning from data, and does not require explicit incorporation of image priors and corruption models. Although our work is motivated by image restoration, the framework is flexible and can be easily generalized to other applications where a similar mathematical structure exists.

The contributions of the work can be summarized as follows:

Preprint. Under review.

- · We propose to use an efficient multi-level conditional random generator for the residual variable. The random generator consists of a deterministic component at the homo-dimensional level, a stochastic component at the low-dimensional level and a merging component to output the final samples. The efficiency for inference stems from two aspects. 1) By fully rendering the stochasticity into the low-dimensional level, direct sampling from a highdimensional space can be avoided. 2) With the main network capacity deployed at the deterministic component, extensive computations of the merging component for massive samples can be reduced by using fewer and simpler layers.
- · We show an alternative derivation of the CVAE loss that admits consistent interpretation with aleatoric uncertainty quantification for image restoration. With the CVAE loss, the multi-level conditional generator can be trained without much extra computational effort since the additional reference net is of small size. Further, the additional hyperparameter introduced by many variants of VAE/CVAE (and their applications) for the KL term in loss functions is naturally included and interpreted in our framework.
- · We demonstrate the feasibility of the framework with extensive experiments on Poisson and Gaussian denoising, and explore the uncertainty information provided by the framework with the case study of Poisson denoising.

## 2 Preliminaries

## 2.1 Residual learning for image restoration

The goal of image restoration is to recover the unobservable ground truth image y from the observed corruptions x . Representative examples of image restoration include Gaussian denoising, Poisson denoising, super-resolution and image inpainting etc.

↦

A common structural characteristic of many image restoration tasks is that y is in a way similar to x . For example, in denoising problems with additive white Gaussian noise (AWGN), the mean of noisy image x is the ground truth y and in Poisson denoising problems, the mean and variance of the noisy image x are the ground truth y . Mathematically, we can formulate the target map H : x → y as the composition of an identity map I and a residual map F , i.e.,

$$H = I - F \quad \text{and} \quad H ( x ) = x - F$$

Although for highly corrupted data, the map from x to y is far from the identity map, it is still reasonable to assume that there is an identity map in the target map H x ( ) .

This structural characteristic has recently been very popular and successfully taken advantage of in Computer Vision (CV). In the first synergy of residual representation and DNNs, He et al [19] introduced the concept of Deep Residual Learning to alleviate the degeneration problem in the training of very deep neural networks. Zhang et al [54] took this idea a step further and proposed a fully convolutional network architecture, i.e. denoising convolutional neural network (DnCNN), to directly predict the image noise for denoising problems. Beyond Gaussian denoising, this idea has also demonstrated state-of-the-art performance on several other image restoration tasks [54, 13, 55, 49, 23, 35, 3, 36, 4, 31], e.g. super-resolution, deblocking, dehazing and rain removing. However there is no uncertainty information with residual learning directly, even though it can be very useful for assessing the reliability of the restorations and decision making.

Note that residual learning can be used to represent two regimes with different focuses. In this paper, we use residual learning to refer to the regime which transforms the learning tasks from the target map to the residual map to better suit the highly nonlinear feature of deep neural networks. Thus, it is different from the regime of ResNet [19, 20, 51, 52] which focuses on the architecture design with residual blocks for very deep neural network learning. To the best of our knowledge, this is the first work addressing uncertainty quantification for residual learning. It is worth noting that the proposed framework is independent of the specific architecture choice of the neural net for residual learning and can be trained from scratch.

## 2.2 Conditional Variational Auto-encoders

The tool in the proposed framework enabling uncertainty quantification and low-dimensional representation of the uncertainty is Conditional Variational Auto-encoders (CVAEs) [47], a variant of

Variational Auto-encoders (VAEs) [27]. VAEs (similarly, Generative Adversarial Nets (GANs) [16], normalizing flows [42], and AutoRegressive models [17]) are deep generative models capable of modeling complex distributions. Their conditional versions, e.g. CVAEs and Conditional GANs [38, 8], are also popular and powerful to model conditional distributions. In this section, we give an alternative derivation of the CV AE loss which is consistent with the statistical interpretation of aleatoric uncertainty quantification for image restoration and inverse problems; see the supplementary material for detailed derivations.

Assume that the data { ( x , y i i ) } i follow an underlying distribution p ∗ ( x, y ) . By denoting the marginalized distribution of x and y by p ∗ ( x ) and p ∗ ( y ) , we have p ∗ ( x, y ) = p ∗ ( y x p | ) ∗ ( x ) = p ∗ ( x y p | ) ∗ ( y ) . The target of CVAE is to build a surrogate model p y x ( | ) for the true conditional distribution p ∗ ( y x | ) , usually with respect to a probabilistic divergence, e.g. Kullback-Leibler (KL) divergence. This leads to the following optimization problem

$$\min _ { p ( y | x ) } \{ J ^ { * } ( p ( y$$

where the expectation with respect to p ∗ ( x ) indicates averaging over all possible data in the space of x . By the definition of KL divergence, the functional J ∗ ( p y x ( | )) admits the following decomposition

$$J ^ { * } ( p ( y | x ) ) = \mathbb { E } _ {$$

By introducing the hidden variable z , we have an upper bound for the negative log density term:

$$\begin{array} {$$

where q z x, y ( | ) , p z x ( | ) and p y x, z ( | ) are auxiliary distributions introduced by the hidden variable z . Consequently, J ∗ ( p y x ( | )) can be upper bounded by

$$J ^ { * } ( p ( y | x ) ) \leq \mathbb { E }$$

Instead of directly minimizing the functional J ∗ ( p y x ( | )) , we minimize the upper bound. Since E p ∗ ( x,y ) [log p ∗ ( y x | )] is fully determined by the underlying data distribution p ∗ ( x, y ) and independent of p y x ( | ) , it suffices to minimize the term

$$J \coloneqq \mathbb { E } _ { p ^ { * } ( x, y ) } [ \mathbb { E } _ { z \sim q ( z | x, y ) } [ - \log p ( y | x, z ) ] + \text{K} ( q ( z | x, y ) | | p ( z | x ) ) ], \quad \text{(2)}$$

which is the objective functional appearing in CV AE [47].

The above derivation is consistent with the probabilistic meaning of aleatoric uncertainty quantification for image restoration. The ground-truth images come from an unknown distribution p ∗ ( y ) (prior distribution) and are corrupted according to some observation process p ∗ ( x y | ) (likelihood). For a given observation x , CVAE provides an approximation p y x ( | ) to p ∗ ( y x | ) (posterior distribution) which encodes the probabilities of all possible consistent ground-truth images (to various content).

## 3 Probabilistic Residual Learning (PRL)

By incorporating the structural assumption with an identity map, residual learning transfers the learning task from modeling the target map H , i.e.,

$$H \colon x \mapsto y = H ( x ),$$

↦

to modeling the residual map F , i.e.,

$$F \, \colon x \mapsto r & = x - H ( x ) \\. \, \dots \dots & \dots \dots \dots. c \dots.$$

↦

and the target variable is recovered by y = x -r . Instead of viewing the residual r = x -y as a deterministic variable, we view it as a random variable and model the conditional distribution map F :

$$\mathbb { F } & \colon x \mapsto p ( r | x ). \\ \cdot \cdot \cdot \cdot \ e$$

↦

The proposed Probabilistic Residual Learning is a framework including a multi-level conditional random generator for p r x ( | ) and a suitable loss to align the generator with a target statistical interpretation. We adopt the CVAE loss in Section 2.2 for aleatoric uncertainty quantification. With the random generator for p r x ( | ) , samples for p y x ( | ) can be easily recovered with a shift of x . It is worth noting that the transformation on all samples from r to y is not always necessary if one aims at a mean and credible interval.

Figure 1: Illustration of the multi-level conditional random generator for the residual variable. Yellow blocks for deterministic quantities, red blocks for samples of z and blue blocks for samples of r .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-e7f570ff-569c-47c8-8c83-fdd97c95296e.jpg)

Instead of modeling p r x ( | ) at a single level, we introduce a low-dimensional latent random variable z and separate the modeling of the distribution into a homo-dimensional level, a low-dimensional level and a merging level. We render the source of stochasticity of p r x ( | ) into the low-dimensional level, thereby avoiding direct sampling from a high-dimensional space. For example, for images of size m × n and latent variable of dimension d , the computational complexity per sample is reduced from mn to d . Moreover, we deploy the main network modeling capacity at the homo-dimensional level, and the capacity needed at the merging level can thus be compensated and greatly reduced. In particular, for the network at the merging level, fewer and simpler layers are sufficient and the expensive computations for sampling are saved. Further, with the stochasticity separation and distribution of model capacity, the conditional distribution can be regarded as a form of structural regularization.

The low-dimensionality of the latent representation z assumes that the distribution of r lives on a low-dimensional manifold for a given observation x . This assumption is reasonable in many imaging tasks. First, due to the correlation among entries in r , the degree of freedom to parameterize r is lower than the ambient dimensionality. Second, since not all elements in the high-dimensional neighborhood of a natural image are reasonable natural images, we may model the distribution of reasonable natural images on a low-dimensional manifold. Moreover, the latent variable z acts as a mixture variable as that in the mixture of distributions. Thus, it also contributes to the expressiveness of the distribution family the framework can model.

Below we denote the training stage of the proposed PRL as training mode and the test stage as inference mode. There are three network components in PRL in the inference mode, i.e., a deterministic feature net D (at homo-dimensional level), a stochastic feature net S (at low-dimensional level) and a residual prediction net R (at merging level). The deterministic feature net takes x as input and outputs a deterministic feature D x ( ) . The stochastic feature net also takes x as input but outputs a distribution p z x ( | ) , from which we can sample z for a given x . One possible way is to output the parameter θ for a distribution p θ ( z x | ) . The residual prediction net takes the deterministic feature D x ( ) and a sample z from p z x ( | ) as input and outputs R D x ,z ( ( ) ) as a sample from p r x ( | ) . The target y is then estimated by simple subtraction y = x -r . We denote the parameters of these three components by ν = ( ν D , ν S , ν R ) .

To train the whole network, we need a reference distribution q z x, y ( | ) , which can be modeled by a second network Q x,y ( ) with parameter ν Q . The roles of q z x, y ( | ) are two-folded: 1) to serve as an insider encoder of x relying on the known ground truth y and 2) to help train the blind encoder p z x ( | ) that is used in the inference mode for which the ground truth y is unavailable. We slightly abuse the notation to denote the augmented parameter ( ν D , ν S , ν R , ν Q ) also by ν . The optimal augmented parameter ν ∗ = ( ν ∗ D , ν ∗ S , ν ∗ R , ν ∗ Q ) is defined as the minimizer of the following functional

$$J ( \nu ) = \mathbb { E } _ { ( x, y ) \sim p ^ { * } ( x, y ) } \mathcal { L } ( \nu | x, y ),$$

where p ∗ ( x, y ) is the joint distribution of data and

$$\mathcal { L } ( \nu | x, y ) = \mathbb { E } _ { z \sim q ( z | x, y ) } [ - \log p ( y | x, z ) ] + \text{KL} ( q ( z | x, y ) | | p ( z | x ) ).$$

With the training data { ( x , y i i ) } N i =1 as samples from the joint distribution p ∗ ( x, y ) , we instead minimize the empirical objective function

$$\hat { J } ( \nu ) = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \mathcal { L } ( \nu | x ^ { i }, y ^ { i } ),$$

which converges to the objective function J v ( ) as N →∞ , by the classical theory in Monte-Carlo approximation. In practice, a 'mini-batch' stochastic iterative algorithm evaluates the gradient of the objective function on batches { ( x , y i i ) } m i =1 of size m glyph[lessmuch] N to alleviate the computational burden.

The evaluation of the loss L ( ν x, y | ) on a single data pair ( x, y ) involves three distributions, i.e. p y x, z ( | ) , q z x, y ( | ) and p z x ( | ) . In our framework, we choose

$$p ( y | x, z ) = \mathcal { N } ( y | x - R ( D ( x ), z ), \beta I ) \\ - \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

with β &gt; 0 being a hyperparameter. This leads to the following negative log-likelihood

$$- \log p ( y | x, z ) = \frac { 1 } { 2 \beta } | | y - ( x - R ( D ( x ), z ) ) | | ^ { 2 } + \log ( 2 \pi ) ^ { \frac { k _ { y } } { 2 } } \beta ^ { \frac { 1 } { 2 } }.$$

Since the second term on the right hand side does not depend on the network parameters ν , its gradient with respect to ν vanishes, and it can be ignored during the optimization (but will be needed within a hierarchical Bayesian treatment).

The choice of p y x, z ( | ) enables the separation of high dimensional distributions into a homodimensional level and a low-dimensional level without compromising the expressiveness of the framework, since we do not require specific distribution family on q z x, y ( | ) and p z x ( | ) , as long as they can be parameterized by neural networks and sampled from. For a general choice of q z x, y ( | ) and p z x ( | ) , one can approximate the KL divergence term KL ( q z x, y ( | ) || p z x ( | )) by evaluating the difference of logarithm density log q z x, y ( | ) -log p z x ( | ) with one sample z from q z x, y ( | ) . For special choices of q z x, y ( | ) and p z x ( | ) , one may evaluate them analytically. In the experiments below, we choose them to be diagonal multivariate Gaussians and use TensorFlow Probability [11] built-in API to evaluate the KL term analytically.

Besides, we employ a Monte Carlo estimator for the expectation with respect to z in the first term E z ∼ q z x,y ( | ) [ -log p y x, z ( | )] . With L samples { z glyph[lscript] q } L glyph[lscript] =1 from q z x, y ( | ) , the loss function on every data pair ( x, y ) can be estimated by

$$\cdot \cdot \cdot \cdot \cdot \\ \mathcal { L } ^ { L } ( \nu | x, y ) & = \frac { 1 } { L } \sum _ { \ell = 1 } ^ { L } \log p ( y | x, z _ { q } ^ { \ell } ) + \text{KL} ( q ( z | x, y ) | | p ( z | x ) ) \\ & = \frac { 1 } { 2 L \beta } \sum _ { \ell = 1 } ^ { L } | | y - ( x - R ( D ( x ), z _ { q } ^ { \ell } ) ) | | ^ { 2 } + \text{KL} ( q ( z | x, y ) | | p ( z | x ) ). \\ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \colon$$

In practice, one sample approximation is often used to reduce the computational cost, i.e. L 1 ( ν x, y | ) . In sum, we obtain the empirical loss on a mini-batch { ( x , y i i ) } m i =1 of size m and single z -sampling

$$\dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \delta$$

In the lens of regularization theory, the first term is data fidelity and the second one is a regularization term. So one can take the hyperparameter 1 β out and optimize an equivalent regularized problem:

$$\hat { J } _ { \beta } ( \nu ) = \frac { 1 } { 2 m } \sum _ { i = 1 } ^ { m } | | y ^ { i } - ( x ^ { i } - R ( D ( x ^ { i } ), z _ { q } ) ) | | ^ { 2 } + \frac { \beta } { m } \sum _ { i = 1 } ^ { m } \text{KL} ( q ( z | x ^ { i }, y ^ { i } ) | | p ( z | x ^ { i } ) ). \quad ( 5 )$$

In our experiments, we employ the default setting with β = 0 1 . .

## 4 Related work

Quantification for aleatoric uncertainty In conventional forward-model based scenarios, prior distributions and likelihood functions are explicitly specified, and the aleatoric uncertainty is quantified with posterior exploration. Except for very few cases where the posterior distributions admit tractable forms, most posterior explorations are performed by approximate inference techniques, e.g., variational inference [24, 1] and expectation propagation [37, 53]. In contrast, in implicitly forward-model based scenarios, data is generated from joint distributions, and the uncertainty is recovered from the data in a learning manner. One popular idea is to fit a presumed distribution to the given data in a frequentist way with neural networks outputting distributional parameters, e.g., mean and variance [40, 26]. However, it is restricted to simple known-form distributions and may suffer from curse of dimensionality for high-dimensional problems. A concurrent work [50] also proposes to use CVAE to address uncertainty quantification for inverse problems and provides an interpretation of CVAE loss from an entropy view, while our work further proposes to leverage a class of mathematical structure and focus more on efficiency. It is worth noting that previous works on uncertainty quantification for neural networks, e.g. Bayesian neural networks [32, 39, 15], admit statistical interpretation for epistemic uncertainty and thus are different from aleatoric uncertainty which is the focus of the present work.

Multi-level random generator Random generators are an inherent part of deep generative models, e.g. V AEs and GANs. Since generative distributions are unconditional, samples are directly generated in a latent space and transformed into the target space. In conditional cases, the latent distributions are conditioned on the input variable which lives in a high-dimensional space for image restoration. The work [50] also renders the stochasticity into the low-dimensional space but directly connect the input with low-dimensional samples. This is equivalent to reducing the model capacity at the homo-dimensional level and thus increases computation complexity needed for massive samples at the merging level. The idea of the multi-level random generator in our framework originates from [28], where the authors combined it with a U-Net [43] to generate segmentation logits and attributed the efficiency to only involving a small part of the network for repetitive sampling. In this work, we further analyze the idea and use it for residual learning, and pinpoint the structural regularization for distribution modeling.

KL multiplier In many VAE/CVAE applications and their variants, a multiplier β to the KL term is often introduced. The work [5] identified the KL vanishing problem and proposed a sigmoid annealing scheme, and the work [46] suggested a linear annealing scheme. By annealing the multiplier β , the objective function converges to the original one. This multiplier β can also be introduced by a relaxed version of the objective function [21, 28] as a Lagrange multiplier. Our framework naturally allows a hyperparameter β with the original loss and interpret it as a presumed variance without compromising the original interpretation of variational inference.

## 5 Experimental results

In this section, we compare the proposed framework with state-of-the-art methods for Gaussian and Poisson denoising and use Poisson denoising as a case study to show the quantified uncertainty. In all experiment, we adopt a 17 -layer DnCNN for the deterministic feature net D , a 7 -layer VGG style encoder for the stochastic feature net S and reference net Q and a 5 -layer 1 × 1 convolutional net for the residual prediction net R . It is trained on the dataset based on 400 images (referred to as Train400) as [7, 54] on a desktop with two Nvidia GeForce 1080 Ti GPU and Intel i7-7700K CPU 4.20GHz × 8 . The test set consists of 12 common images in CV community (referred to as Test12 and shown in supplementary materials) and Berkeley segmentation dataset (BSD68) [44]. All three datasets are publicly available and can be found with the implementation of our paper ( https://github.com/chenzxyz/prob\_res\_learning ). We refer to the supplementary material for further implementation details and results.

## 5.1 Comparison with benchmarks

Since few work addresses aleatoric uncertainty quantification for image restorations, first we compare the mean estimates derived from the PRL framework with restorations by state-of-the-art deterministic methods. For Gaussian denoising, we compare our framework with BM3D [9], TNRD [7] and

DnCNN [54] on noise levels σ = 15 25 50 , , . For Poisson denoising, we compare our framework with NLSPCA [45], I+VST+BM3D [2] and Deep Class Aware Denoising (DCAD) [41] on peak values 1 2 4 8 , , , . The SSIM and PSNR values (averaged over the test dataset) are shown in Tables 1 and 2. it is clearly observed that the mean estimate is comparable with other state-of-the-art methods in terms of SSIM and PSNR values. Therefore, the PRL framework can maintain the good accuracy of DNN type restoration methods and deliver reconstructions competitive with state-of-the-art methods.

Table 1: Comparisons of average SSIM/PSNR on benchmark datasets for Gaussian denoising

| Dataset   |   Noise level | BM3D         | TNRD         | DnCNN          | Ours           |
|-----------|---------------|--------------|--------------|----------------|----------------|
| Test12    |            15 | 0.8953/32.39 | 0.8965/32.51 | 0.9056/32.88   | 0.9057 / 32.89 |
| Test12    |            25 | 0.8500/29.96 | 0.8518/30.05 | 0.8629/30.42   | 0.8636 / 30.44 |
| Test12    |            50 | 0.7676/26.71 | 0.7674/26.81 | 0.7819 / 27.21 | 0.7792/27.12   |
| BSD68     |            15 | 0.8720/31.07 | 0.8824/31.42 | 0.8953/31.54   | 0.8960 / 31.73 |
| BSD68     |            25 | 0.8013/28.56 | 0.8152/28.92 | 0.8287 /28.86  | 0.8246/ 29.00  |
| BSD68     |            50 | 0.6863/25.61 | 0.7015/25.95 | 0.7191 / 26.23 | 0.7190/26.19   |

Table 2: Comparisons of average SSIM/PSNR on benchmark datasets for Poisson denoising.

| Dataset   | Peak value   | NLSPCA                                 | I+VST+BM3D                              | DCAD                                      | Ours                               |
|-----------|--------------|----------------------------------------|-----------------------------------------|-------------------------------------------|------------------------------------|
| Test12    | 1 2 4        | 0.5383/20.32 0.5783/21.20 0.6030/21.82 | 0.5855 /21.45 0.6297/22.70 0.6841/24.27 | 0.5746/ 21.71 0.6376 / 23.16 0.6952/24.63 | 0.5650/21.18 0.6302/23.12 0.6981 / |
| Test12    |              |                                        |                                         |                                           | 24.72                              |
| Test12    | 8            | 0.6113/22.01                           | 0.7404/25.88                            | 0.7506/26.17                              | 0.7539 / 26.30                     |
| BSD68     | 1            | 0.4900/21.05                           | 0.5282 /21.81                           | 0.5160/ 21.90                             | 0.4873/19.55                       |
| BSD68     | 2            | 0.5189/21.77                           | 0.5646/22.74                            | 0.5686/23.05                              | 0.5706 / 23.06                     |
| BSD68     | 4            | 0.5368/22.28                           | 0.6162/23.84                            | 0.6296/24.12                              | 0.6364 / 24.30                     |
| BSD68     | 8            | 0.5463/22.58                           | 0.6721/25.07                            | 0.6954/25.44                              | 0.6970 / 25.59                     |

## 5.2 Exploration of the uncertainty

Now we use Poisson denoising as a case study to show the aleatoric uncertainty quantified by our framework. For Poisson denoising, the signal-to-noise ratio (SNR) of the noisy images decreases as the peak value decreases. Thus, the aleatoric uncertainty is expected to be higher for denoising problems with a lower peak value. It is observed from Figures 7 and 8 that the restoration degenerates and more structural features emerge in variance as the the peak value decreases. Further, the 0.95 credible interval also broadens as the peak value decreases, cf. Figure 4. The credible interval can be used for assessing the reliability of a particular point estimate.

## 6 Conclusion

In this paper, we propose a novel framework termed as Probabilistic Residual Learning (PRL) for aleatoric uncertainty quantification for image restoration. It is designed for the efficient sampling in high-dimensional spaces and leverages a mathematical structure existing in many real world problems, e.g. image restoration. The numerical results show that the framework can provide state-of-the-art point estimates from reasonable natural image samples and higher uncertainty indicator for higher aleatoric uncertainty cases.

Avenues for further study can be pursued in both practical and theoretical aspects. Practically, it is of interest to investigate the framework for multimodal / dynamic image restoration problems, and to integrate it with advanced inversion techniques, e.g., various plug-and-play type schemes, for the uncertainty quantification of relevant point estimates. Theoretically, it is of enormous interest to analyze the convergence rate of the Monte Carlo approximation for the CVAE loss and the bound of distance between the approximate distribution p y x ( | ) and the underlying truth p ∗ ( y x | ) as well as the representation learning property of the CV AE (in analogy with classical PCA).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-7e540f82-2e47-45ad-9db2-3438776e8021.jpg)

Figure 2: Poisson denoising with peak values 1,2,4 shown with ground truth image, noisy image, sample mean and sample variance for Cameraman of size 256 × 256

Figure 3: Poisson denoising with peak values 1,2,4 shown with ground truth image, noisy image, sample mean and sample variance for Lena of size 512 × 512

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-a8865eb0-94ba-4729-a93c-3f5ff9295350.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-2afa0ad2-7491-424a-ac2f-dbec433cabf5.jpg)

Figure 4: Poisson denoising with peak values 1,2,4 shown with the sample mean and 0.95 credible interval of the 200-th horizontal slice for Cameraman of size 256 × 256

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-b1791bcf-379a-44d7-aff6-324ce6c78499.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-71bda89d-2b17-4e1e-9f72-b08bb8c044f0.jpg)

## References

|      | [1] S. R. Arridge, K. Ito, B. Jin, and C. Zhang. Variational Gaussian approximation for Poisson data. Inverse Problems , 34(2):025005, 2018.                                                                                                                                                           |
|------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [2]  | L. Azzari and A. Foi. Variance stabilization for noisy+ estimate combination in iterative Poisson denoising. IEEE Signal Proc. Lett. , 23(8):1086-1090, 2016.                                                                                                                                          |
| [3]  | W. Bae, J. Yoo, and J. Chul Ye. Beyond deep residual learning for image restoration: Persistent homology- guided manifold simplification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 145-153, 2017.                                            |
| [4]  | S. A. Bigdeli, M. Zwicker, P. Favaro, and M. Jin. Deep mean-shift priors for image restoration. In Advances in Neural Information Processing Systems , pages 763-772, 2017.                                                                                                                            |
| [5]  | S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning , pages 10-21, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. |
| [6]  | H. C. Burger, C. J. Schuler, and S. Harmeling. Image denoising: Can plain neural networks compete with bm3d? In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 2392-2399. IEEE, 2012.                                                                                         |
| [7]  | Y. Chen and T. Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Trans. Patt. Anal. Mach. Int. , 39(6):1256-1272, 2017.                                                                                                                |
| [8]  | G. G. Chrysos, J. Kossaifi, and S. Zafeiriou. Roc-GAN: Robust conditional GAN. In International Conference on Learning Representations , 2019.                                                                                                                                                         |
| [9]  | K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image restoration by sparse 3d transform-domain collaborative filtering. In Image Processing: Algorithms and Systems VI , volume 6812, page 681207. International Society for Optics and Photonics, 2008.                                           |
| [10] | A. Der Kiureghian and O. Ditlevsen. Aleatory or epistemic? Does it matter? Structural Safety , 31(2):105- 112, 2009.                                                                                                                                                                                   |
| [11] | J. V. Dillon, I. Langmore, D. Tran, E. Brevdo, S. Vasudevan, D. Moore, B. Patton, A. Alemi, M. Hoffman, and R. A. Saurous. Tensorflow distributions. arXiv preprint arXiv:1711.10604 , 2017.                                                                                                           |
| [12] | C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision , pages 184-199. Springer, 2014.                                                                                                                   |
| [13] | Y. Du and X. Li. Recursive deep residual learning for single image dehazing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 730-737, 2018.                                                                                                         |
| [14] | X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pages 3855-3863, 2017.                                                                       |
| [15] | Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning , pages 1050-1059, 2016.                                                                                                               |
| [16] | I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , pages 2672-2680, 2014.                                                                                   |
| [17] | K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra. Deep autoregressive networks. In Proceedings of the 31st International Conference on Machine Learning , pages 1242-1250. PMLR, 22-24 Jun 2014.                                                                                         |
| [18] | D. Hafner, D. Tran, A. Irpan, T. Lillicrap, and J. Davidson. Reliable uncertainty estimates in deep neural networks using noise contrastive priors. arXiv preprint arXiv:1807.09289 , 2018.                                                                                                            |
| [19] | K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.                                                                                                      |
| [20] | K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision , pages 630-645. Springer, 2016.                                                                                                                                           |
| [21] | I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , volume 3, 2017.                                         |
| [22] | V. Jain and S. Seung. Natural image denoising with convolutional networks. In Advances in Neural Information Processing Systems , pages 769-776, 2009.                                                                                                                                                 |
| [23] | J. Jiao, W.-C. Tu, S. He, and R. W. Lau. Formresnet: formatted residual learning for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages                                                                                             |

| [24]   | M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Mach. Learn. , 37(2):183-233, 1999.                                                                                                             |
|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [25]   | J. Kaipio and E. Somersalo. Statistical and Computational Inverse Problems , volume 160. Springer, 2006.                                                                                                                                                                  |
| [26]   | A. Kendall and Y. Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems , pages 5574-5584, 2017.                                                                                         |
| [27]   | D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR) , 2014.                                                                                                                   |
| [28]   | S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. A. Eslami, D. J. Rezende, and O. Ronneberger. A probabilistic U-Net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems , pages 6965-6975, 2018. |
| [29]   | B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , pages 6402-6413, 2017.                                                               |
| [30]   | D. Lee, J. Yoo, and J. C. Ye. Deep residual learning for compressed sensing mri. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) , pages 15-18. IEEE, 2017.                                                                                   |
| [31]   | D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang. Non-local recurrent network for image restoration. In Advances in Neural Information Processing Systems , pages 1673-1682, 2018.                                                                                      |
| [32]   | D. J. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation , 4(3):448-472, 1992.                                                                                                                                                       |
| [33]   | A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems , pages 7047-7058, 2018.                                                                                                              |
| [34]   | X. Mao, C. Shen, and Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems , pages 2802-2810, 2016.                                                  |
| [35]   | T. Meinhardt, M. Moller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In Proceedings of the IEEE International Conference on Computer Vision , pages 1781-1790, 2017.                    |
| [36]   | C. Metzler, A. Mousavi, and R. Baraniuk. Learned D-AMP: Principled neural network based compressive image recovery. In Advances in Neural Information Processing Systems , pages 1772-1783, 2017.                                                                         |
| [37]   | T. P. Minka. Expectation propagation for approximate Bayesian inference. In Proceedings of the Seven- teenth Conference on Uncertainty in Artificial Intelligence , pages 362-369. Morgan Kaufmann Publishers Inc., 2001.                                                 |
| [38]   | M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.                                                                                                                                                                 |
| [39]   | R. M. Neal. Bayesian learning for neural networks , volume 118. Springer, 2012.                                                                                                                                                                                           |
| [40]   | D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 1, pages 55-60. IEEE, 1994.                                              |
| [41]   | T. Remez, O. Litany, R. Giryes, and A. M. Bronstein. Class-aware fully convolutional Gaussian and Poisson denoising. IEEE Trans. Image Proc. , 27(11):5707-5722, 2018.                                                                                                    |
| [42]   | D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning - Volume 37 , pages 1530-1538. JMLR.org, 2015.                                                                        |
| [43]   | O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention , pages 234-241. Springer, 2015.                                      |
| [44]   | S. Roth and M. J. Black. Fields of experts: Aframework for learning image priors. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 2, pages 860-867. Citeseer, 2005.                                                |
| [45]   | J. Salmon, Z. Harmany, C.-A. Deledalle, and R. Willett. Poisson noise reduction with non-local PCA. J. Math. Imag. Vis. , 48(2):279-294, 2014.                                                                                                                            |
| [46]   | H. Shah and D. Barber. Generative neural machine translation. In Advances in Neural Information Processing Systems , pages 1346-1355, 2018.                                                                                                                               |
| [47]   | K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems , pages 3483-3491, 2015.                                                                            |
| [48]   | A. M. Stuart. Inverse problems: a Bayesian perspective. Acta Numer. , 19:451-559, 2010.                                                                                                                                                                                   |
| [49]   | Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision , pages 4539-4547, 2017.                                                                            |

| [50]   | F. Tonolini, A. Lyons, P. Caramazza, D. Faccio, and R. Murray-Smith. Variational inference for computa- tional imaging inverse problems. arXiv preprint arXiv:1904.06264 , 2019.                         |
|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [51]   | A. Veit, M. J. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems , pages 550-558, 2016.              |
| [52]   | X. Yu, Z. Yu, and S. Ramalingam. Learning strict identity mappings in deep residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4432-4440, 2018.  |
| [53]   | C. Zhang, S. R. Arridge, and B. Jin. Expectation propagation for Poisson data. Inverse Problems , 2019.                                                                                                  |
| [54]   | K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Proc. , 26(7):3142-3155, 2017.                        |
| [55]   | K. Zhang, W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3929-3938, 2017. |

## A Proof of Section 2.2

The arguments in Section 2.2 can be mathematically summarized in the following two claims.

Claim A.1. By solving the variational problem

$$\min _ { p ( y | x ) } \{ J ^ { * } ( p ( y | x ) ) = \mathbb { E } _ { p ^ { * } ( x ) } K L ( p ^ { * } ( y | x ) | | p ( y | x ) ) \},$$

p y x ( | ) can approximately quantify the aleatoric uncertainty of the unobservable variable y for a given observation x .

Proof. Recall that, in the context of Bayesian inversion, the distribution p ∗ ( y ) encodes the prior information of the hidden variable y and the likelihood p ∗ ( x y | ) encodes the forward model that relates the probability of an observation x to a given hidden variable y . Thus, the posterior distribution p ∗ ( y x | ) quantifies the aleatoric uncertainty of the unobservable variable y for a given observation x in the Bayesian sense. Now by the definition in (6), p y x ( | ) is the best approximation to p ∗ ( y x | ) in the sense of expectation on all possible observation x . Thus, p y x ( | ) can approximately quantify the aleatoric uncertainty of y for a given x .

Claim A.2. Minimizing the CVAE functional (7) is equivalent to minimizing an upper bound of problem (6) , where the CVAE functional is defined by

$$J \coloneqq \mathbb { E } _ { p ^ { * } ( x, y ) } [ \mathbb { E } _ { z \sim q ( z | x, y ) } [ - \log p ( y | x, z ) ] + K L ( q ( z | x, y ) | | p ( z | x ) ) ].$$

Note that J is a functional of the variational distribution p y x ( | ) and other auxiliary distributions introduced with z .

Proof. By the definition of KL divergence and Fubini theorem,

$$\text{By the definition of KL divergence and Fubini theorem}, & J ^ { * } ( p ( y | x ) ) & = \mathbb { E } _ { p ^ { * } ( x ) } K L ( p ^ { * } ( y | x ) | p ( y | x ) ) \\ & = \int p ^ { * } ( x ) \int p ^ { * } ( y | x ) \log \frac { p ^ { * } ( y | x ) } { p ( y | x ) } d y d x \\ & = \int p ^ { * } ( x, y ) [ \log p ^ { * } ( y | x ) - \log p ( y | x ) ] d ( x, y ) \\ & = \mathbb { E } _ { p ^ { * } ( x, y ) } [ \log p ^ { * } ( y | x ) ] + \mathbb { E } _ { p ^ { * } ( x, y ) } [ - \log p ( y | x ) ]. \\ / \text{ the classical derivation of a lower bound for the logarithm loop} p ( y | x ) \text{ of the conditional distribution}$$

Then by the classical derivation of a lower bound for the logarithm log p y x ( | ) of the conditional distribution p y x ( | ) ,

$$\text{$ = \mathbb{m}_{p}^{*}(x,y)\{\log p_{\}(x\})+\mathbb{m}_{p}^{*}(x,y)\}^{-} \log p_{\}(y\}^{x\}).$} \\ \text{$ by the classical derivation of a lower bound for the logarithm log p(y|x) of the conditional distrib$} \\ x ), \\ \log p(y|x) = \int q(z|x,y) \log p(y|x)dz \\ & = \int q(z|x,y) \log \frac { p(y|x)p(z|x,y) } { p(z|x,y) } dz \\ & = \int q(z|x,y) \log \frac { p(y, z|x) } { p(z|x,y) } dz \\ & = \int q(z|x,y) \log \frac { p(y, z|x) } { q(z|x,y) } dz \\ & = \int q(z|x,y) \log \frac { p(y, z|x) } { q(z|x,y) } dz + \int q(z|x,y) \log \frac { q(z|x,y) } { p(z|x,y) } dz \\ & = \int q(z|x,y) \log \frac { p(y, z|x) } { q(z|x,y) } dz \\ & = \int q(z|x,y) \log \frac { p(z|x)p(y|x,z) } { q(z|x,y) } dz + \int q(z|x,y) \log p(y|x,z)dz \\ & = - KL(q(z|x,y)||p(z|x)) + \mathbb{E} _ { z } \sim _ { q(z|x,y) [ \log p(y|x,z)],} \\ \text{re the inequality is due to the nonnegativity of the Kullback-Leibler divergence. Consequently, we hav} \\ - \log p(y|x) \leq KL(q(z|x,y)||p(z|x)) + \mathbb{E} _ { z } \sim _ { q(z|x,y) [ - \log p(y|x,z)],}.$$

= -KL ( q z x, y ( | ) || p z x ( | )) + z ∼ q z x,y ( | ) [log p y x, z ( | )] , where the inequality is due to the nonnegativity of the Kullback-Leibler divergence. Consequently, we have

$$- \log p ( y | x ) \leq K L ( q ( z | x, y ) | | p ( z | x ) ) + \mathbb { E } _ { z \sim q ( z | x, y ) } [ - \log p ( y | x, z ) ].$$

Substituting inequity (9) into equation (8) yields

$$J ^ { * } ( p ( y | x ) ) \leq \mathbb { E } _ { p ^ { * } ( x, y ) } [ \log p ^ { * } ( y | x ) ] + J.$$

Since the term E p ∗ ( x,y ) [log p ∗ ( y x | )] is independent of the variational distribution p y x ( | ) and other auxiliary distributions introduced with z , minimizing J is equivalent to minimizing an upper bound of J ∗ ( p y x ( | )) .

## B Supplementary material for experimental results

First we comment on the data generation process. The training data of clean images is randomly augmented from Train400 with flipping, rotation and split into patches. It is worth noting that here we refer epoch as one ergodicity on the augmented clean images, since the noisy images are sampled from the corruption process, e.g. Poisson distribution for Poisson denoising problems, just before fed into training. In doing so, we incorporate the stochasticity of both prior distribution p ∗ ( y ) and likelihood p ∗ ( x y | ) into the training data. Besides, the networks for different corruption types and levels are trained separately.

In the PRL framework, there is only one hyperparameter β , i.e., the multiplier of the KL term in the loss function. A too large β can result in vanishing of the KL term, a too small β can affect the convergence of training. In our experiments, we search over the interval β ∈ [1 e -2 1 1] , e and find that the default setting β = 0 1 . can balance well the magnitude of the two terms in the loss function, i.e. the data fidelity term and the KL term.

For the numerical stability of training, the data is scaled before fed into the networks. For Gaussian denoising, we use f ( ) = · · 255 for both x and y , whereas for Poisson denoising, we use f ( ) = · · peak of clean image -0 5 . for both x and y . The results are recovered into original scales before quantitative evaluations. The metrics for evaluating the experimental results are Structural Similarity Index (SSIM) and Peak Signal to Noise Ratio (PSNR). We set the data range to 255 for Gaussian denoising and to the peak value of clean images for Poisson denoising. Note that deterministic transformations on data do not change the statistical interpretation of conditional modeling of p y x ( | ) as long as they are consistent during training and inference.

Then we briefly comment on the computing time. In our current implementation (without optimizing the code), the training on average takes around 15h50m for 150 epochs and 1862 batches per epoch. The sample statistics (mean and variance) reported in the paper are evaluated based on 1000 samples of restoration y for each noisy image x . Sampling 12000 images in total takes around 340s-350s on Test12 dataset (7 of size 256 × 256 and 5 of size 512 × 512 ). Sampling 68000 images in total takes around 2380s-2390s on BSD68 dataset (15 of size 321 × 481 and 53 of size 481 × 321 ). Overall, it takes about 0.03s to sample one image for both datasets.

## B.1 Supplementary material for Section 5.1

In this part, we show the mean of the restored images from the proposed PRL framework on Test12 dataset. The results are shown for 1) Gaussian noise levels 15 and 50 and 2) Poisson peak values 1 and 8. In dataset Test12, the first 7 images are of size 256 × 256 (results given in Figure 5) and the remaining images are of size 512 × 512 (results given in Figure 6).

## B.2 Supplementary material for Section 5.2

In this part, we show more 0.95 credible intervals in addition to that in Section 5.2. The results are shown with Cameraman (c.f. Figure 7) and Lena (c.f. Figure 8) from the dataset Test12. Further we show the histograms of SSIM and PNSR based on 1000 samples for each case. From Figure 9 and 10, one can observe that samples from the modeled distributions are all reasonable natural images with the high SSIM and PSNR values. Further, the distribution of these indicators (SSIM and PSNR) are not necessarily Gaussian and can exhibit high skewness.

Figure 5: Denoising examples on Test12 dataset (01-07, of size 256 × 256 ) shown with clean image and restored image sample means from Gaussian 15, 50 and Poisson 1, 8 (left to right)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-99fe64b3-4952-44fc-910f-aa60aec30379.jpg)

Figure 6: Denoising examples on Test12 dataset (08-12, of size 512 × 512 ) shown with clean image and restored image sample means from Gaussian 15, 50 and Poisson 1, 8 (left to right)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-1a39f8aa-24dc-4fa7-9ed8-f641dfb9877f.jpg)

Figure 7: Poisson denoising with peak values 1,2,4 (left to right) shown with the sample mean and 0.95 credible interval of the 100-th and 200-th horizontal slices (upper to lower) for Cameraman of size 256 × 256

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-3098031a-f562-4d34-b7b9-da497cecfa08.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-3418b8c5-7e09-483d-bc24-3c449ef0147a.jpg)

Figure 8: Poisson denoising with peak values 1,2,4 (left to right) shown with the sample mean and 0.95 credible interval of the 200-th and 400-th horizontal slices (upper to lower) for Lena of size 512 × 512

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-579bee65-1d0e-4df9-90d3-d256b01012d6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-eadd1584-a0a3-4d9f-9d8d-fdc26ba8f303.jpg)

SSIM histogram

PSNR

histogram

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-0ee3ddef-70be-4429-90e4-96baa0ca25e2.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-eee483a1-3f26-4045-96bb-ac0cdb2f5955.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-f3566106-82ca-42c8-9b01-cc335c24feb1.jpg)

PSNR histogram

Figure 9: SSIM and PNSR histograms of 1000 samples from Poisson denoising with peak value 1,2 and 4 (upper to lower) on Cameraman

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-8347be4d-b524-40ab-b89d-b628cafa0abb.jpg)

Figure 10: SSIM and PNSR histograms of 1000 samples from Poisson denoising with peak value 1,2 and 4 (upper to lower) on Lena

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01010v1-figure-7ea39fdf-70d0-4d6e-9eee-d1d79f3c6616.jpg)
Chen Zhang Bangti Jin
Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration
Aleatoric uncertainty is an intrinsic property of ill-posed inverse and
imaging problems. Its quantification is vital for assessing the reliability of
relevant point estimates. In this paper, we propose an efficient framework for
quantifying aleatoric uncertainty for deep residual learning and showcase its
significant potential on image restoration. In the framework, we divide the
conditional probability modeling for the residual variable into a deterministic
homo-dimensional level, a stochastic low-dimensional level and a merging level.
The low-dimensionality is especially suitable for sparse correlation between
image pixels, enables efficient sampling for high dimensional problems and acts
as a regularizer for the distribution. Preliminary numerical experiments show
that the proposed method can give not only state-of-the-art point estimates of
image restoration but also useful associated uncertainty information.