## Few-Shot Meta-Denoising

Leslie Casas , Attila Klimmek , Gustavo Carneiro , Nassir Navab , and Vasileios Belagiannis 1 2 3 1 2

## 1 Technische Universit¨ at M¨nchen, Garching bei M¨ unchen, Germany

Universit¨ at Ulm, Ulm, Germany 3 The University of Adelaide, Adelaide, Australia firstname.lastname@ { tum.de, uni-ulm.de, adelaide.edu.au }

u 2

## Abstract

We study the problem of few-shot learning-based denoising where the training set contains just a handful of clean and noisy samples. A solution to mitigate the small training set issue is to pre-train a denoising model with small training sets containing pairs of clean and synthesized noisy signals, produced from empirical noise priors, and fine-tune on the available small training set. While such transfer learning seems effective, it may not generalize well because of the limited amount of training data. In this work, we propose a new meta-learning training approach for few-shot learningbased denoising problems. Our model is meta-trained using known synthetic noise models, and then fine-tuned with the small training set, with the real noise, as a few-shot learning task. Meta-learning from small training sets of synthetically generated data during meta-training enables us to not only generate an infinite number of training tasks, but also train a model to learn with small training sets - both advantages have the potential to improve the generalisation of the denoising model. Our approach is empirically shown to produce more accurate denoising results than supervised learning and transfer learning in three denoising evaluations for images and 1-D signals. Interestingly, our study provides strong indications that meta-learning has the potential to become the main learning algorithm for denoising.

## 1. Introduction

Learning-based denoising methods typically require a large training set with pairs of clean and noisy samples, where the simultaneous access to the pair is usually challenging. In this framework, training sets are often built using synthetic noise that is added to images based on a prior noise model [42]. The use of prior noise models for training data generation models is also common in medical imaging, where the noise distribution is usually assumed to be known [45, 48]. In other cases, such as with motion or image signals [1, 9], it is possible to employ multiple sensors for simultaneous capturing of clean and noisy signals. However, this setup cannot be easily replicated in other real-world scenarios. Consequently, the availability of training sets that contain a large amount of pairs of clean and real noisy samples is limited, making the study of fewshot learning-based denoising an important research topic.

To frame our problem, we assume that: 1) we have access to clean samples (i.e. noiseless), 2) we have empirical prior knowledge of the real noise distribution, and 3) we have a small training set containing pairs of clean and noisy samples affected by real noise. In this context, we introduce a new meta-learning training approach for few-shot learning-based denoising problems. In our proposed training approach, the meta-training stage consists of learning how to quickly denoise from a limited number of synthetically generated noisy samples. Afterwards, the fine-tuning stage comprises the learning from the limited training set containing pairs of clean and noisy samples affected by real noise. To the best of our knowledge, this paper is the first to propose a meta-learning method for few-shot learning in denoising problems.

Traditionally, signal denoising has been addressed by non-learning approaches [13, 26, 38], where solutions have only made the assumption (2) above. However, learningbased works have shown more effective denoising results, when there is sufficient amount of training data [49]. Denoising with supervised learning has been explored in three different ways. First, one can directly train a denoising model based on a small training set (assumption 3 above), but current deep learning models are likely to over-fit such small training sets. Second, the denoising model can be trained with supervision using samples formed from synthetic noise (assumptions 1 and 2), while it is tested on the

real noise. The lack of fine-tuning on real noise typically results in low denoising accuracy [40]. Alternatively, in transfer learning, a denoising model is pre-trained with pairs of clean and synthetically generated noisy samples, where the noise is based on a known model (e.g. Gaussian or Poisson). Then, this pre-trained model is fine-tuned with the small training set containing real noise. Unfortunately, depending on the size of the small training set with real noise, transfer learning may not generalize well [46].

In this paper, we propose a new and effective metalearning approach for learning-based few-shot denoising problems. Although we share the same assumptions of transfer learning, our paper is motivated by the success of meta-learning in few-shot classification [22]. The most successful few-shot approaches are based on meta-learning, where the model learns to quickly adapt to new learning tasks using small training sets [32, 37]. We argue that the success of meta-learning is due to: 1) the ability to generate potentially infinite training sets containing synthetic noise (like transfer learning); and 2) the ability to train a model that quickly adapts to new learning tasks with small training sets (unlike transfer learning). In our experiments, we empirically show that the denoising results produced by meta-learning are better than the ones from transfer learning, and other non-learning and learning-based denoising approaches.

Weassess our work on three different data sets: low dose CT scans, natural images, and electrocardiogram (ECG) sequences. The demonstration of functionality in these diverse data sets shows the generalization ability of our proposed meta-denoising learning method, and allows us to compare it with relevant denoising methods. In all data sets, we rely on prior noise models for synthesizing noisy signals that partially capture the real noise affecting the image or signal. Our evaluations show that meta-learning outperforms supervised learning and transfer learning in few-shot denoising scenarios, given a sufficient number of learning tasks. Therefore, the main contribution of this paper is to be the first to show the potential of meta-learning to become a standard few-shot learning-based denoising algorithm.

## 2. Related Work

Traditional Denoising This type of denoising has been performed with signal processing algorithms. On the image domain, it is common to assume that the real noise is related to the Gaussian and Poisson noise models [25]. Similarly, the variance-stabilizing transformation has been proposed for converting the noise to Gaussian noise and then denoise with standard algorithms such as BM3D [13] or [7]. In signal denoising, wavelets and filtering are the usual algorithms for denoising [26, 34, 38]. However, recent works have shown that learning-based approaches outperform such traditional denoising algorithms [1, 49].

Learning-based Denoising Auto-encoders [41, 42] have standardized learning-based methods, followed by more complex network architectures [2, 44]. Convolutional neural networks [17, 33], multi-layer perceptron [8] and encoder-decoder [6, 23, 27, 49] architectures have also been proposed for image denoising. Unfortunately, since such methods are generally trained with supervision, they need large training sets of clean and noisy image pairs. However, as explained in Sec. 1, some denoising problems can have relatively small training sets and, consequently, be characterized as few-shot learning problem. Differently from previous learning-based denoising methods, our approach explicitly assumes that denoising is a few-shot learning problem. An alternative to be considered for solving this fewshot learning problem is transfer learning [5, 10], where the denoising model is pre-trained in some related denoising tasks (e.g., using large-scale data sets that may be available or building large-scale data sets by synthesizing noisy signals or images). Then, this pre-trained model is fine-tuned with the small training set. However, transfer learning does not usually generalize well from small data sets and requires substantial hyper-parameter tuning [46]. We show the transfer learning limitations in our evaluations, when compared with our meta-learning approach.

Meta-learning The motivation for meta-learning is to train a model that is optimized to adapt to new learning tasks (sampled from a common, but latent, distribution of tasks), with relatively small training sets [16, 29, 36, 39]. Meta-learning happens at two levels. First, the meta-learner obtains knowledge across multiple tasks (sampled from the latent distribution of tasks) in order to improve the base learner to adapt to new tasks. Second, the base learner acquires fast knowledge of a specific task using a small number of annotated data. Based on this scheme, several approaches have been proposed, which explore: metric learning [43, 21, 37], memory augmentation (i.e. model-based) [35, 28] and optimization [14, 32, 30]. Meta-learning approaches that explore optimization methods learn how to update the base learner with an iterative method [3, 16, 24]. For example, the meta-learner in [32] is an LSTM that learns the update rule for the base learner. Instead of the recurrent net, the meta-learner is modelled as batch stochastic gradient descent in model agnostic metalearning (MAML) [14] and Reptile [30]. Both approaches deliver state-of-the-art performance in few-shot classification. We have tried both algorithms for denoising and found that Reptile provides better training convergence and generalization than MAML. Furthermore, Reptile requires firstorder gradients, which is computationally more efficient than MAML that works with second-order gradients [30]. For those reasons, we focus on Reptile in our study of metalearning few-shot denoising algorithms.

To the best of our knowledge, this paper is the first to adapt meta-learning for the few-shot learning-based denoising problem.

## 3. Few-Shot Meta-Denoising Learning

We reformulate the optimization-based meta-learning method Reptile [30], which has been originally developed for few-shot classification, to work for few-shot image and signal denoising. Our meta-denoising approach is trained in two stages: 1) during meta-training, the tasks consist of small sets of synthesized noisy samples that have been formed by taking clean samples and applying synthetic noise sampled from known models; and then 2) a small training set of real noisy samples is used to fine-tune the denoising model. Note that the two stages listed above allow us to access an infinite number of tasks for the metatraining stage that can help in the learning of a generic denoising model that quickly adapts to new denoising tasks that contain small training sets.

## 3.1. Problem Definition

We define the clean signal as y ∈ R D and the corrupted signal as x ∈ R D , where this corruption occurs due to noise that is specific to the signal formation. At each step of metalearning, each clean signal is affected by a noise model, where task τ refers to the selected noise model and its parameters. For each task τ , sampled from a latent distribution of tasks p τ ( ) , we build a training set defined by {X τ , Y } τ , where X τ = { x i } k i =1 , Y τ = { y i } k i =1 , and x i = h τ ( y i ) , with h τ : R D → R D denoting the function that transforms the clean signal y i into the noisy signal x i by adding synthetic noise, according to the task τ . The data set containing real noise is represented by {X τ ∗ , Y τ ∗ } , where we assume that τ ∗ ∼ p τ ( ) , and this set is divided into a training and a test set, denoted by {X ( ) t τ ∗ , Y ( ) t τ ∗ } and {X ( v ) τ ∗ , Y ( v ) τ ∗ } , respectively.

## 3.2. Denoising Objective

The objective is to learn the parameters θ of a deep neural network that recovers the clean signal. The network is represented by f θ : R D → R D and is trained with backpropagation. Given a large training set of pairs of clean and noisy samples, the objective to minimize is

$$\mathcal { L } ( \theta, \boldsymbol x, \boldsymbol y ) = \mathbb { E } _ { \boldsymbol x, \boldsymbol y } [ f _ { \theta } ( \boldsymbol x ) - \boldsymbol y ] ^ { 2 }. \quad \quad ( 1 )$$

This is the standard configuration for gradient-based supervised learning, which tends to work well when there is sufficient amount of training data. Since we assume that the available pairs of clean and noisy samples are limited, we define the problem of denoising within the context of fewshot meta-learning, which we refer to as meta-denoising .

## 3.3. Meta-denoising

We define the base learner to be the network f θ ( ) . . The learning process estimates the parameter θ that can be quickly adapted to denoising real signals, given a small training set, containing few pairs of clean and noisy signals. In the context of meta-denoising, a task corresponds to denoising pairs of clean and synthesized signals, formed by applying a known noise model to the clean signals. Note that a task τ is formed not only by noise models, but also by noise parameters. Given a sampled task τ i ∼ p τ ( ) , Reptile performs s steps of gradient descent update on the loss from (1), described by:

$$\theta _ { i } ^ { \prime } = g ( \mathcal { L } ( \theta, h _ { \tau } ( \mathbf y ), \mathbf y ), \theta, s ), \text{ \quad \ \ } ( 2 )$$

where g . ( ) is the update operator that represents stochastic gradient descent or another optimization algorithm. This is the inner-loop update. After iterating over n tasks, the model parameters θ are adapted towards the new parameters, defined as:

$$\theta \leftarrow \theta + \epsilon \frac { 1 } { n } \sum _ { i = 1 } ^ { n } ( \theta _ { i } ^ { \prime } - \theta ),$$

where glyph[epsilon1] is the step-size. This is the outer loop of the algorithm, where across tasks knowledge is acquired. Note that instead of being explicitly modeled, the meta-learner is implicitly represented by the gradient-based learning in (3). The number of iterations for the inner and outer loops and the batch-size of the inner loop compose the hyperparameters of the learning algorithm. At the end of the learning process, the model is fine-tuned with {X ( ) t τ ∗ , Y ( ) t τ ∗ } and tested with {X ( v ) τ ∗ , Y ( v ) τ ∗ } - this is the only step that involves annotated samples, i.e. pairs of clean and noisy samples, where the noise is real. The complete learning and assessment processes are presented in Algorithm 1.

Algorithm 1: Meta-Denoising Training.

- 1: Initialize θ
- 2: for iteration = 1 2 , , . . . do
- 3: for task = 1 2 , , . . . , n do
- 4: Sample noise model τ ∼ p τ ( )
- 5: Sample k clean samples Y τ = { y i } k i =1
- 7: Compute θ ′ task = g ( L ( θ, X τ , Y τ , θ, s )
- 6: Generate k noisy samples X τ = { h τ ( y i ) } k i =1
- 8: end for
- 9: Update θ ← θ + glyph[epsilon1] 1 n ∑ n task =1 ( θ ′ task -θ )
- 10: end for
- 11: Train f θ ( x ) with {X ( ) t τ ∗ , Y ( ) t τ ∗ }
- 12: Test f θ ( x ) with {X ( v ) τ ∗ , Y ( v ) τ ∗ } .

|                     | # Tasks   | # Tasks   | Tube current   | Tube current   |
|---------------------|-----------|-----------|----------------|----------------|
|                     | Poisson   | Gaussian  | 10%            | 5%             |
| Initial Noise       | -         | -         | 38.28          | 35.18          |
| Supervised Learning | 4         | 0         | 36.50          | 34.15          |
| Supervised Learning | 4         | 4         | 35.50          | 33.80          |
| Supervised Learning | 8         | 0         | 36.31          | 33.94          |
| Supervised Learning | 8         | 8         | 35.01          | 33.42          |
| Transfer Learning   | 4         | 0         | 38.98          | 36.96          |
| Transfer Learning   | 4         | 4         | 37.91          | 35.99          |
| Transfer Learning   | 8         | 0         | 38.39          | 36.70          |
| Transfer Learning   | 8         | 8         | 39.19          | 36.76          |
| Meta-Denoising      | 4         | 0         | 39.77          | 37.37          |
| Meta-Denoising      | 4         | 4         | 39.42          | 37.50          |
| Meta-Denoising      | 8         | 0         | 39.99          | 37.96          |
| Meta-Denoising      | 8         | 8         | 39.84          | 37.49          |

Table 1: CT-Scan Evaluation. We examine two types of noise models, namely Gaussian and Poisson. We present the results for Gaussian and / or Poisson noise models. We evaluate on tube current with dose 10% and 5% . The evaluation metric is the peak signal-to-noise ratio (PSNR), where larger is better. Bold numbers indicate the best performance per column.

|                     | Tube current   | Tube current   | Execution (sec.)   |
|---------------------|----------------|----------------|--------------------|
|                     | 10%            | 5%             |                    |
| Initial Noise       | 38.28          | 35.18          | -                  |
| BM3D                | 38.70          | 36,27          | 2.42               |
| LGP-PCA             | 38.98          | 36,29          | 206.84             |
| Supervised Learning | 36.50          | 34.15          | 0.0008             |
| Transfer Learning   | 39.19          | 36.96          | 0.0008             |
| Meta-Denoising      | 39.99          | 37.96          | 0.0008             |

Table 2: Comparison with Non-learning Algorithms on CTScan. We compare the three learning-based algorithms with two classic denoising approaches (BM3D and LGPPCA). The evaluation metric is the peak signal-to-noise ratio (PSNR), where larger is better. The execution time is shown in seconds. Bold numbers indicate the best performance per column.

## 4. Experiments

We compare the proposed meta-denoising few-shot learning algorithm with other non-learning and learningbased methods. We assess our few-shot meta-denoising approach in two image-based datasets and one 1-D dataset to demonstrate the generalization of our approach to different types of signals. Since this paper is the first to explore fewshot denoising with meta-learning, we propose a new assessment protocol.

Protocol. For meta-training, noise models are sampled and synthetic tasks are generated as a k-shot denoising problem, where k-pairs of clean and noisy samples are produced. During meta-testing, real tasks are used for k-shot finetuning. The evaluation is performed on the test set, which is composed of previously unseen pairs of clean and noisy samples from the real tasks. The evaluation metric is the peak signal-to-noise ratio (PSNR) for images and signalto-noise ratio (SNR) for 1-D data. These are the standard metrics for evaluating denoising methods [18, 44].

Other Training Algorithms. We define two baselines for comparison with our approach. First, we train from scratch a supervised learning model, where the training set is composed of synthetically generated data and is as large as the training set of all tasks in meta-training. As a result, the amount of data processed by supervised learning is the same as by meta-training. Second, we perform transfer learning with the trained model using the same k-pairs of clean and real noisy samples that we employ for fine-tuning in meta-training. By fixing the model and data, we can compare meta-learning with supervised- and transfer-learning in a fair manner. Finally, we also report the initial noise for each evaluation that corresponds to the test set noise prior to applying a denoising algorithm.

Implementation. We present three experiments with different data types, so we employ different network architectures, optimization methods and hyper-parameter values. All employed architectures are taken from related publications, since proposing a new architecture is not the aim of this paper. The hyper-parameters have been chosen with grid-search optimization. The implementation of our approach, as well as the evaluation protocol will be made publicly available - both implementations were made using PyTorch [31].

Below, for each data set, we discuss the task generation, implementation details and denoising results. Finally, we examine the results significance using the t-test, different k-shots and we also provide results when training with the MAMLalgorithm.

## 4.1. CT-Scan Denoising Evaluation

Task Generation. The CT database in [45] contains CT scans of a deceased piglet. The scans are produced using a normal dose of tube current (100%) and reduced doses. We use the doses of 10% and 5% because they include substantial amount of noise, where each dose set has 850 images of size D = 512 × 512 . Following the data set protocol [45], 708 of those images are selected for training and 108 for testing. During training, each task consists of synthetic images generated by simulating reduced dose CT images by adding Poisson and Gaussian noise to the sinograms of the normal dose CT images, as in x = h τ = P ( b , ) N (0 ,σ 2 ) ( y ) , where x = S -1 ( z ) ,

(a) Ground-truth.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00111v2-figure-bcc8603f-8517-487a-8226-c3be501b0416.jpg)

(b) Noisy Input.

(c) Denoised Output.

Figure 1: CT-Scan Denoising. We show two examples of results (top and bottom rows) on CT-Scan [45] denoising dataset, where the columns (from left to right) show the clean image, noisy image, and denoised image produced by our approach. We also show a magnified visualization for each case.

z ∼ P ( b × exp( -S ( y ))) + r , with P ( ) . denoting the Poisson distribution, S . ( ) representing the process to generate the sinogram or the fan-beam projected data [48] from the clean CT image y r , ∼ N (0 , 1) denoting the read-out electronic noise represented by N ( ) . the Gaussian distribution with mean µ and variance σ , and b denoting the X-ray source intensity or a blank scan factor that controls the noise level. We create the tasks with r = 0 , namely one per blank scan factor, where we use the following factors b ∈ { 10 4 , 3 × 10 4 , 5 × 10 4 , 7 × 10 4 , 10 5 , 2 × 10 5 , 3 × 10 5 , activation functions are all ReLU, except for the last layer that is linear. The optimization algorithm is Adam [19] with learning rate 10 -4 . The inner loop has 20 epochs with minibatch size 20. The outer loop runs for 1000 epochs with step-size glyph[epsilon1] = 0 1 . and the number of fine-tuning samples k = 10 .

5 × 10 5 } . Consequently, each factor contributes to the generation of a different task. To simulate the effect of the Gaussian noise, we add noise directly on the CT image x . The Gaussian task τ is represented by Gaussian noise η i, j ( ) ∼ N (0 , σ 2 ) with σ 2 ∼ U (0 001 . , 0 003) . , where U a, b ( ) denoting uniform distribution in [ a, b ] . The result is the noisy image x ( i, j ) = y ( i, j ) + η i, j ( ) for i ∈ { 1 , ..., D height } and j ∈ { 1 , ..., D width } . Finally, we keep k images for the meta-learning for each reduced dose set. The rest of the images are used for evaluation.

Model. We use a residual encoder-decoder architecture from [12] that has been proposed for low dose CT denoising [45]. The network consists of a 10-layer convolutional neural network with input and output sizes of 55 × 55 . The

Results. We evaluate CT-Scan denoising on tube current with doses 10% and 5% , where smaller percentage denotes more noise - results are summarized in Table 1. We generate from 4 to 8 Gaussian and Poisson tasks, with each task representing an individual noise model. Notice that we also applied Gaussian and Poisson noise, at the same time, on an image as another task, but the performance was not as good as sampling independently. In the case of task combination (e.g. 4 Gaussian and 4 Poisson), we equally sample tasks from both noise models. We attempted to generate less than 4 and more than 8 tasks, but there was not a considerable performance difference.

The same evaluation is performed for the supervisedand transfer-learning. Comparing with these two algorithms, it is clear that our approach delivers the highest PSNR. Although the results from transfer learning are closer, the difference is significant based on the t-test as we show in Sec. 4.4. Since this is the only experiment where we have access to a relatively large training set containing pairs of clean and real noisy data, we check the upper bound performance for our approach, which is the supervised learning performance using this large training set. Note that such full supervision violates one of our three assumptions mentioned in Sec. 1 - in particular, the one regarding the availability of a large number of training samples. Supervised learning achieves 40.28dB for 10% and 38.84dB for 5% , when trained with clean and real noisy training data. Our results in Table 1 indicate that we are close to this upper bound, although we train with synthetic training data. To further explore our approach, we meta-train it with clean and real noisy training data. Notably, our meta-denoising reaches 40.47dB for 10% and 38.31dB for 5% . This result shows the potential of meta-learning to become the standard training algorithm for the task of denoising. We present visual results in Fig. 1.

Furthermore, we investigate how our method and the other learning-based algorithms from Table 1 compare to non-learning denoising algorithms. In Table 2, we present the best results from all learning-based approaches and compare with the non-learning approaches BM3D [13] and LGP-PCA [50]. It is clear that our meta-denoising, as well as, supervised- and transfer-learning outperform the nonlearning approaches. Moreover, BM3D and in particular LGP-PCA are significantly slower in terms of execution time.

## 4.2. Natural Images Evaluation

Task Generation. We use the color data set BSD500 [4] as training set and evaluate on the SIDD database [1]. BSD500 data set is standard for image denoising [27, 49], while the SIDD has been recently introduced for denoising images from smartphones. The training and testing sets from BSD500 have 200 images which we employ as training data during meta-training. SIDD has 160 images from which we keep 10 for the fine-tuning stage and the rest as test set. In BSD500, the size of images is D = 481 × 321 × 3 . In the training images, we add Gaussian noise in the spatial domain to generate tasks with synthetic noise, as in x = h τ = N ( µ,σ ) ( y ) . This Gaussian noise has mean µ = 0 and standard deviation σ ∈ { 15 25 50 , , } , which have been chosen from the literature in image denoising [44, 49]). Second, we introduce Poisson noise in the image domain as in x = h τ = P ( b ) ( y ) , where x ∼ P ( b × 10 12 × x ) / 10 12 , with P ( ) . denoting the Poisson distribution, and b ∈ { 10 9 , 10 9 5 . , 10 10 } denotes the scale factor that controls the noise level. In total, we have three tasks for Poisson and three for Gaussian noise.

Model. The network is DnCNN [49], an encoder-decoder ConvNet that reconstructs the residual noisy image. We extend the original architecture from [49] to have a depth of 13 layers to reduce the amount of resources required. The de-

Table 3: Natural Image Evaluation. We evaluate on the smartphone dataset [1] our meta-denoising approach, supervised- and transfer- learning. The evaluation metric is the peak signal-to-noise ratio (PSNR) and tasks are generated from a Gaussian and Poisson distributions. The best result per column is in bold font.

|                   | # Tasks   | # Tasks   | PSNR   | PSNR   |
|-------------------|-----------|-----------|--------|--------|
|                   | Gaussian  | Poisson   | DnCNN  | xDnCNN |
| Initial Noise     | -         | -         | 27.61  | 27.61  |
| Supervised        | 3         | 0         | 22.23  | 26.71  |
| Learning          | 3         | 3         | 25.74  | 26.25  |
| Transfer Learning | 3         | 0         | 28.54  | 29.11  |
| Transfer Learning | 3         | 3         | 30.01  | 28.30  |
| Meta-Denoising    | 3         | 0         | 29.88  | 29.58  |
| Meta-Denoising    | 3         | 3         | 31.71  | 29.51  |

noised signal is obtained by subtracting the predicted noise from the input noisy image. All layers of this network consists of convolutions, with batch normalization and ReLU activation, except the last layer, which is linearly activated. The input and output resolution image resolution is 55 × 55 during training, while 40 × 40 in testing. In addition, we evaluate a more recent version of the architecture DnCNN, xDnCNN [20] which uses the new activation function xUnit adapted for image restoration scenarios. The optimizer for the inner loop is Adam with learning rate 10 -3 , with 20 inner epochs. The step-size of the outer loop is glyph[epsilon1] = 1 e -4 and the number of samples k = 10 .

Results. Although the noise of natural images is more complicated than the proposed Gaussian and Poisson tasks, we are able to denoise images with real noise [1] - Table 3 summarizes the results. We have used three Gaussian and Poisson tasks jointly, but independently. The results in Table 3 indicate that the supervised learning performs poorly because the generated tasks may not represent well the real noise in natural images. However, the tasks are enough for meta-learning, as shown by the results.

Similarly to the CT-Scans, we compare all learningbased algorithms to non-learning denoising approaches. In Table 4, we present the best results of meta-denoising and the other two learning algorithms (from Table 3), next to BM3D [13] and LGP-PCA [50]. In this evaluation, the non-learning algorithms perform considerably better at the cost of a significantly longer execution time. This finding suggests a limitation of learning-based denoising methods, which is that they depend on training sets that represent well the noise seen in the evaluation data set. A few visual results of our approach are presented in Fig. 2 and Fig. 3.

## 4.3. ECG Denoising Evaluation

Task Generation. We use the Physionet ECG-ID database [15], which includes 310 ECG records from 90

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00111v2-figure-c969591e-98e2-4234-a3bc-f466cdbf1775.jpg)

(a)

Ground-truth.

(b)

Noisy Input.

(c) Denoised Output.

Figure 2: Natural Image Results. We show two examples of results (top and bottom rows) on natural image [1] denoising, where the columns (from left to right) show the clean image, noisy image, and denoised image produced by our approach. We also provide magnified visualization for each case.

Table 4: Comparison with Non-learning Algorithms on Natural Images. We compare the three learning-based algorithms with two classic non-learning denoising approaches (BM3D and LGP-PCA). The execution time in seconds is reported too. The best result per column is in bold font.

|                     |   PSNR | Execution (sec.)   |
|---------------------|--------|--------------------|
| Initial Noise       |  27.61 | -                  |
| BM3D                |  34.21 | 273.7              |
| LGP-PCA             |  32.41 | 11009              |
| Supervised Learning |  26.71 | 0.001              |
| Transfer Learning   |  30.01 | 0.001              |
| Meta-Denoising      |  31.71 | 0.001              |

ing set for generating the synthetic tasks, where in each task, we add Gaussian noise to the clean training samples, as in x = h τ = N ( µ,σ ) ( y ) , where h τ ( ) . is defined by x ( j ) = y ( j ) + η j ( ) for j ∈ { 1 , ..., D } , τ is represented by a Gaussian noise η j ( ) ∼ N ( µ, σ ) , with µ ∼ U ( -0 1 . , 0 1) . and standard deviation σ ∼ U (0 , 0 3) . for U a, b ( ) denoting uniform distribution in [ a, b ] and N ( ) . representing the Gaussian distribution with mean µ and variance σ . We generate the training samples per task, where each sample is represented by a window of D = 30 consecutive signal measurements with stride one. To perform fine-tuning for transfer learning or meta-learning, we randomly draw k samples from the test set and use the rest for evaluation.

subjects. Each record contains the raw noisy ECG signal and the filtered version that represents the clean signal. The ECG signal is often corrupted by power line interference, contact noise and motion artefacts. We choose 217 ECG records (70% of the sequences) for training, 62 (20%) for validation and 31 (10%) for testing. The total number of samples is around 385k. We use the train-

Model. The network is a denoising autoencoder [11], consisting of 3 encode and 3 decoder linear layers with 150 units, and a latent representation layer with 25 units [11]. The activation function is ReLU, which is applied in all layers, except the last one that uses linear activation. The input and output for the model consists of a 30-element vector. During meta-training, we use the AdaDelta [47] optimization for the inner loop (i.e. , per task optimization). AdaDelta showed a more stable performance than stochastic gradient descent. The learning rate is 1.5 and the number of epochs is 10. For the outer loop, we iterate for 800

Table 5: We evaluate our approach on ECG sequences. The supervised-learning is trained on the same data as our approach. The transfer-learning has been fine-tuned on the same k-shot data that we use during fine-tuning. The evaluation metric is the signal-to-noise ratio (SNR), larger is better. Best result is shown in bold font.

|                | # Tasks   |   SNR(db) |
|----------------|-----------|-----------|
| Initial Noise  | -         |     -6.16 |
| Supervised     | 50        |     -5.8  |
| Learning       | 100       |     -5.45 |
| Transfer       | 50        |      0.53 |
| Learning       | 100       |      2.82 |
| Meta-Denoising | 50        |      2    |
| Meta-Denoising | 100       |      2.97 |

epochs with step-size glyph[epsilon1] = 0 01 . . The number of samples for few-shot evaluation is k = 10 .

Results. The results are summarized in Table 5. We evaluate for 50 and 100 generated tasks using the Gaussian noise model. We experimentally found that a much larger number of tasks is necessary for this experiment, compared to image-based denoising. The same experiments have been conducted for supervised- and transfer-learning. We observe that increasing the number of tasks improves the performance almost by 1dB. Generating more tasks for 10-shot learning did not have a large impact on the overall performance of our model. Comparing to the other two approaches, we see that only transfer learning achieves considerable denoising and is closer to our results. We also compare our approach with a non-learning algorithm, as with the earlier experiments, where we evaluate a Wavelets approach [38], resulting in -5.90dB, which is much worse than our approach. Here, learning-based denoising produces more accurate results than non-learning approaches.

## 4.4. Statistical Significance

To examine the statistical significance of our approach, weconduct one-tailed paired t-test for all experiments, comparing our approach to supervised and transfer learning. For all experiments, the test results show that under the significance level of 10 -3 , the performance of our approach is always significantly better.

## 4.5. K-shot Learning

We further study the performance of our approach under different k-shots of learning. The goal is to observe the performance drop when reducing the number fine-tuning samples k , which is a common experiment for few-shot classification. We selected as reference the CT-Scan data set with eight Poisson tasks, since this model achieve the best performance as it is shown in Table 1. We have trained this

Table 6: K-shot learning experiment. We evaluate our approach and transfer learning for 1-, 3-, 5-, 7- and 10-shot learning on the CT-Scan database. The evaluation metric is PSNR. Best result per column is shown in bold font.

|                   | k-shot   | 10%   | 5%    |
|-------------------|----------|-------|-------|
| Initial Noise     | -        | 38.28 | 35.18 |
| Transfer Learning | 1        | 36.82 | 34.94 |
| Transfer Learning | 3        | 38.43 | 36.21 |
| Transfer Learning | 5        | 39.28 | 36.79 |
| Transfer Learning | 7        | 38.87 | 36.88 |
| Transfer Learning | 10       | 38.98 | 36.96 |
| Meta-Denoising    | 1        | 35,94 | 33,82 |
| Meta-Denoising    | 3        | 38.90 | 36.90 |
| Meta-Denoising    | 5        | 39.30 | 37.40 |
| Meta-Denoising    | 7        | 39.72 | 37.87 |
| Meta-Denoising    | 10       | 39.99 | 37.96 |

approach for 1-, 3-, 5-, 7- and 10-shot learning, and followed the same protocol for the transfer learning. Table 6 summarizes the results. It is clear that larger values for k improves the results. Interestingly, we are able to denoise even with 3-shot learning. The same conclusion holds for transfer learning, where we chose the model with four Poisson tasks as reference. By comparing the two algorithms, we see that our method always reaches higher PSNR for both dose tube currents.

## 4.6. Meta-Learning with MAML

We have considered MAML [14] as a second metalearning algorithm for the few-shot denoising problem. We have observed though that it is less stable during training and significantly slower in terms of convergence. MAML training has been at least four times slower than Reptile independently of the amount of generated tasks. Finally, it does not generalize as well as Reptile. In particular, we have again considered the CT-Scan data set where MAML performs at best when using eight Gaussian tasks. The PSNR is 35.08 for 5% tube current and 37.68 for for 10% tube current. These results are far behind Reptile results in Table 1.

## 5. Conclusion

We have introduced few-shot meta-learning for denoising. Our approach applies gradient-based meta-learning to signal denoising based on synthetically generated tasks. In our evaluation, we explored two image- and a sequencebased problems, where we demonstrate that: 1) metalearning is effective given sufficient amount of synthetic tasks; 2) meta-learning works better than supervised- and transfer-learning under the assumptions listed in Sec. 1; 3) meta-learning can produce competitive denoising results, compared with a fully supervised training with real data;

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00111v2-figure-80ed6571-9a62-413c-bdf6-2c6d1d7d33c6.jpg)

(a)

Ground-truth.

(b)

Noisy Input.

(c)

Supervised Learning.

(d)

Transfer Learning.

(e)

Meta-Learning (Ours).

Figure 3: More Natural Image Results. We provide additional image patches with visual comparison of supervised learning (c), transfer learning (d) and our approach based on meta-learning (e).

4) it is more accurate than non-learning approaches, when the synthetic noise represents well the (latent) noise model used to form the real noisy image; and 5) learning-based denoising inference is significantly faster than non-learning models. Our study provides strong indication that metalearning has the potential to become the standard learning algorithm for denoising.

## References

- [1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smartphone cameras. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018. 1, 2, 6, 7
- [2] Forest Agostinelli, Michael R Anderson, and Honglak Lee. Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS , 2013. 2
- [3] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems , pages 3981-3989, 2016. 2
- [4] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence , 33(5):898-916, 2011. 6
- [5] Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning , pages 17-36, 2012. 2
- [6] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 11036-11045, 2019. 2
- [7] Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , volume 2, pages 60-65. IEEE, 2005. 2
- [8] Harold C Burger, Christian J Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with bm3d? In CVPR , 2012. 2
- [9] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. The euroc micro aerial vehicle datasets. The International Journal of Robotics Research , 2016. 1
- [10] Rich Caruana. Learning many related tasks at the same time with backpropagation. In Advances in neural information processing systems , pages 657-664, 1995. 2
- [11] Leslie Casas, Nassir Navab, and Vasileios Belagiannis. Adversarial signal denoising with encoder-cecoder networks. arXiv preprint arXiv:1812.08555 , 2018. 7
- [12] Hu Chen, Yi Zhang, Mannudeep K Kalra, Feng Lin, Yang Chen, Peixi Liao, Jiliu Zhou, and Ge Wang. Low-dose ct with a residual encoder-decoder convolutional neural network. IEEE transactions on medical imaging , 36(12):25242535, 2017. 5
- [13] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transformdomain collaborative filtering. IEEE Transactions on image processing , 16(8):2080-2095, 2007. 1, 2, 6
- [14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Modelagnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 , 2017. 2, 8
- [15] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet. Circulation , 2000. 6
- [16] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks , pages 87-94. Springer, 2001. 2
- [17] Viren Jain and Sebastian Seung. Natural image denoising with convolutional networks. In NIPS , 2009. 2
- [18] Md Ashfanoor Kabir and Celia Shahnaz. Denoising of ecg signals based on noise reduction algorithms in emd and wavelet domains. Biomedical Signal Processing and Control , 7(5):481-489, 2012. 4
- [19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 5
- [20] Idan Kligvasser, Tamar Rott Shaham, and Tomer Michaeli. xunit: learning a spatial activation function for efficient image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 24332442, 2018. 6
- [21] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop , volume 2, 2015. 2
- [22] Brenden M Lake, Ruslan R Salakhutdinov, and Josh Tenenbaum. One-shot learning by inverting a compositional causal process. In Advances in neural information processing systems , pages 2526-2534, 2013. 2
- [23] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. arXiv preprint arXiv:1512.09300 , 2015. 2
- [24] Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885 , 2016. 2
- [25] Florian Luisier, Thierry Blu, and Michael Unser. Image denoising in mixed poisson-gaussian noise. IEEE Transactions on image processing , 20(3):696-708, 2011. 2
- [26] S. O. H. Madgwick, A. J. L. Harrison, and R. Vaidyanathan. Estimation of imu and marg orientation using a gradient descent algorithm. In 2011 IEEE International Conference on Rehabilitation Robotics , pages 1-7, June 2011. 1, 2
- [27] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS . 2016. 2, 6
- [28] Tsendsuren Munkhdalai and Hong Yu. Meta networks. arXiv preprint arXiv:1703.00837 , 2017. 2
- [29] Devang K Naik and RJ Mammone. Meta-neural networks that learn by learning. In Neural Networks, 1992. IJCNN., International Joint Conference on , volume 1, pages 437-442. IEEE, 1992. 2
- [30] Alex Nichol, Joshua Achiam, and John Schulman. On firstorder meta-learning algorithms. CoRR, abs/1803.02999 , 2018. 2, 3
- [31] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 4
- [32] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016. 2
- [33] Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5069-5073. IEEE, 2018. 2
- [34] A. M. Sabatini. Quaternion-based extended kalman filter for determining orientation by inertial and magnetic sensing. IEEE Transactions on Biomedical Engineering , 53(7):13461356, July 2006. 2
- [35] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850, 2016. 2
- [36] J¨rgen Schmidhuber. u Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4(1):131-139, 1992. 2
- [37] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , pages 4077-4087, 2017. 2
- [38] Mallat St´ ephane. Front matter. In A Wavelet Tour of Signal Processing (Third Edition) , pages iii -. Academic Press, Boston, third edition edition, 2009. 1, 2, 8
- [39] S Thrun. Learning to learn: Introduction. In In Learning To Learn . Citeseer, 1996. 2
- [40] Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting deep visuomotor representations with weak pairwise constraints. arXiv preprint arXiv:1511.07111 , 2015. 2
- [41] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning , pages 1096-1103. ACM, 2008. 2
- [42] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. , 11:3371-3408, Dec. 2010. 1, 2
- [43] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems , pages 3630-3638, 2016. 2
- [44] Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural networks. In NIPS . 2012. 2, 4, 6
- [45] Xin Yi and Paul Babyn. Sharpness-aware low-dose ct denoising using conditional generative adversarial network. Journal of digital imaging , pages 1-15, 2018. 1, 4, 5
- [46] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems , pages 3320-3328, 2014. 2
- [47] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 , 2012. 7
- [48] Dong Zeng, Jing Huang, Zhaoying Bian, Shanzhou Niu, Hua Zhang, Qianjin Feng, Zhengrong Liang, and Jianhua Ma. A simple low-dose x-ray ct simulation from high-dose scan. IEEE transactions on nuclear science , 62(5):22262233, 2015. 1, 5
- [49] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of
- deep cnn for image denoising. IEEE Transactions on Image Processing , 26(7):3142-3155, 2017. 1, 2, 6
- [50] Lei Zhang, Weisheng Dong, David Zhang, and Guangming Shi. Two-stage image denoising by principal component analysis with local pixel grouping. Pattern recognition , 43(4):1531-1549, 2010. 6
Leslie Casas Attila Klimmek Gustavo Carneiro Nassir Navab Vasileios Belagiannis
Few-Shot Meta-Denoising
We study the problem of few-shot learning-based denoising where the training
set contains just a handful of clean and noisy samples. A solution to mitigate
the small training set issue is to pre-train a denoising model with small
training sets containing pairs of clean and synthesized noisy signals, produced
from empirical noise priors, and fine-tune on the available small training set.
While such transfer learning seems effective, it may not generalize well
because of the limited amount of training data. In this work, we propose a new
meta-learning training approach for few-shot learning-based denoising problems.
Our model is meta-trained using known synthetic noise models, and then
fine-tuned with the small training set, with the real noise, as a few-shot
learning task. Meta-learning from small training sets of synthetically
generated data during meta-training enables us to not only generate an infinite
number of training tasks, but also train a model to learn with small training
sets -- both advantages have the potential to improve the generalisation of the
denoising model. Our approach is empirically shown to produce more accurate
denoising results than supervised learning and transfer learning in three
denoising evaluations for images and 1-D signals. Interestingly, our study
provides strong indications that meta-learning has the potential to become the
main learning algorithm for denoising.