## CRNet: Image Super-Resolution Using A Convolutional Sparse Coding Inspired Network

## Menglei Zhang, Zhou Liu, Lei Yu School of Electronic and Information, Wuhan University, China

{ zmlhome, liuzhou, ly.wd @whu.edu.cn }

## Abstract

Convolutional Sparse Coding (CSC) has been attracting more and more attention in recent years, for making full use of image global correlation to improve performance on various computer vision applications. However, very few studies focus on solving CSC based image Super-Resolution (SR) problem. As a consequence, there is no significant progress in this area over a period of time. In this paper, we exploit the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and it can be implemented using CNN architectures. Then we develop a novel CSC based SR framework analogy to the traditional SC based SR methods. Two models inspired by this framework are proposed for pre-/post-upsampling SR, respectively. Compared with recent state-of-the-art SR methods, both of our proposed models show superior performance in terms of both quantitative and qualitative measurements.

## 1. Introduction

Single Image Super-Resolution (SISR), which aims to restore a visually pleasing High-Resolution (HR) image from its Low-Resolution (LR) version, is still a challenging task within computer vision research community [34, 36]. Since multiple solutions exist for the mapping from LR to HR space, SISR is highly ill-posed. To regularize the solution of SISR, various priors of natural images have been exploited, especially the current leading learning-based methods [41, 6, 22, 15, 16, 32, 33, 21, 1, 11, 20, 50] are proposed to directly learn the non-linear LR-HR mapping.

By modeling the sparse prior in natural images, the Sparse Coding (SC) based methods for SR [46, 47, 44] with strong theoretical support are widely used owing to their excellent performance. Considering the complexity in images, these methods divide the image into overlapping patches and aim to jointly train two over-complete dictionaries for LR/HR patches. There are usually three steps in

Figure 1: PSNRs of recent state-of-the-arts for scale factor × 4 on Set5 [2] and Urban100 [14]. Red names represent our proposed models.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-8709a554-c1d9-41f4-a4c8-3275af96d0f3.jpg)

these methods' framework. First, overlapping patches are extracted from input image. Then to reconstruct the HR patch, the sparse representation of LR patch can be applied to the HR dictionary with the assumption that LR/HR patch pair shares similar sparse representation . The final HR image is produced by aggregating the recovered HR patches.

Recently, with the development of Deep Learning (DL), many researchers attempt to combine the advantages of DL and SC for image SR. Dong et al . [6] firstly proposed the seminal CNN model for SR termed as SRCNN, which exploits a shallow convolutional neural network to learn a nonlinear LR-HR mapping in an end-to-end manner and dramatically overshadows conventional methods [47, 35]. However, sparse prior is ignored to a large extent in SRCNN for it adopts a generic architecture without considering the domain expertise. To address this issue, Wang et al . [41] implemented a Sparse Coding based Network (SCN) for image SR, by combining the merits of sparse coding and deep learning, which fully exploits the approximation of sparse coding learned from the LISTA [9] based sub-network.

Its worth to note that most of SC based methods utilize the sparse prior locally [28], i.e ., coping with overlapping image patches. Thus the consistency of pixels in overlapped patches has been ignored [10, 28]. To address this issue, CSC is proposed to serve sparse prior as a global prior [48,

Figure 2: Visual comparisons between our model CRNetB and other state-of-the-art methods on 'img004' from Urban100 [14] for scale factor × 4

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-778a5470-8ec5-4c96-9b74-d617f119405d.jpg)

28, 27] and it furnishes a way to fill the local-global gap by working directly on the entire image by convolution operation. Consequently, CSC has attained much attention from researchers [48, 4, 13, 10, 30, 7]. However, very few studies focus on the validation of CSC for image SR [10], resulting in no work been reported that CSC based image SR can achieve state-of-the-art performance. Can CSC based image SR show highly competitive results with recent stateof-the-art methods [6, 22, 15, 16, 32, 33, 37, 21, 20, 50]? To answer this question, the following issues need to be considered:

Framework Issue . Compared with SC based image SR methods [46, 47], the lack of a unified framework has hindered progress towards improving the performance of CSC based image SR.

Optimization Issue . The previous CSC based image SR method [10] contains several steps and they are optimized independently. Hundreds of iterations are required to solve the CSC problem in each step.

Memory Issue . To solve the CSC problem, ADMM [3] is commonly employed [4, 42, 13, 43, 7], where the whole training set needs to be loaded in memory. As a consequence, it is not applicable to improve the performance by enlarging the training set.

Multi-Scale Issue . Training a single model for multiple scales is difficult for the previous CSC based image SR method [10].

Based on these considerations, in this paper, we attempt to answer the aforementioned question. Specifically, we exploit the advantages of CSC and the powerful learning ability of deep learning to address image SR problem. Moreover, massive theoretical foundations for CSC [28, 27, 8]

make our proposed architectures interpretable and also enable to theoretically analyze our SR performance. In the rest of this paper, we first introduce CISTA, which can be naturally implemented using CNN architectures for solving the CSC problem. Then we develop a framework for CSC based image SR, which can address the Framework Issue . Subsequently, CRNet-A (CSC and Residual learning based Network) and CRNet-B inspired by this framework are proposed for image SR. They are classified as pre- and postupsampling models [40] respectively, as the former takes Interpolated LR (ILR) images as input while the latter processes LR images directly. By adopting CNN architectures, Optimization Issue and MemoryIssue would be mitigated to some extent. For Multi-Scale Issue , with the help of the recently introduced scale augmentation [15, 16] or scalespecific multi-path learning [21, 40] strategies, both of our models are capable of handling multi-scale SR problem effectively, and achieve favorable performance against stateof-the-arts, as shown in Fig. 1.

- · We introduce CISTA, which can be naturally implemented using CNN architectures for solving the CSC problem.

The main contributions of this paper include:

- · A novel framework for CSC based image SR is developed. Two models, CRNet-A and CRNet-B, inspired by this framework are proposed for image SR.
- · The differences between our proposed models and several SR models with recursive learning strategy, e.g ., DRRN [32], SCN [41], DRCN [16], are discussed.
- · Experimental results demonstrate our proposed models outperform the previous CSC based image SR method [10] by a large margin and show superior performance against recent state-of-the-arts, e.g ., EDSR/MDSR [21], RDN [50], as depicted in Fig. 2.

## 2. Related Work

## 2.1. Sparse Coding for Image Super-Resolution

Sparse coding has been widely used in a variety of applications [51]. As for SISR, Yang et al . [46] proposed a representative Sparse coding based Super-Resolution (ScSR) method. In the training stage, ScSR attempts to learn the LR/HR overcomplete dictionary pair D D l / h jointly by given a group of LR/HR training patch pairs x x l / h . In the test stage, the HR patch x h is reconstructed from its LR version x l by assuming they share the same sparse code. Specifically, the optimal sparse code is obainted by minimizing the following sparsity-inducing glyph[lscript] 1 -norm regularized objective function

$$z ^ { * } = \arg \min _ { z } \| x _ { l } - D _ { l } z \| _ { 2 } ^ { 2 } + \lambda \| z \| _ { 1 }, \quad ( 1 )$$

and then the HR patch is obtained by x h = D z h ∗ . Finally, the HR image can be estimated by aggregating all the recon-

Figure 3: Our framework for CSC based image SR.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-d7566316-4ccf-4557-bd94-4f62362628b1.jpg)

structed HR patches. Inspired by ScSR, many SC based SR methods have been proposed by using various constraints on sparse code and dictionary [45, 38].

## 2.2. Convolutional Sparse Coding for Image SuperResolution

Traditional SC based SR algorithms usually process images in a patch based manner to reduce the burden of modeling and computation, resulting in the inconsistency problem [28]. As a special case of SC, CSC is inherently suitable for this issue [48]. CSC is proposed to avoid the inconsistency problem by representing the whole image directly. Specifically, an image y ∈ R n r × n c can be represented as the summation of m feature maps z i ∈ R n r × n c convolved with the corresponding filters f i ∈ R s × s : y = ∑ m i =1 f i ⊗ z i , where ⊗ is the convolution operation.

Gu et al . [10] proposed the CSC-SR method and revealed the potential of CSC for image SR. In [10], CSC-SR requires to solve the following CSC based optimization problem in both the training and testing phase:

$$\min _ { f, z } \frac { 1 } { 2 } \left \| y - \sum _ { i = 1 } ^ { m } f _ { i } \otimes z _ { i } \right \| _ { 2 } ^ { 2 } + \lambda \sum _ { i = 1 } ^ { m } \left \| z _ { i } \right \| _ { 1 }. \quad ( 2 ) \quad \text{the}$$

[10] solves this problem by alternatively optimizing the z and f subproblems [42]. The z subproblem is a standard CSC problem. Hundreds of iterations are required to solve the CSC problem and the aforementioned Optimization Issue and Memory Issue cannot be completely avoided. Inspired by the success of deep learning based sparse coding [9], we exploit the natural connection between CSC and CNN to solve the CSC problem efficiently.

## 3. CISTA for solving CSC problem

CSC can be considered as a special case of conventional SC, due to the fact that convolution operation can be replaced with matrix multiplication, so the objective function of CSC can be formulated as:

$$\min _ { z } \left \| y - \sum _ { i = 1 } ^ { m } F _ { i } z _ { i } \right \| _ { 2 } ^ { 2 } + \lambda \sum _ { i = 1 } ^ { m } \left \| z _ { i } \right \| _ { 1 }. \quad \text{ (3)} \quad \text{ (4)}$$

⊗}{∖∫˜√]̂√∐⌈

⊗}{∖∫˜√]̂√∐⌈



![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-ec3df1c4-0b5e-4dcb-a466-f979cf9eeceb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-51190dd9-64a4-4369-a766-a1e413368311.jpg)



Figure 4: Performance curve for residual/non-residual networks with different recursions. The tests are conducted on Set5 for scale factor × 3 .

y z , i are in vectorized form and F i is a sparse convolution matrix with the following attributes:

$$F _ { i } z _ { i } & \equiv f _ { i } \otimes z _ { i } \\ F _ { i } ^ { T } z _ { i } & \equiv \text{flip} ( \text{flip} { r } _ { i } ) ) \otimes z _ { i } \quad ( 4 ) \\ & \equiv \text{flip} ( f _ { i } ) \otimes z _ { i }$$

where fliplr ( ) · and flipud ( ) · are following the notations of Zeiler et al . [48], representing that array is flipped in left/right or up/down direction.

Iterative Soft Thresholding Algorithm (ISTA) [5] can be utilized to solve (3), at the k th iteration:

$$z _ { k + 1 } = h _ { \theta } \left ( z _ { k } + \frac { 1 } { L } F ^ { T } \left ( y - F z _ { k } \right ) \right ) \quad ( 5 )$$

where L is the Lipschitz constant, F = [ F 1 , F 2 , . . . , F m ] and Fz = ∑ m i =1 F z i i . Using the relation in (4) to replace the matrix multiplication with convolution operator, we can reformulate (5) as:

$$z _ { k + 1 } & = h _ { \theta } \left ( I z _ { k } + \frac { 1 } { L } \, f l i p ( f ) \otimes ( y - f \otimes z _ { k } ) \right ) \\ \intertext { d } \quad \text{share $I$ is the admin} \, \text{matrix} \, \text{or} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, -\, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$} \, - \, \text{$\ell$}$$

where I is the identity matrix, f = [ f 1 , f 2 , . . . , f m ] and flip ( f ) = [ flip ( f 1 ) , flip ( f 2 ) , . . . , flip ( f m )] . Note that identity matrix I is also a sparse convolution matrix, so according to (4), there existing a filter n satisfies:

$$I z = n \otimes z,$$

so (6) becomes:

$$z _ { k + 1 } = h _ { \theta } \left ( W \otimes y + S \otimes z _ { k } \right ), \quad \ \ ( 8 )$$

where W = 1 L flip ( f ) and S = n -1 L flip ( f ) ⊗ f . Even though (3) is for a single image with one channel, the extension to multiple channels (for both image and filters) and multiple images is mathematically straightforward. Thus for y ∈ R b × × c n r × n c representing b images of size n r × n c with c channels, (8) is still true with W ∈ R m c × × × s s and S ∈ R m m s × × × s .

Figure 5: The architecture of the pre-upsampling model CRNet-A. The proposed CISTA block with K recursions is surrounded by the dashed box and its unfolded version is shown in the bottom. S is shared across every recursion.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-2403c001-b495-4ab6-8623-bccb64ec8936.jpg)

As for h θ , [26] reveals two important facts: (1) the expressiveness of the sparsity inspired model is not affected even by restricting the coefficients to be nonnegative; (2) the ReLU [25] activation function and the soft nonnegative thresholding operator are equal, that is:

$$h _ { \theta } ^ { + } ( \alpha ) = \max ( \alpha - \theta, 0 ) = R e L U ( \alpha - \theta ). \quad ( 9 ) \quad \text{filter} _ { - }$$

of LR feature maps are obtained using CISTA with LR dictionary W l and shared parameter S , as indicated in (10). Under the assumption that HR feature maps share the same convolutional sparse codes with LR feature maps , HR feature maps can be recovered by W h ⊗ z . Finally, the HR image is reconstructed by utilizing the learned HR filters.

We set θ = 0 for simplicity. So the final form of (8) is:

$$z _ { k + 1 } = R e L U \left ( W \otimes y + S \otimes z _ { k } \right ). \quad ( 1 0 ) \quad \underset { \text{mo} } { \text{m} }$$

One can see that (10) is a convolutional form of (5), so we name it as CISTA. It provides the solution of (3) with theoretical guarantees [5]. Furthermore, this convolutional form can be implemented employing CNN architectures. So W and S in (10) would be trainable.

## 4. Proposed Method

In this section, our framework for CSC based image SR is first introduced. And then we implement it using CNN techniques. Since most of image SR methods can be attributed to two frameworks with different upsampling strategies, i.e ., pre-upsampling and post-upsampling, we propose two models, CRNet-A for pre-upsampling and CRNet-B for post-upsampling.

## 4.1. The framework for CSC based Image SR

Analogy to sparse coding based SR, we develop a framework for CSC based image SR. As shown in Fig. 3, LR feature maps are extracted from the input LR image using the learned LR filters. Then convolutional sparse codes z

In this work, we implement this framework using CNN techniques. However, when combining CSC with CNN, the characteristics of CNN itself must be considered. With more recursions used in CISTA, the network becomes deeper and tends to be bothered by the gradient vanishing/exploding problems. Residual learning [15, 16, 32] is such a useful tool that not only mitigates these difficulties, but helps network converge faster. In Fig. 4, residual/nonresidual networks with different recursions are compared experimentally and the residual network converges much faster and achieves better performance. Based on these observations, both of our proposed models adopt residual learning.

## 4.2. CRNet-A Model for Pre-upsampling

As shown in Fig. 5, CRNet-A takes the ILR image I y with c channels as input, and predicts the output HR image as I x . Two convolution layers, F 0 ∈ R n 0 × × × c s s consisting of n 0 filters of spatial size c × s × s and F 1 ∈ R n 0 × n 0 × × s s containing n 0 filters of spatial size n 0 × s × s are utilized for hierarchical features extraction from ILR image:

$$y = R e L U \left ( F _ { 1 } \otimes R e L U ( F _ { 0 } \otimes I _ { y } ) \right ). \quad ( 1 1 )$$

The ILR features are then fed into a CISTA block to learn the convolutional sparse codes. As stated in (10),

Figure 6: The architecture of the post-upsampling model CRNet-B.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-e9982ef2-3b25-46b6-87c3-3be9918c4301.jpg)

two convolutional layers W l ∈ R m 0 × n 0 × × s s and S ∈ R m 0 × m 0 × × s s are needed:

$$z _ { k + 1 } = R e L U ( W _ { l } \otimes y + S \otimes z _ { k } ), \quad ( 1 2 ) \quad \rangle$$

| Dataset   | Scale       | Bicubic                 | CSC-SR [10]             | CRNet-B (ours)          | Our Improvement   |
|-----------|-------------|-------------------------|-------------------------|-------------------------|-------------------|
| Set5      | × 2 × 3 × 4 | 33 . 66 30 . 39 28 . 42 | 36 . 62 32 . 65 30 . 36 | 38 . 13 34 . 75 32 . 57 | 1.51 2.10 2.21    |

where z 0 is initialized to ReLU ( W l ⊗ y ) . The convolutional sparse codes z are learned after K recursions with S shared across every recursion. When the convolutional sparse codes z are obtained, it is then passed through a convolution layer W h ∈ R n 0 × m 0 × × s s to recover the HR feature maps. The last convolution layer H ∈ R c × n 0 × × s s is used as HR filters:

$$R = H \otimes R e L U ( W _ { h } \otimes z ). \quad \ \ ( 1 3 ) \quad \ _ {. } ^ { m }$$

Note that we pad zeros before all convolution operations to keep all the feature maps to have the same size, which is a common strategy used in a variety of methods [15, 16, 32]. So the residual image R has the same size as the input ILR image I y , and the final HR image I x would be reconstructed by:

$$I _ { x } = I _ { y } + R. \quad \ \ \ \ \ ( 1 4 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

Given N ILR-HR image patch pairs { I ( ) i y , ˜ I ( ) i x } N i =1 as a training set, our goal is to minimize the following objective function:

$$\mathcal { L } ( \Theta ) = \frac { 1 } { 2 N } \sum _ { i = 1 } ^ { N } \left \| I _ { x } ^ { ( i ) } - \tilde { I } _ { x } ^ { ( i ) } \right \| _ { 2 } ^ { 2 } \quad \ \ ( 1 5 ) \quad \ \ 8 0 \\ [ 2 ^ { i } ]$$

where Θ denotes the learnable parameters. The network is optimized using the mini-batch Stochastic Gradient Descent (SGD) with backpropagation [18].

## 4.3. CRNet-B Model for Post-upsampling

We extend CRNet-A to its post-upsampling version to further mine its potential. Notice that most post-upsampling models [19, 37, 20, 50] need to train and store many scaledependent models for various scales without fully using the

Table 1: Average PSNRs of CSC-SR [10] and CRNet-B for scale factor × 2 , × 3 and × 4 on Set5. The performance gain of our model over CSC-SR is shown in the last column.

inter-scale correlation, so we adopt the scale-specific multipath learning strategy [40] presented in MDSR [21] with minor modifications to address this issue. The complete model is shown in Fig. 6. The main branch is our CRNet-A module. The pre-processing modules are used for reducing the variance from input images of different scales and only one residual unit with 3 × 3 kernels is used in each of the pre-processing module. At the end of CRNet-B, upsampling modules are used for multi-scale reconstruction.

## 5. Experimental Results

## 5.1. Datasets and metrics

Training Set By following [15, 32], the training set of CRNet-A consists of 291 images, where 91 of these images are from Yang et al . [47] with the addition of 200 images from Berkeley Segmentation Dataset [23]. For CRNet-B, 800 training images of DIV2K [34] are used for training.

Metrics Both PSNR and SSIM [39] on Y channel ( i.e ., luminance) of transformed YCbCr space are calculated for evaluation.

Testing Set During testing, Set5 [2], Set14 [49], B100 [23] and Urban100 [14] are employed. As recent postupsampling methods [21, 20, 11, 50] also evaluate their performance on Manga109 [24], so does CRNet-B.

## 5.2. Implementation details

CRNet-A Data augmentation and scale augmentation [15, 16, 32] are used for training a single model for all dif-

Table 2: Average PSNR/SSIMs of Pre-upsampling models for scale factor × 2 , × 3 and × 4 on datasets Set5, Set14, BSD100 and Urban100. Red color indicates the best performance and blue color indicates the second best performance.

| Dataset   | Scale       | Bicubic                                                  | SRCNN [6]                                                | RED30 [22]                                               | VDSR [15]                                                | DRCN [16]                                                | DRRN [32]                                                | MemNet [33]                                              | CRNet-A (ours)                                           |
|-----------|-------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|
| Set5      | × 2 × 3 × 4 | 33 . 66 / 0 . 9299 30 . 39 / 0 . 8682 28 . 42 / 0 . 8104 | 36 . 66 / 0 . 9542 32 . 75 / 0 . 9090 30 . 48 / 0 . 8628 | 37 . 66 / 0 . 9599 33 . 82 / 0 . 9230 31 . 51 / 0 . 8869 | 37 . 53 / 0 . 9587 33 . 66 / 0 . 9213 31 . 35 / 0 . 8838 | 37 . 63 / 0 . 9588 33 . 82 / 0 . 9226 31 . 53 / 0 . 8854 | 37 . 74 / 0 . 9591 34 . 03 / 0 . 9244 31 . 68 / 0 . 8888 | 37 . 78 / 0 . 9597 34 . 09 / 0 . 9248 31 . 74 / 0 . 8893 | 37 . 79 / 0 . 9600 34 . 11 / 0 . 9254 31 . 82 / 0 . 8907 |
| Set14     | × 2 × 3 × 4 | 30 . 24 / 0 . 8688 27 . 55 / 0 . 7742 26 . 00 / 0 . 7027 | 32 . 45 / 0 . 9067 29 . 30 / 0 . 8215 27 . 50 / 0 . 7513 | 32 . 94 / 0 . 9144 29 . 61 / 0 . 8341 27 . 86 / 0 . 7718 | 33 . 03 / 0 . 9124 29 . 77 / 0 . 8314 28 . 01 / 0 . 7674 | 33 . 04 / 0 . 9118 29 . 76 / 0 . 8311 28 . 02 / 0 . 7670 | 33 . 23 / 0 . 9136 29 . 96 / 0 . 8349 28 . 21 / 0 . 7720 | 33 . 28 / 0 . 9142 30 . 00 / 0 . 8350 28 . 26 / 0 . 7723 | 33 . 33 / 0 . 9152 29 . 99 / 0 . 8359 28 . 29 / 0 . 7741 |
| B100      | × 2 × 3 × 4 | 29 . 56 / 0 . 8431 27 . 21 / 0 . 7385 25 . 96 / 0 . 6675 | 31 . 36 / 0 . 8879 28 . 41 / 0 . 7863 26 . 90 / 0 . 7101 | 31 . 99 / 0 . 8974 28 . 93 / 0 . 7994 27 . 40 / 0 . 7290 | 31 . 90 / 0 . 8960 28 . 82 / 0 . 7976 27 . 29 / 0 . 7251 | 31 . 85 / 0 . 8942 28 . 80 / 0 . 7963 27 . 23 / 0 . 7233 | 32 . 05 / 0 . 8973 28 . 95 / 0 . 8004 27 . 38 / 0 . 7284 | 32 . 08 / 0 . 8978 28 . 96 / 0 . 8001 27 . 40 / 0 . 7281 | 32 . 09 / 0 . 8985 28 . 99 / 0 . 8021 27 . 44 / 0 . 7302 |
| Urban100  | × 2 × 3 × 4 | 26 . 88 / 0 . 8403 24 . 46 / 0 . 7349 23 . 14 / 0 . 6577 | 29 . 50 / 0 . 8946 26 . 24 / 0 . 7989 24 . 52 / 0 . 7221 | 30 . 85 / 0 . 9148 27 . 25 / 0 . 8283 25 . 28 / 0 . 7555 | 30 . 76 / 0 . 9140 27 . 14 / 0 . 8279 25 . 18 / 0 . 7524 | 30 . 75 / 0 . 9133 27 . 15 / 0 . 8276 25 . 14 / 0 . 7510 | 31 . 23 / 0 . 9188 27 . 53 / 0 . 8378 25 . 44 / 0 . 7638 | 31 . 31 / 0 . 9195 27 . 56 / 0 . 8376 25 . 50 / 0 . 7630 | 31 . 36 / 0 . 9207 27 . 64 / 0 . 8403 25 . 59 / 0 . 7680 |

Table 3: Average PSNR/SSIMs of Post-upsampling models for scale factor × 2 , × 3 and × 4 on datasets Set5, Set14, BSD100, Urban100 and Manga109. Red color indicates the best performance and blue color indicates the second best performance.

| Dataset   | Scale       | SRDenseNet [37]                     | MSRN [20]                                                | D-DBPN [11]                               | EDSR [21]                                                | MDSR [21]                                                | RDN [50]                                                 | CRNet-B (ours)                                           | CRNet-B+ (ours)                                          |
|-----------|-------------|-------------------------------------|----------------------------------------------------------|-------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|
| Set5      | × 2 × 3 × 4 | -/- 38 -/- 34 32 . 02 / 0 . 8934 32 | . 08 / 0 . 9605 . 38 / 0 . 9262 . 07 / 0 . 8903          | 38 . 09 / 0 . 9600 -/- 32 . 47 / 0 . 8980 | 38 . 11 / 0 . 9601 34 . 65 / 0 . 9282 32 . 46 / 0 . 8968 | 38 . 11 / 0 . 9602 34 . 66 / 0 . 9280 32 . 50 / 0 . 8973 | 38 . 24 / 0 . 9614 34 . 71 / 0 . 9296 32 . 47 / 0 . 8990 | 38 . 13 / 0 . 9610 34 . 75 / 0 . 9296 32 . 57 / 0 . 8991 | 38 . 25 / 0 . 9614 34 . 83 / 0 . 9303 32 . 71 / 0 . 9008 |
| Set14     | × 2 × 3 × 4 | -/- -/- 28 . 50 / 0 . 7782          | 33 . 74 / 0 . 9170 30 . 34 / 0 . 8395 28 . 60 / 0 . 7751 | 33 . 85 / 0 . 9190 -/- 28 . 82 / 0 . 7860 | 33 . 92 / 0 . 9195 30 . 52 / 0 . 8462 28 . 80 / 0 . 7876 | 33 . 85 / 0 . 9198 30 . 44 / 0 . 8452 28 . 72 / 0 . 7857 | 34 . 01 / 0 . 9212 30 . 57 / 0 . 8468 28 . 81 / 0 . 7871 | 34 . 09 / 0 . 9219 30 . 58 / 0 . 8465 28 . 79 / 0 . 7867 | 34 . 15 / 0 . 9227 30 . 67 / 0 . 8481 28 . 93 / 0 . 7894 |
| B100      | × 2 × 3 × 4 | -/- -/- 27 . 53 / 0 . 7337          | 32 . 23 / 0 . 9013 29 . 08 / 0 . 8041 27 . 52 / 0 . 7273 | 32 . 27 / 0 . 9000 -/- 27 . 72 / 0 . 7400 | 32 . 32 / 0 . 9013 29 . 25 / 0 . 8093 27 . 71 / 0 . 7420 | 32 . 29 / 0 . 9007 29 . 25 / 0 . 8091 27 . 72 / 0 . 7418 | 32 . 34 / 0 . 9017 29 . 26 / 0 . 8093 27 . 72 / 0 . 7419 | 32 . 32 / 0 . 9014 29 . 26 / 0 . 8091 27 . 73 / 0 . 7414 | 32 . 38 / 0 . 9020 29 . 32 / 0 . 8103 27 . 80 / 0 . 7430 |
| Urban100  | × 2 × 3 × 4 | -/- -/- 26 . 05 / 0 . 7819          | 32 . 22 / 0 . 9326 28 . 08 / 0 . 8554 26 . 04 / 0 . 7896 | 32 . 55 / 0 . 9324 -/- 26 . 38 / 0 . 7946 | 32 . 93 / 0 . 9351 28 . 80 / 0 . 8653 26 . 64 / 0 . 8033 | 32 . 84 / 0 . 9347 28 . 79 / 0 . 8655 26 . 67 / 0 . 8041 | 32 . 89 / 0 . 9353 28 . 80 / 0 . 8653 26 . 61 / 0 . 8028 | 32 . 93 / 0 . 9355 28 . 87 / 0 . 8667 26 . 69 / 0 . 8045 | 33 . 14 / 0 . 9370 29 . 09 / 0 . 8697 26 . 90 / 0 . 8089 |
| Manga109  | × 2 × 3 × 4 | -/- -/- -/-                         | 38 . 82 / 0 . 9868 33 . 44 / 0 . 9427 30 . 17 / 0 . 9034 | 38 . 89 / 0 . 9775 -/- 30 . 91 / 0 . 9137 | 39 . 10 / 0 . 9773 34 . 17 / 0 . 9476 31 . 02 / 0 . 9148 | 38 . 96 / 0 . 9769 34 . 17 / 0 . 9473 31 . 11 / 0 . 9148 | 39 . 18 / 0 . 9780 34 . 13 / 0 . 9484 31 . 00 / 0 . 9151 | 39 . 07 / 0 . 9778 34 . 17 / 0 . 9481 31 . 16 / 0 . 9154 | 39 . 28 / 0 . 9784 34 . 52 / 0 . 9498 31 . 52 / 0 . 9187 |

ferent scales ( × 2 , × 3 and × 4 ). Every convolution layer in CRNet-A contains 128 filters ( n 0 = 128 ) of size 3 × 3 while W l and S have 256 filters ( m 0 = 256 ). The network is optimized using SGD. The learning rate is initially set to 0 1 . and then decreased by a factor of 10 every 10 epochs. L2 loss is used for CRNet-A, and we train a total of 35 epochs.

CRNet-B Every weight layer in CRNet-B has 64 filters ( n 0 = 64 ) with the size of 3 × 3 except W l and S have 1 024 , filters ( m 0 = 1 024 , ). CRNet-B is updated using Adam [17]. The initial learning rate is 10 -4 and halved every 200 epochs. We train CRNet-B for 800 epochs. Unlike CRNet-A, CRNet-B is trained using L1 loss for better convergence speed.

Recursion We choose K = 25 in both of our models. We implement our models using the PyTorch [29] framework with NVIDIA Titan Xp. It takes approximately 4 5 . days to train CRNet-A, and 15 days to train CRNet-B.

## 5.3. Comparison with CSC-SR

We first compare our proposed models with the existing CSC based image SR method, i.e ., CSC-SR [10]. Since CSC-SR utilizes LR images as input image, it can be considered as a post-upsampling method, thus CRNet-B is used for comparison. Tab. 1 presents that our CRNet-B clearly outperforms CSC-SR by a large margin.

## 5.4. Comparison with State of the Arts

We now compare the proposed models with other stateof-the-arts in recent years. We compare CRNet-A with pre-upsampling models ( i.e ., SRCNN [6], RED30 [22], VDSR [15], DRCN [16], DRRN [32], MemNet [33]) while CRNet-B with post-upsampling architectures ( i.e ., SRDenseNet [37], MSRN [20], D-DBPN [11], EDSR/MDSR [21], RDN [50]). Similar to [21, 50], self-ensemble strategy [21] is also adopted to further improve the performance of CRNet-B, and we denote the self-ensembled version as CRNet-B+.

Tab. 2 and Tab. 3 show the quantitative comparisons on the benchmark testing sets. Both of our models achieve superior performance against the state-of-the-arts, which indicates the effectiveness of our models. Qualitative results are provided in Fig. 7. Our methods tend to produce shaper edges and more correct textures, while other images may be blurred or distorted. More visual comparisons are available in the supplementary material.

Fig. 8 shows the performance versus the number of parameters, our CRNet-B and CRNet-B+ achieve better results with fewer parameters than EDSR [21] and RDN [50].

Figure 7: SR results of 'img016' and 'img059' from Urban100 with scale factor × 4 . Red indicates the best performance.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-39a11b6a-bd92-48b5-95ff-bc386d967e44.jpg)

It's worth noting that EDSR/MDSR and RDN are far deeper than CRNet-B ( e.g ., 169 vs . 36 ), but CRNet-B is quite wider ( W l and S have 1 024 , filters). As reported in [21], when increasing the number of filters to a certain level, e.g ., 256 , the training procedure of EDSR (for × 2 ) without residual scaling [31, 21] is numerically unstable, as shown in Fig. 9(a). However, CRNet-B is relieved from the residual scaling trick. The training loss of CRNet-B is depicted in Fig. 9(b), it converges fast at the begining, then keeps decreasing and finally fluctuates at a certain range.

i.e ., 256 , CRNet-B still outperforms MSRN [20] with fewer parameters ( 2 0 . M vs . 6 1 . M).

## 5.5. Parameter Study

Number of Recursions We also have trained and tested CRNet-A with 15 20 25 48 , , , recursions, so the depth of the these models are 20 , 25 , 30 , 53 respectively. The results are presented in Fig. 11(a). It's clear that CRNet-A with 20 layers still outperforms DRCN with the same depth and increasing K can promote the final performance. The results of using different recursions in CRNet-B are shown in Fig. 11(b), which demonstrate that more recursions facilitate the performance improved.

The key parameters in both of our models are the number of filters ( n , m 0 0 ) and recursions K .

Number of Filters We set n 0 = 128 , m 0 = 256 , K = 25 for CRNet-A as stated in Section 5.2. In Fig. 10(a), CRNet-A with different number of filters are tested (DRCN [16] is used for reference). We find that even n 0 is decreased from 128 to 64 , the performance is not affected greatly. On the other hand, if we decrease m 0 from 256 to 128 , the performance would suffer an obvious drop, but still better than DRCN[16]. Based on these observations, we set the parameters of CRNet-B by making m 0 larger and n 0 smaller for the trade off between model size and performance. Specifically, we use n 0 = 64 , m 0 = 1024 , K = 25 for CRNet-B. As shown in Fig. 10(b), the performance of CRNet-B can be significantly boosted with larger m 0 (MDSR [21] and MSRN [20] are used for reference). Even with small m 0 ,

## 6. Discussions

Wediscuss the differences between our proposed models and several recent CNN models for SR with recursive learning strategy , i.e ., DRRN [32], SCN [41] and DRCN [16]. Due to the fact that CRNet-B is an extension of CRNet-A, i.e ., the main part of CRNet-B has the same structure as CRNet-A, so we use CRNet-A here for comparison. The simplified structures of these models are shown in Fig. 12, where the digits on the left of the recursion line represent the number of recursions.

Difference to DRRN . The main part of DRRN [32] is the recursive block structure, where several residual units with BN layers are stacked. On the other hand, guided by (10), CRNet-A contains no BN layers. Coinciding with EDSR/MDSR [21], by normalizing features, BN layers get rid of range flexibility from networks. Further-

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-ee331d4e-b545-4363-8207-17c56b438248.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-97156783-34ca-40b0-86d4-87de9020d5fa.jpg)

(a) Pre-upsampling models

(b) Post-upsampling models

Figure 8: PSNR of recent state-of-the-arts versus the number of parameters for scale factor × 4 on Set5. The number of layers are marked in the parentheses.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-b64bb59b-6437-4356-9d76-f1e59afb6d3d.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-0ede64c6-f370-4da9-832d-e6f352a98b39.jpg)

Loss

(b) CRNet-B for all scales

Figure 9: Training loss of EDSR ( × 2 ) without residual scaling and CRNet-B (for all scales).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-7bc7d537-8a83-4567-9dcf-53a462340652.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-b300ec08-59f7-4a5a-b349-7ff3f7b2d31e.jpg)

Figure 10: PSNRof proposed models versus different number of filters on Set5 with scale factor × 3 .

Figure 11: PSNRof proposed models versus different number of recursions on Set5 with scale factor × 3 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-d86c39b8-da4e-44c9-b0e4-38a0ec853028.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-afa57f23-b71b-4591-a8f7-ea35bfce0c7d.jpg)

more, BN consumes much amount of GPU memory and increases computational complexity. Experimental results on benchmark datasets under common-used assessments demonstrate the superiority of CRNet-A.

Difference to SCN . There are two main differences between CRNet-A and SCN [41]: CISTA block and resid-

Figure 12: Simplified network structures of (a) DRRN [32], (b) SCN [41], (c) DRCN [16], (d) our model CRNet-A.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01166v1-figure-45dfabdc-751e-4352-890b-373ccefee137.jpg)

ual learning. Specifically, CRNet-A takes consistency constraint into consideration with the help of CISTA block, while SCN uses linear layers and ignores the information from the consistency prior. On the other hand, CRNet-A adopts residual learning, which is a powerful tool for training deeper networks. CRNet-A ( 30 layers) is much deeper than SCN ( 5 layers). As indicated in [15], a deeper network has larger receptive fileds, so more contextual information in an image would be utilized to infer high-frequency details. In Fig. 11(a), we show that more recursions, e.g ., 48 , can be used to achieve better performance.

Difference to DRCN . CRNet-A differs with DRCN [16] in two aspects: recursive block and training techniques. In the recursive block, both local residual learning [32] and pre-activation [12, 32] are utilized in CRNet-A, which are demonstrated to be effective in [32]. As for training techniques, DRCN is not easy to train, so recursive-supervision is introduced to facilitate the network to converge. Moreover, an ensemble strategy (in Fig. 12(c), the final output is the weighted average of all intermediate predictions) is used to further improve the performance. CRNet-A is relieved from these techniques and can be easily trained with more recursions.

## 7. Conclusions

In this work, we propose two effective CSC based image SR models, i.e ., CRNet-A and CRNet-B, for pre-/postupsampling SR, respectively. By combining the merits of CSC and CNN, we achieve superior performance against recent state-of-the-arts. Furthermore, our framework and CISTA block are expected to be applicable in various CSC based tasks, though in this paper we focus on CSC based image SR.

## References

- [1] N. Ahn, B. Kang, and K.-A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV , 2018. 1
- [2] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. AlberiMorel. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. BMVC , pages 135.1-135.10, 2012. 1, 5
- [3] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning , 3(1):1-122, 2011. 2
- [4] H. Bristow, A. Eriksson, and S. Lucey. Fast Convolutional Sparse Coding. In CVPR , 2013. 2
- [5] I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics , 57(11):1413-1457, 2004. 3, 4
- [6] C. Dong, C. C. Loy, K. He, and X. Tang. Image superresolution using deep convolutional networks. TPAMI , 38(2):295-307, 2016. 1, 2, 6
- [7] C. Garcia-Cardona and B. Wohlberg. Convolutional Dictionary Learning: A Comparative Review and New Algorithms. IEEE Transactions on Computational Imaging , 4(3):366381, 2018. 2
- [8] C. Garcia-Cardona and B. Wohlberg. Convolutional dictionary learning: A comparative review and new algorithms. IEEE Transactions on Computational Imaging , 4(3):366381, 2018. 2
- [9] K. Gregor and Y. LeCun. Learning Fast Approximations of Sparse Coding. In ICML , 2010. 1, 3
- [10] S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng, and L. Zhang. Convolutional Sparse Coding for Image Super-Resolution. In ICCV , 2015. 1, 2, 3, 5, 6
- [11] M. Haris, G. Shakhnarovich, and N. Ukita. Deep backprojection networks for super-resolution. In CVPR , 2018. 1, 5, 6
- [12] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV , 2016. 8
- [13] F. Heide, W. Heidrich, and G. Wetzstein. Fast and flexible convolutional sparse coding. In CVPR , 2015. 2
- [14] J.-B. Huang, A. Singh, and N. Ahuja. Single image superresolution from transformed self-exemplars. In CVPR , 2015. 1, 2, 5
- [15] J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image superresolution using very deep convolutional networks. In CVPR , 2016. 1, 2, 4, 5, 6, 8
- [16] J. Kim, J. K. Lee, and K. M. Lee. Deeply-Recursive Convolutional Network for Image Super-Resolution. In CVPR , 2016. 1, 2, 4, 5, 6, 7, 8
- [17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR , 2014. 6
- [18] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 5
- [19] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR , 2017. 5
- [20] J. Li, F. Fang, K. Mei, and G. Zhang. Multi-scale residual network for image super-resolution. In ECCV , 2018. 1, 2, 5, 6, 7
- [21] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In CVPR Workshops , 2017. 1, 2, 5, 6, 7
- [22] X. Mao, C. Shen, and Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS , 2016. 1, 2, 6
- [23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 5
- [24] Y. Matsui, K. Ito, Y . Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications , 2017. 5
- [25] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML , 2010. 4
- [26] V. Papyan, Y. Romano, and M. Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research , 18(1):2887-2938, 2017. 4
- [27] V. Papyan, Y. Romano, J. Sulam, and M. Elad. Theoretical foundations of deep learning via sparse representations: A multilayer sparse model and its connection to convolutional neural networks. IEEE Signal Processing Magazine , 35(4):72-89, 2018. 1, 2
- [28] V. Papyan, J. Sulam, and M. Elad. Working locally thinking globally: Theoretical guarantees for convolutional sparse coding. IEEE Transactions on Signal Processing , 65(21):5687-5701, 2017. 1, 2, 3
- [29] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, et al. Automatic differentiation in pytorch. In NIPS-W , 2017. 6
- [30] H. Sreter and R. Giryes. Learned convolutional sparse coding. In ICASSP , 2018. 2
- [31] C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv:11602.07261 , 2018. 7
- [32] Y. Tai, J. Yang, and X. Liu. Image Super-Resolution via Deep Recursive Residual Network. In CVPR , 2017. 1, 2, 4, 5, 6, 7, 8
- [33] Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In ICCV , 2017. 1, 2, 6
- [34] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang, et al. Ntire 2017 challenge on single image superresolution: Methods and results. In CVPR Workshops , 2017. 1, 5
- [35] R. Timofte, V. De Smet, and L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV , 2014. 1
- [36] R. Timofte, S. Gu, J. Wu, and L. Van Gool. NTIRE 2018 challenge on single image super-resolution: methods and results. In CVPR Workshops , 2018. 1
- [37] T. Tong, G. Li, X. Liu, and Q. Gao. Image super-resolution using dense skip connections. In ICCV , 2017. 2, 5, 6
- [38] S. Wang, L. Zhang, Y. Liang, and Q. Pan. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In CVPR , pages 2216-2223, 2012. 3
- [39] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IEEE TIP , 13(4):600-612, 2004. 5
- [40] Z. Wang, J. Chen, and S. C. Hoi. Deep learning for image super-resolution: A survey. arXiv:1902.06068 , 2019. 2, 5
- [41] Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang. Deep networks for image super-resolution with sparse prior. In ICCV , 2015. 1, 2, 7, 8
- [42] B. Wohlberg. Efficient convolutional sparse coding. In ICASSP , 2014. 2, 3
- [43] B. Wohlberg. Boundary handling for convolutional sparse representations. In ICIP , 2016. 2
- [44] C.-Y. Yang, C. Ma, and M.-H. Yang. Single-image superresolution: A benchmark. In ECCV , 2014. 1
- [45] J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang. Coupled dictionary training for image super-resolution. IEEE TIP , 21(8):3467-3478, 2012. 3
- [46] J. Yang, J. Wright, T. Huang, and Y. Ma. Image superresolution as sparse representation of raw image patches. In CVPR , 2008. 1, 2
- [47] J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image super-resolution via sparse representation. IEEE TIP , 19(11):2861-2873, 2010. 1, 2, 5
- [48] M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus. Deconvolutional networks. In CVPR , 2010. 1, 2, 3
- [49] R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces , pages 711-730. Springer, 2010. 5
- [50] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image super-resolution. In CVPR , 2018. 1, 2, 5, 6
- [51] Z. Zhang, Y. Xu, J. Yang, X. Li, and D. Zhang. A Survey of Sparse Representation - Algorithms and Applications. IEEE Access , 3:490-530, 2015. 2
Menglei Zhang Zhou Liu Lei Yu
CRNet: Image Super-Resolution Using A Convolutional Sparse Coding Inspired Network
Convolutional Sparse Coding (CSC) has been attracting more and more attention
in recent years, for making full use of image global correlation to improve
performance on various computer vision applications. However, very few studies
focus on solving CSC based image Super-Resolution (SR) problem. As a
consequence, there is no significant progress in this area over a period of
time. In this paper, we exploit the natural connection between CSC and
Convolutional Neural Networks (CNN) to address CSC based image SR.
Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is
introduced to solve CSC problem and it can be implemented using CNN
architectures. Then we develop a novel CSC based SR framework analogy to the
traditional SC based SR methods. Two models inspired by this framework are
proposed for pre-/post-upsampling SR, respectively. Compared with recent
state-of-the-art SR methods, both of our proposed models show superior
performance in terms of both quantitative and qualitative measurements.