## Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception

Jiancong Feng, Yuan-Gen Wang, Mingjie Li, and Fengchuang Xing

Abstract -Self-similarity techniques are booming in blind super-resolution (SR) due to accurate estimation of the degradation types involved in low-resolution images. However, highdimensional matrix multiplication within self-similarity computation prohibitively consumes massive computational costs. We find that the high-dimensional attention map is derived from the matrix multiplication between Query and Key, followed by a softmax function. This softmax makes the matrix multiplication between Query and Key inseparable, posing a great challenge in simplifying computational complexity. To address this issue, we first propose a second-order Taylor expansion approximation (STEA) to separate the matrix multiplication of Query and Key, resulting in the complexity reduction from O ( N 2 ) to O ( N ) . Then, we design a multi-scale large field reception (MLFR) to compensate for the performance degradation caused by STEA. Finally, we apply these two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experimental results tested on five synthetic datasets demonstrate that our LabNet sets a new benchmark in qualitative and quantitative evaluations. Tested on the RealWorld38 dataset, our RealNet achieves superior visual quality over existing methods. Ablation studies further verify the contributions of STEA and MLFR towards both LabNet and RealNet frameworks.

Index Terms -Blind Super-Resolution, Self-similarity, Taylor Expansion, High-dimensional matrix multiplication

## I. INTRODUCTION

S INGLE-Image Super-Resolution (SISR) [1-4] has obtained widespread application in various fields, such as remote sensing imagery, security surveillance, and medical imaging. SISR methods are generally divided into two categories: non-blind SR and blind SR. The non-blind SR [5-11] assumes that the degradation type is known, which degradation function is typically represented as:

$$x _ { l r } = x _ { h r } \downarrow _ { s }, \quad \quad \ \ ( 1 ) \ \text{opt}$$

where x hr and x lr denote the HR and LR images respectively, and ↓ s denotes the bicubic downsampling operation with a scale factor of s . However, the non-blind SR represented by Eq. (1) fails to accurately approximate the real-world degradation due to the inaccessibility of the reference image.

Subsequently, researchers have developed blind SR methods, which assume that the LR image is obtained from the HR image through a degradation function with a complex and

Feng, J., Wang, Y.-G., Li, M., and Xing F. are with the School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510006, China (e-mail: fengjiancong@e.gzhu.edu.cn; wangyg@gzhu.edu.cn; limingjie@gzhu.edu.cn; xfchuang@e.gzhu.edu.cn).

Fig. 1. The training and testing procedures of RealNet.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-3a2b3bbe-e250-4a5b-b2c3-ab08e753ee41.jpg)

unknown degradation type, and this degradation function is typically assumed to follow the equation:

$$x _ { l r } = ( x _ { h r } \otimes \text{Kernel} ) \downarrow _ { s } + N, \text{ \quad \ \ } ( 2 )$$

where ⊗ denotes the convolutional operation, Kernel represents the unknown blur kernel, ↓ s signifies the downsampling operation with a scale factor of s , and N accounts for sophisticated noise. According to Eq. (2), we can clearly see that a single HR image corresponds to infinitely many possible LR images, which is typically an ill-posed problem. Nonetheless, a multitude of blind SR methods has been proposed to overcome this problem. They can be broadly divided into two main types: methods without blur kernel estimation [12-15] and methods with blur kernel estimation [16-18].

As for methods without blur kernel estimation, SRMD [12] is the first to combine the preset blur kernel, noise, and LR image as the input to the SR network. After that, [14, 19] incorporated Maximum A Posteriori Probability into iterative optimization. For these methods, the selection of parameters for the preset blur kernel is crucial for achieving effective results in practical applications. However, it is challenging to manually choose appropriate preset blur kernel parameters in the real world. To overcome this challenge, researchers have developed some methods to estimate the blur kernel. IKC [16] proposed to iteratively estimate the blur kernel, which divides the SR reconstruction process into two steps: 1) estimating the blur kernel from the LR image and 2) combining the estimated blur kernel and the LR image to reconstruct the SR image. However, the independence between the training of the blur kernel estimation model and the SR model leads to incompatibility issues. To address this issue, DANv1 [17] proposed an alternating optimization strategy between the blur kernel estimation and SR models, which

makes the two models form a closed loop that can be trained in an end-to-end manner. Although DANv1 has achieved impressive performance in blind SR tasks, NLCUNet [20] has made significant advancements by incorporating the NonLocal Sparse Attention (NLSA) [21] module. It screens global features in the feature map by calculating self-similarities, enabling implicit estimation of the blur kernel. Therefore, the self-similarity technique has received the widest application in blind SR tasks. However, blind SR models that incorporate the calculation of self-similarity suffer from high computational complexity, making them unsuitable for resource-constrained devices. It is worth noting that the commonly used module for calculating self-similarity is Non-Local Attention (NLA) [22].

In this paper, we go deep into the execution process of NLA. We find that during the self-similarity calculation, a matrix multiplication operation between the high-dimensional attention map and the Value is required, leading to high computational complexity. The attention map is obtained by multiplying the Query matrix with the Key matrix. Then the softmax function is further performed, which hinders the separation of Query and Key. Based on this finding, we propose a second-order Taylor expansion approximation (STEA) to significantly reduce the computational complexity. Compared to self-similarity calculations, STEA loses the ability to filter global features. Hence, we design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA. Finally, we apply our two core designs to laboratory and real-world scenarios by constructing LabNet and RealNet, respectively. Extensive experiments validate the effectiveness of the proposed method. The major contributions of this paper are summarized as follows.

- · We propose the second-order Taylor expansion approximation (STEA) strategy, which significantly reduces the computational complexity in self-similarity-based blind SR tasks.
- · We design a multi-scale large field reception (MLFR) to compensate for the performance loss caused by STEA, showcasing competitive performance over the state-ofthe-art.
- · We construct the LabNet and the RealNet by integrating our two core designs (STEA and MLFR) to respectively verify the effectiveness of our designs in the laboratory and real-world scenarios. Extensive experimental results demonstrate that our LabNet achieves state-of-the-art performance while maintaining linear computational complexity, and our RealNet effectively handles various realworld degradations and achieves superior performance in diverse blind SR scenarios.

The structure of this paper is organized as follows: Section II provides a comprehensive overview of the proposed methodology. Specifically, Section II-A presents a detailed explanation of the intricate aspects of STEA. Section II-B elucidates the specific details of MLFR. Section II-C outlines the implementation specifics of LabNet, while Section II-D delves into the implementation details of RealNet. Subsequently, Section III encompasses the dataset and evaluation metrics (Section III-A), implementation details of LabNet

(Section III-B), implementation details of RealNet (Section III-C), comparison with state-of-the-art techniques (Section III-D), ablation studies and analysis of LabNet (Section IIIE), ablation study and analysis of RealNet (Section III-F) and adjustable parameter details in RealNet (Section III-G). Section IV concludes the paper.

## II. METHODOLOGY

In this section, we provide a detailed explanation of the STEA and MLFR algorithms, as well as their deployment to the LabNet (testing for laboratory settings) and RealNet (testing for real-world scenarios).

## A. Second-Order Taylor Expansion Approximation (STEA)

The self-similarity has been widely applied to the blind SR task, which is commonly implemented by a Non-Local Attention (NLA) [22]. However, we found that the NLA brings extremely high computational complexity. This mainly arises from the matrix multiplication between the high-dimensional Attention Map and the V matrix, as highlighted by the red dashed box of Fig. 3. Without loss of generality, Q K V , , ∈ R N × d denote query, key, and value matrix, respectively. Then, we know the computational complexity of NLA is O ( N 2 × d ) , where N and d represent the dimension of the feature map and the number of channels, respectively. In this part, we propose an STEA algorithm to significantly reduce the computational complexity of NLA. To this end, we first analyze the execution process of the NLA module. As shown in Fig. 3, the Attention Map is derived from the matrix multiplication of Q and K T . Due to the limitation imposed by the softmax operation, the matrix multiplication between Q and K T cannot be separated, resulting in a O ( N 2 ) complexity. To address this challenge, we propose a SecondOrder Taylor Expansion to approximate the NLA module. Here, we mathematically rewrite the execution process of the NLA as follows:

$$Q = X W _ { q }, K = X W _ { k }, V = X W _ { v }, \text{ \quad \ \ } ( 3 )$$

where W q , W k and W v stand for the parameters of three 1 × 1 convolutions, respectively. Q K , and V represent the Query, Key and Value, respectively. X denotes the input feature map. Next, we perform the matrix multiplication between Q and K T , followed by the softmax operation, to obtain the Attention Map:

$$\ A t t M a p = \text{SoftMax} ( Q K ^ { T } ), \text{ \quad \ \ } ( 4 )$$

where AttMap represents the Attention Map, and SoftMax ( ) · denotes the softmax function. By substituting Eq. (3) to Eq. (4), we have

$$\ A t t M a p = \text{SoftMax} ( X W _ { q } W _ { k } ^ { T } X ^ { T } ). \quad \ \ ( 5 )$$

We denote W qk = W W q T k , resulting in the below expression:

$$\ A t t M a p = \text{SoftMax} ( X W _ { q k } X ^ { T } ). \text{ \quad \ \ } ( 6 )$$

In the original NLA module, the multiplication between Q and K T followed by the softmax operation gives rise to softmax attention. Although softmax attention can effectively enhance

Fig. 2. The overview of our proposed LabNet.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-2f432eec-c8ba-4a83-b890-52881f3fe058.jpg)

Fig. 3. Analysis of the execution process of the Non-Local Attention (NLA) module.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-28a8f985-468c-46d3-944c-d620caa278eb.jpg)

the performance of the SR network by making the matrix full-rank, it introduces high computational complexity since such a high-dimensional Attention Map ( AttMap ) must be multiplied with V in the subsequent computation. For that, we propose to simplify the softmax function as an exponential form:

$$\ A t t M a p = \text{SoftMax} ( Q K ^ { T } ) = \frac { 1 } { k } \times e ^ { Q K ^ { T } } = \frac { 1 } { k } \times e ^ { X W _ { q k } X ^ { T } }, \quad \text{(7)} \quad \text{ empiric} \quad \\ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \end{array}$$

where k is a learnable parameter used to approximate the denominator of softmax. By now, we can see that the operation between QK T and V is still nonlinear due to the presence of an exponential function, which disables us from separating Q and K from QK T . In subsequent multiplication by V , the basic operation of the matrix cannot be flexibly used, failing to reduce the high computational complexity. For this purpose, we exploit a second-order Taylor expansion as a substitute for the exponential function, achieving a good trade-off between efficiency and accuracy.

Fig. 4. The Second-Order Taylor Expansion Approximation (STEA) module.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-145ad251-fda4-4e00-8dd1-da7683821549.jpg)

We know that in principle, the more remainder terms kept in the Taylor expansion of the exponential function, the more the expansion approximates the raw function. According to our empirical results and considering a good trade-off between accuracy and complexity, we adopt the second-order Taylor expansion on Eq. (7) as follows:

$$\text{ence of} \quad \ A t t M a p = \frac { 1 } { k } \times e ^ { X W _ { q k } X ^ { T } } \\ V, \text{ the} \quad \text{$\quad} = \frac { 1 } { k } \times ( I _ { n } + X W _ { q k } X ^ { T } + \frac { ( X W _ { q k } X ^ { T } ) ^ { 2 } } { 2! } + O ( X W _ { q k } X ^ { T } ) ), \\ \text{tute for}$$

where I n stands for identity matrix and O ( ) · stands for the high-order remainder of the Taylor expansion. By ignoring

the constant and remainder terms of Eq.(8), we achieve:

$$\ A t t M a p = \frac { 1 } { k } \times ( I _ { n } + X W _ { q k } X ^ { T } + \begin{array} { c } \ e t t e q \\ X W _ { q k } X ^ { T } X W _ { q k } X ^ { T } ). \end{array} \quad \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e tt e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t eq \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ o t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t T e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e q \ e t t e x \ e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e$$

Then, we multiply AttMap by V and refer to Eq.(3) to obtain:

$$O u t p u t = A t t M a p \times V = \frac { 1 } { k } \times ( X W _ { v } + & \quad \cdots \\ X W _ { q k } X ^ { T } X W _ { v } + X W _ { q k } X ^ { T } X W _ { q k } X ^ { T } X W _ { v } ). & ( 1 0 )$$

Eq. (10) clearly indicates that the constraint of the softmax function has been removed, and therefore we can apply the associative property of the matrix operation in Eq. (10), which allows us to calculate X X T first. Considering that X X T is a symmetric matrix, we can transform X X T into a diagonalization expression as follows:

$$X ^ { T } X = Z B Z ^ { T }, \quad \quad \ \ ( 1 1 ) \ \ B y \.$$

where B is a diagonal matrix with its diagonal elements being the eigenvalues of X X T , and Z is an orthogonal matrix whose columns are the eigenvectors of X X T corresponding to the eigenvalues. By substituting Eq. (11) into Eq. (10), we obtain:

$$O u t p u t = \frac { 1 } { k } \times ( X W _ { v } + X W _ { q k } Z B Z ^ { T } W _ { v } + \\ X W _ { q k } Z B Z ^ { T } W _ { q k } Z B Z ^ { T } W _ { v } ). \quad ( 1 2 )$$

$$( 1 2 )$$

According to Eq. (11), we can see that obtaining B through the matrix eigen-decomposition operation is computationally expensive. If we randomly initialize a learnable diagonal matrix B to avoid the high computational complexity from the eigen-decomposition operation of X X T . Unfortunately, this strategy makes the matrix B independent of the input feature maps ( X ). Besides, in the random initialization, the matrix B is required to be a fixed size, making X have to be the fixed size. This disables the input LR image to be an arbitrary size. To reduce the complexity of the matrix eigen-decomposition and implement the SR task on the LR images with an arbitrary size, we design two separative eigenvalue extraction modules to respectively obtain two diagonal matrices B 1 and B 2 , as shown in Fig. 4. Our design avoids randomly initializing a single B , and meanwhile enhances the representation ability of the model. Therefore, we can obtain a learning-based approximation expression as follows:

$$O u t p u t = \frac { 1 } { k } \times ( X W _ { v } + X W _ { q k } Z B _ { 1 } Z ^ { T } W _ { v } + \begin{array} { c c } \text{comp} \\ \text{terms o} \\ \text{terms o} \\ \text{advice} \end{array}.$$

It is observed from Fig. 4 that our design modules can dynamically extract global information from the matrices, which can well match the matrix eigen-decomposition where global information is exploited dynamically.

Similarly, we adopt learnable parameters to substitute Z , solving the high computational complexity of the eigendecomposition of X X T . Next, we let W 1 = W Z W qk , 2 = Z T W Z qk , and W 3 = Z T W v . Hence, we obtain:

$$O u t p u t = \frac { 1 } { k } \times ( X W _ { v } + X W _ { 1 } B _ { 1 } W _ { 3 } + X W _ { 1 } B _ { 1 } W _ { 2 } B _ { 2 } W _ { 3 } ). \quad \text{in Fig.} \\ ( 1 4 ) \quad \text{ we use } 1$$

Fig. 5. The Multi-Scale Large Field Reception (MLFR) module.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-25053d1e-e7d2-4649-be22-a19568cea008.jpg)

By now, we equate the softmax function to an exponential function and perform a Taylor expansion, making the operations of Q K , , and V more flexible. It is worth noting that the softmax function can transform the resulting matrix derived from Q and K T into a full-rank matrix, while the matrix obtained by direct multiplication of Q and K T is not a fullrank matrix, which significantly reduces the feature diversity, in turn, leads to a degradation of the model's performance [23]. Therefore, as done in [23], we perform a depthwise convolution (DWC) operation on XW v of Eq. (14) to compensate for the performance degradation. Finally, we obtain:

$$\text{gonal} \quad F i n a l O u t p u t & = \frac { 1 } { k } \times [ D W C ( X W _ { v } ) + \\ ; \text{ this} \quad & \quad X W _ { 1 } B _ { 1 } W _ { 3 } + X W _ { 1 } B _ { 1 } W _ { 2 } B _ { 2 } W _ { 3 } ]. \quad ( 1 5 ) \text{.}$$

As shown in Fig. 4, we design the same network block to learn W 1 , W 2 , and W 3 , which first reduces the number of channels to a quarter and then restores it to the original channel number. Meanwhile, an efficient Conv 1 × 1 operation is adopted to approximate W v .

As shown in Fig. 3, N and d are a multiplication of the width and height of a feature map and the number of channels. In general, N is far larger than d . According to Eq. (15), our STEA strategy reduces the complexity of the original NLA from O ( N 2 × d ) to O ( N × d 2 ) . The subsequent experimental results validate that our STEA is more efficient compared to the NLA module, however, exhibits a slight performance degradation due to the ignoring of remaining terms of Taylor extension and the use of the learning-based matrix. To compensate for such performance degradation, we design a Multi-Scale Large Field Reception (MLFR), which will be detailedly described in the following.

## B. Multi-Scale Large Field Reception (MLFR)

As mentioned in Section II-A, compared to the NLA module, our STEA achieves higher efficiency but sacrifices the ability to extract global features from the feature maps. To this end, we design a Multi-Scale Large Field Reception (MLFR) strategy to obtain a large receptive field, as shown in Fig. 5. For the unfilled parts of the dilated convolution, we use normal convolution to fill in the dilation. Specifically,

Fig. 6. The overview of our proposed RealNet.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-f2576396-5212-42b6-9343-218f38c7a4d6.jpg)

we process the input feature map X through three separate branches. In the first branch, the input is processed by a dilated convolution with an expansion rate of 6 and a depthwise convolution with a kernel size of 7 × 7 , respectively. The output results are concatenated along the channel dimension. Simultaneously, in the second branch, the input is processed by a dilated convolution with an expansion rate of 4 and a depthwise convolution with a kernel size of 5 × 5 , respectively. Similarly, the output results are concatenated along the channel dimension. In the last branch, the input is processed by a dilated convolution with an expansion rate of 2 and a depthwise convolution with a kernel size of 3 × 3 , respectively. Again, the output results are concatenated along the channel dimension. We represent the above execution process using the following equation:

$$O u t p u t _ { ( i, j ) } = & \text{Concat(DDWC} ( X ) _ { d r = i }, \\ & \text{DWC} ( X ) _ { k s = j } \underset { j = 3, 5, 7 } { \underset { j = 3, 5, 7 } { \sum } }, \quad ( 1 6 ) \quad \text{iferen} \\ & \text{For the}$$

where Concat( · , · ) stands for concatenate operation, DDWC ( ) · stands for dilation convolution operation on the input, dr is the dilation rate, and DWC ( ) · stands for depth-wise convolution operation on the input, ks is the convolution kernel size. Furthermore, each value of i corresponds uniquely to one value of j . Subsequently, we employ 1 × 1 convolutions to perform channel-wise feature fusion on the outputs of the three branches, effectively reducing the dimension of the channels:

$$\ F u s i o n _ { ( i, j ) } = \text{Conv} _ { 1 \times 1 } ( O u t p u t _ { ( i, j ) } ) _ { \substack { i = 2, 4, 6 } \\ j = 3, 5, 7 } }, \quad ( 1 7 ) \quad \text{To s} \\ \quad \text{in a lal}$$

where Fusion ( i,j ) represents the output obtained after channel-wise feature fusion of the input, and Conv 1 × 1 ( ) ·

denotes the 1 × 1 operation performed on the input. Similarly, each value of i corresponds uniquely to one value of j . In addition, the input X undergoes a 1 × 1 bypass convolution in addition to the aforementioned three branches:

$$B y p a s s = \text{Conv} _ { 1 \times 1 } ( X ). \text{ \quad \ \ } ( 1 8 )$$

The purpose of the 1 × 1 convolution in Eq. (18) is to perform subtle adjustments to the channels of the input X . Finally, we concatenate the outputs of the three branches with the output of the bypass 1 × 1 convolution along the channel dimension and perform channel-wise feature fusion using a 1 × 1 convolution to reduce the channel dimension:

$$\text{using} \quad F i n a l O u t p u t = C o n v _ { 1 \times 1 } ( C o n c a t ( F u s i o n _ { ( i, j ) }, \\ \ B y p a s s ) _ { \substack { i = 2, 4, 6 \\ j = 3, 5, 7 } } ). \quad ( 1 9 )$$

In summary, we utilize multiple dilated convolutions with different scales to extract features with various receptive fields. For the unpadded regions in dilated convolutions, we employ depth-wise convolutions to fill in the dilation. Additionally, we utilize 1 × 1 convolutions to perform subtle adjustments on the input as bypass convolutions. In the final step of feature fusion, we concatenate all the branches along the channel dimension and then employ a 1 × 1 convolution for channel-wise feature fusion, resulting in the final output as shown in Eq. (19).

## C. Application to Laboratory Scenario (LabNet)

To show the effectiveness of our STEA and MLFR designs in a laboratory scenario, we construct a novel network termed LabNet for testing. As shown in Fig. 2, our LabNet is an end-to-end blind SR framework composed of three parts:

Fig. 7. Users can adjust the eight parameters in the RealNet according to their needs. The above figure shows the effect obtained by adjusting different parameters. When adjusting one of the parameters, we set the other parameters to 0.01.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-83cda6d9-942d-4657-8463-c33fbd74bdf6.jpg)

shallow feature extraction, deep feature extraction, and HR image reconstruction. For the shallow feature extraction part, we employ a 3 × 3 convolution to extract shallow features from the input LR image. Subsequently, the extracted shallow features are fed into the UNet for deep feature extraction. Each LSTEA module in the UNet consists of multiple LSTEA blocks. When the input features pass through an LSTEA block, they undergo a channel-wise LayerNorm operation, followed by two branches. One branch utilizes multiple depthwise convolutions for local feature extraction, while the other branch employs the STEA and MLFR modules for large-scale feature extraction. The channel features are then fused using a 1 × 1 convolution and filtered using the channel attention module. Subsequently, the features pass through the GDFN [24] module, which performs a nonlinear mapping operation on the input to enhance network performance. Finally, for the HR image reconstruction part, the channel dimension is adjusted using a 3 × 3 convolution. The H and W dimensions are then expanded to the target size using a pixel shuffle operation, and a final refinement is performed using a 3 × 3 convolution to obtain the ultimate output results.

## D. Application to Real-world Scenario (RealNet)

To show the effectiveness of our STEA and MLFR designs in real-world scenarios, we construct a novel network termed

RealNet for testing. As illustrated in Fig. 6, our RealNet comprises three key components: shallow feature extraction, deep feature extraction, and HR image reconstruction. Firstly, we perform shallow feature extraction on the input LR image using two separate 3 × 3 convolutions. Subsequently, the extracted features are separately subjected to deep feature extraction in the denoising branch and the deblurring branch. Both the Denoising and Deblurring modules are composed of multiple LSTEA blocks. The features extracted from the Denoising module and the Deblurring module are then fused in proportion using Adapter modules. Finally, we employ a Fusion module for HR image reconstruction.

Adapter Module. For images contaminated by noise, there are numerous detailed features within the image that can be learned by the deblurring branch. Similarly, for blurry images, there are many low-frequency features within the image that can be learned by the denoising branch. Therefore, as depicted in Fig. 6, we propose the Adapter module. The parameters α and β control the proportions of denoising and deblurring, respectively. Analyzing the denoising branch, we take the features extracted by the deblurring branch in the previous step, apply a 3 × 3 convolution, and perform a channel-wise concatenation with the features extracted by the denoising branch. Subsequently, a 1 × 1 convolution is employed for channel-wise feature fusion, followed by a channel attention module for channel-wise feature selection. Similarly, a similar processing approach is utilized for the deblurring branch.

Fusion Module. To adjust the proportions of the outputs from the final denoising module and deblurring module, we also utilize two adjustable parameters ( α and β ). After adjusting the two sets of output feature maps, we perform a channel-wise concatenation and then apply a 1 × 1 convolution for channel information fusion and adjustment of channel dimensions. Finally, the output feature maps are resized to match the target size using a pixel shuffle operation and fine-tuned with a 3 × 3 convolution.

RealNet-GAN. To achieve better visual results, we employ a GAN training strategy for the proposed RealNet. Specifically, we use RealNet as the generator and utilize a U-Net as the discriminator [25]. The training process consists of two parts. Firstly, we train a PSNR-oriented model using the ℓ 1 loss, which is then used as the generator in the GAN training strategy. We then train a GAN-oriented model using a combination of ℓ 1 loss [26], perceptual loss [27], and adversarial loss [28]. The overall loss function of our network is

$$\mathcal { L } _ { e n t i r e } = \mathcal { L } _ { 1 } + \mathcal { L } _ { p e r c } + 0. 1 \times \mathcal { L } _ { a d v }, \quad \ ( 2 0 )$$

where L 1 , L perc , and L adv represent the ℓ 1 loss, perceptual loss, and adversarial loss, respectively. According to our empirical study, their weights are set to be 1 , 1 , and 0 1 . , respectively. We utilize the { conv1,..., conv5 } feature maps before activation in the pre-trained VGG19 [29] model (with weights { 0 1 . , 0 1 . , 1 , 1 , 1 } ) as the perceptual loss.

## III. EXPERIMENTS

## A. Datasets and Evaluation Metrics

We train all our proposed SR models using the DF2K dataset ( DIV2K [30] + Flickr2K [31]). There are 3450 2K-

TABLE I COMPARISON OF INFERENCE TIMES AND GFLOPS WITH DIFFERENT BLIND SR METHODS.

| (1,3,64,64) → (1,3,256,256)   | Infer time on GPU   |   GFLOPs |
|-------------------------------|---------------------|----------|
| DCLS                          | 0.86 s              |       46 |
| NLCUNet                       | 2.23 s              |      432 |
| LabNet (Ours)                 | 1.18 s              |       49 |

resolution images in the dataset. For the quantitative evaluation, we conducted tests on five standard super-resolution image benchmark datasets: Set5 [32], Set14 [33], BSD100 [34], Urban100 [35], and Manga109 [36]. To conduct a qualitative evaluation, we compared the visual results of our approach with previous methods on the RealWorld38 [25] dataset. The quantitative performance metrics we use for comparison are the Peak Signal-to-Noise Ratio (PSNR) [37] and Structural Similarity (SSIM) [38].

## B. Implementation Details of LabNet

Our LabNet is built on the foundation of the LSTEA module, forming a U-Net. Each LSTEA module consists of multiple LSTEA blocks, as shown in Fig. 2, the number of LSTEA blocks in each LSTEA module from left to right is: { 6 , 8 , 10 , 6 , 8 , 8 } . During the training process, we set the size of the input image to 64 × 64, and we use the Mean Absolute Error (MAE) as the loss function. The total iterations are 1.2 × 10 . The initial 6 learning rate is set to 4 × 10 -4 and is halved every 3 × 10 5 iterations. We use Adam [44] as the optimizer with β 1 = 0 9 . and β 2 = 0 99 . . All models are trained on Tesla T4 GPUs.

Data Preprocessing of LabNet. For the training dataset, a blur operation with a blur kernel size of 21 × 21 is applied. For different scaling factors (×2, ×3, and ×4), the variance of the blur kernel is uniformly sampled during the training process from the ranges [0.2, 2.0], [0.2, 3.0], and [0.2, 4.0], respectively. For the test dataset, we uniformly select 8 blur kernels from the ranges [0.80, 1.60], [1.35, 2.40], and [1.8, 3.2] for different scaling factors (×2, ×3, and ×4), respectively.

## C. Implementation Details of RealNet

Our RealNet is constructed by two branches: denoising and deblurring. As illustrated in Fig. 6, both the denoising branch and the deblurring branch consist of 4 modules, each composed of 5 LSTEA modules. The entire network is trained for 30 epochs. The initial learning rate is set to 1 × 10 -3 and is halved every 15 epochs. We use Adam as the optimizer with β 1 = 0 9 . and β 2 = 0 99 . . All models are trained on Tesla T4 GPUs.

Data Preprocessing of RealNet. As shown in Fig. 1, we first independently and in parallel apply second-order noise and second-order blur processing to the HR image, resulting in two corresponding LR images. Then, these two LR images are passed through the denoising and deblurring branches of RealNet, respectively. Finally, a fusion operation is performed to obtain the restored SR image.

## D. Comparisons with State-of-the-art

Quantitative Evaluations in Lab Scenarios. From Table II, it can be observed that our LabNet significantly outperforms other state-of-the-art deep SR models on nearly all benchmarks and scaling factors. For instance, compared to NLCUNet [20] with a scale factor of ×2, our LabNet achieves performance improvements of 0.03 dB, 0.05 dB, 0.11 dB, and 0.02 dB on the Set14, BSD100, Urban100, and Manga109 datasets, respectively. Similarly, compared to NLCUNet with a scale factor of ×4, our LabNet achieves performance improvements of 0.04 dB, 0.02 dB, and 0.11 dB on the Set14, BSD100, and Manga109 datasets, respectively. Our LabNet consistently outperforms DCLS [42] across all scale factors. Additionally, as shown in Table I, when the input image size is (1,3,64,64) and the scaling factor is ×4, our LabNet achieves an inference time of 1.18 seconds on a GPU, which is 1.05 seconds faster than NLCUNet but slightly slower than DCLS. Furthermore, our LabNet has significantly fewer GFLOPs compared to NLCUNet and is comparable to DCLS. In summary, our LabNet not only achieves comparable performance to NLCUNet but also exhibits higher efficiency. This can be attributed to the utilization of LSTEA in our LabNet, which combines STEA and MLFR to capture multi-scale features with a large receptive field while maintaining linear complexity.

Qualitative Evaluations in Lab Scenarios. As shown in Fig. 9, by comparing the reconstruction result of the ′ zebra ′ in Fig. 9, we can observe that the black and white stripes on the legs of the zebra generated by our LabNet are close to the GT image. Furthermore, by comparing the reconstruction result of ′ YamatoNoHane , we can observe that our LabNet ′ performs better in recovering the textual content in the image compared to other methods.

## Quantitative Evaluation in Real-World Scenarios. As

commonly observed, GAN-based SR methods often generate lower values in terms of PSNR and SSIM metrics while producing visually captivating results. As shown in Table II, Real-ESRGAN and BSRGAN perform significantly lower than non-GAN-based SR methods on the five benchmark test datasets. However, Fig. 10 reveals the impressive visual outcomes achieved by Real-ESRGAN and BSRGAN. Additionally, due to the absence of corresponding GT images for the degraded images in the RealWorld38 dataset, we only conducted visual comparisons between our RealNet and other blind SR methods.

Qualitative Evaluation in Real-World Scenarios. The experimental results are presented in Fig. 10. We evaluated the generalization capability of our model using the RealWorld38 dataset, which consists of real-world images. Our RealNet outperforms previous methods for reconstructing LR images in real-world scenarios. Among all the methods, the nondeep learning bicubic interpolation method performs the worst, followed by DANv2, DCLS, and NLCUNet. These methods only employ Gaussian blur and downsampling during training, resulting in visually unsatisfactory outcomes. BSRGAN [43] and Real-ESRGAN achieve favorable visual results due to their training on datasets with complex degradations. However, upon comparing the ′ OST 009 ′ image in Fig. 10, we observe

Fig. 8. The images in the above figure were randomly obtained by us from the Internet. We can adjust the eight parameters in RealNet at the same time according to the effect we need.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-c122541a-3f29-43ed-8bcb-8868f72f3fbc.jpg)

that the result obtained by BSRGAN exhibits a significant color deviation from the LR input, while the result produced by Real-ESRGAN evokes a strong sense of discordance. In contrast, our RealNet reconstruction result closely resembles the color of the input LR image and demonstrates improved overall coherence.

than FTEA. Ultimately, we used STEA to build our LabNet and RealNet.

## E. Ablation Study and Analysis of LabNet

- 1) Impact of STEA : Table III showcases the effect of employing first-order Taylor expansion approximation (FTEA) and second-order Taylor expansion approximation (STEA) on the performance of blind SR tasks. By comparing the PSNR and SSIM values of the first and second rows in Table III, we observed an improvement in performance across all test datasets when DWC( ) was applied to the Value. Furthermore, · comparing the PSNR and SSIM values of the second and third rows in Table III, we found that utilizing STEA for the exponential function outperformed FTEA across all test datasets. Particularly on the Urban100 dataset, the PSNR was higher by 0.11 dB when using a STEA compared to a FTEA. Furthermore, we found that when the order of Taylor expansion is greater than 2, the performance does not show a significant improvement.

Efficiency Analysis. As shown in Table IV, when applying DWC( ) to Value, there is a slight increase in the number of · parameters and a slight rise in MFLOPs. By comparing rows 2 to 4 in Table IV, as the order of Taylor expansion increases, the third-order Taylor expansion approximation (TTEA) parameter count and MFLOPs are greater than STEA, which are greater

- 2) Impact of MLFR : Table VI highlights the influence of incorporating multi-scale dilated convolutions in MLFR on blind SR performance. By comparing the PSNR and SSIM values of the first to third rows in Table VI, we found that in MLFR, when utilizing dilated convolutions with dilations of 2, 4, and 6 simultaneously, the performance was the best across all test datasets. Particularly on the Urban100 dataset, the PSNR was 0.12 dB higher when using dilated convolutions with dilations of 2, 4, and 6 compared to dilated convolutions with dilations of 4 and 6.

Efficiency Analysis. As shown in Table V, in MLFR, the maximum number of parameters and MFLOPs are achieved when utilizing dilated convolutions with Dilated = 6, 4, and 2 simultaneously.

- 3) Impact of STEA and MLFR : As shown in Table VII, LabNet achieves the best performance on all five benchmarks when both STEA and MLFR are used simultaneously.

## F. Ablation Study and Analysis of RealNet

The experimental results from Table VIII highlight the following insights: Our RealNet without NLA has the fastest inference speed, processing a 64 × 64 input image in just 0.40 seconds. In contrast, Our RealNet with NLA and NLSA have much higher computational costs, with inference times of 4.21 seconds and 5.15 seconds, respectively. Our RealNet with the STEA, achieves high efficiency by processing an input image in only 0.48 seconds. Our RealNet with the STEA and

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-4a57341e-884e-40c3-8202-7763c2be25fd.jpg)

zebra

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-e04cf70a-6f39-404b-8e32-cf61fe752661.jpg)

21077

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-9e91199d-0bd8-40fa-80fc-4ab102061da0.jpg)

img\_100

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-46a06aed-d68b-4aca-bed0-4c4443df26c4.jpg)

DANv2

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-a820f8fc-c234-4337-9a20-009a390db3c6.jpg)

LabNet (Ours)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-cc1de000-d4ae-4471-8932-69653f88b3b3.jpg)

DANv2

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-a9468b70-11a5-45b1-8a1a-3dc4616b20a5.jpg)

LabNet (Ours)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-cfe44074-095a-4356-81b1-4d968009f103.jpg)

DANv2

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-4c4f29a9-211f-4a99-ba0c-4d77d5cf6f20.jpg)

LabNet (Ours)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-9fffefdf-805c-4d3c-b1b2-fab2cf4d68bb.jpg)

HR

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-e59e1c6a-9563-4b95-8aaa-9c18a1cc8802.jpg)

DCLS

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-eec5689f-6973-454d-95ab-086ddb40fc7b.jpg)

HR

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-18378136-1c8e-45aa-85a9-3eb875dbd56b.jpg)

DCLS

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-d637f2ed-ce1f-454e-8218-7b4bf3cb108c.jpg)

HR

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-ca8593d4-fe56-45d9-a085-a2f09298b413.jpg)

DCLS

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-e21317fc-9875-405f-acc3-43ff3024278f.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-9a7091aa-e6a9-47a3-aeab-b370fd9d2340.jpg)

NLCUNet

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-3a357e6c-aadb-4a21-b622-83f44095ee9e.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-4e9403e3-aa5e-4348-a784-9dd60da77bf0.jpg)

NLCUNet

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-c3e10a1f-dd23-4e0a-a5fb-b34aa46a6c56.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-1452ef3f-fb0f-46b1-af51-39316d20ed18.jpg)

NLCUNet

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-071b6363-594b-47b0-9175-3a398f038e84.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-414df74f-a21b-4dbe-b923-3b1c954d4889.jpg)

DANv2

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-a9aeb0cb-9997-43dc-aeff-2311a09a9a21.jpg)

HR

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-06956cb9-30b0-4861-b66b-72fae6c2072e.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-deced5fb-b615-4794-a286-d21fdbcb97e7.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-3928fad2-cf2a-4eb6-b9ee-6c2f506b1dd9.jpg)

YamatoNoHane

LabNet (Ours)

NLCUNet

Fig. 9. Visual results are from Set14, BSD100, Urban100 and Manga109 at scale x4. Here, the width of the blur kernel is 3.2.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-963602e7-2bbb-457d-8c42-013ec7c2ad2b.jpg)

TABLE II

QUANTITATIVE RESULTS ON SISR BENCHMARK DATASETS. THE BEST AND THE SECOND BEST RESULTS ARE HIGHLIGHTED AND UNDERLINED. ALL METHODS UNDER THE DOTTED LINE WERE TESTED USING THE CROPPED DATASETS.

| Method                    | Scale   | Set5   | Set5          | Set14       | Set14         | BSD100      | BSD100        | Urban100    | Urban100      | Manga109    | Manga109      |
|---------------------------|---------|--------|---------------|-------------|---------------|-------------|---------------|-------------|---------------|-------------|---------------|
| Method                    | Scale   | PSNR   | SSIM          | PSNR        | SSIM          | PSNR        | SSIM          | PSNR        | SSIM          | PSNR        | SSIM          |
| Bicubic                   |         | 28.82  | 0.8577        | 26.02       | 0.7634        | 25.92       | 0.7310        | 23.14       | 0.7258        | 25.60       | 0.8498        |
| CARN [39]                 |         | 30.99  | 0.8779        | 28.10       | 0.7879        | 26.78       | 0.7286        | 25.27       | 0.7630        | 26.86       | 0.8606        |
| Bicubic+ZSSR [40]         |         | 31.08  | 0.8786        | 28.35       | 0.7933        | 27.92       | 0.7632        | 25.25       | 0.7618        | 28.05       | 0.8769        |
| Deblurring [41]+CARN [40] |         | 24.20  | 0.7496        | 21.12       | 0.6170        | 22.69       | 0.6471        | 18.89       | 0.5895        | 21.54       | 0.7946        |
| CARN [40]+Deblurring [41] |         | 31.27  | 0.8974        | 29.03       | 0.8267        | 28.72       | 0.8033        | 25.62       | 0.7981        | 29.58       | 0.9134        |
| IKC [16]                  |         | 37.19  | 0.9526        | 32.94       | 0.9024        | 31.51       | 0.8790        | 29.85       | 0.8928        | 36.93       | 0.9667        |
| DANv1 [17]                |         | 37.24  | 0.9551        | 33.07       | 0.9042        | 31.75       | 0.8857        | 30.59       | 0.9059        | 37.20       | 0.9709        |
| DANv2 [18]                |         | 37.54  | 0.9571        | 33.44       | 0.9095        | 31.99       | 0.8904        | 31.43       | 0.9173        | 38.05       | 0.9733        |
| DCLS [42]                 |         | 37.60  | 0.9552        | 33.46       | 0.9105        | 32.03       | 0.8906        | 31.69       | 0.9202        | 38.28       | 0.9739        |
| NLCUNet [20]              |         | 37.71  | 0.9571        | 33.60       | 0.9092        | 32.09       | 0.8901        | 31.77       | 0.9202        | 38.60       | 0.9744        |
| BSRGAN [43]               |         | 32.24  | 0.9096        | 29.53       | 0.8336        | 28.88       | 0.8021        | 26.70       | 0.8303        | 30.78       | 0.9189        |
| Real-ESRGAN [25]          |         | 30.20  | 0.8906        | 27.96       | 0.7966        | 27.56       | 0.7639        | 25.37       | 0.7921        | 29.18       | 0.9046        |
| LabNet (Ours)             |         | 37.68  | 0.9575        | 33.63       | 0.9111        | 32.14       | 0.8924        | 31.88       | 0.9220        | 38.62       | 0.9748        |
| Bicubic                   |         | 26.21  | 0.7766        | 24.01       | 0.6662        | 24.25       | 0.6356        | 21.39       | 0.6203        | 22.98       | 0.7576        |
| CARN [39]                 |         | 27.26  | 0.7855        | 25.06       | 0.6676        | 25.85       | 0.6566        | 22.67       | 0.6323        | 23.85       | 0.7620        |
| Bicubic+ZSSR [40]         |         | 28.25  | 0.7989        | 26.15       | 0.6942        | 26.06       | 0.6633        | 23.26       | 0.6534        | 25.19       | 0.7914        |
| Deblurring [41]+CARN [40] |         | 19.05  | 0.5226        | 17.61       | 0.4558        | 20.51       | 0.5331        | 16.72       | 0.5895        | 18.38       | 0.6118        |
| CARN [40]+Deblurring [41] |         | 30.31  | 0.8562        | 27.57       | 0.7531        | 27.14       | 0.7152        | 24.45       | 0.7241        | 27.67       | 0.8592        |
| IKC [16]                  |         | 33.06  | 0.9146        | 29.38       | 0.8233        | 28.53       | 0.7899        | 24.43       | 0.8302        | 32.43       | 0.9316        |
| DANv1 [17]                |         | 33.81  | 0.9199        | 30.08       | 0.8287        | 28.94       | 0.7932        | 27.64       | 0.8351        | 33.08       | 0.9375        |
| DANv2 [18]                |         | 33.87  | 0.9208        | 30.20       | 0.8311        | 29.02       | 0.7962        | 27.82       | 0.8395        | 33.24       | 0.9398        |
| DCLS [42]                 |         | 33.95  | 0.9214        | 30.28       | 0.8328        | 29.06       | 0.7970        | 28.02       | 0.8443        | 33.48       | 0.9412        |
| NLCUNet [20]              |         | 34.31  | 0.9231        | 30.37       | 0.8337        | 29.14       | 0.7983        | 28.30       | 0.8479        | 33.71       | 0.9436        |
| LabNet (Ours)             |         | 34.21  | 0.9228        | 30.38       | 0.8349        | 29.14       | 0.7998        | 28.25       | 0.8494        | 33.73       | 0.9431        |
| Bicubic                   |         | 24.57  | 0.7108        | 22.79       | 0.6032        | 23.29       | 0.5786        | 20.35       | 0.5532        | 21.50       | 0.6933        |
| CARN [39]                 |         | 26.57  | 0.7420        | 24.62       | 0.6226        | 24.79       | 0.5963        | 22.17       | 0.5865        | 21.85       | 0.6834        |
| Bicubic+ZSSR [40]         |         | 26.45  | 0.7279        | 24.78       | 0.6268        | 24.97       | 0.5989        | 22.11       | 0.5865        | 23.53       | 0.7240        |
| Deblurring [41]+CARN [40] |         | 18.10  | 0.4843        | 16.59       | 0.3994        | 18.46       | 0.4481        | 15.47       | 0.3872        | 16.78       | 0.5371        |
| CARN [40]+Deblurring [41] |         | 28.69  | 0.8092        | 26.40       | 0.6926        | 26.10       | 0.6528        | 23.46       | 0.6597        | 25.84       | 0.8035        |
| IKC [16]                  |         | 31.67  | 0.8829        | 28.31       | 0.7643        | 27.37       | 0.7192        | 25.33       | 0.7504        | 28.91       | 0.8782        |
| DANv1 [17]                |         | 31.83  | 0.8868        | 28.40       | 0.7687        | 27.82       | 0.7457        | 25.05       | 0.7365        | 29.78       | 0.8910        |
| DANv2 [18]                |         | 31.96  | 0.8891        | 28.48       |               | 27.86       | 0.7484        | 25.13       | 0.7386        | 29.73       | 0.8911        |
| DCLS [42]                 |         | 32.07  |               |             | 0.7715        |             |               |             |               |             |               |
| NLCUNet [20]              |         | 32.29  | 0.8893 0.8931 | 28.52 28.66 | 0.7727 0.7751 | 27.89 27.96 | 0.7488 0.7518 | 25.34 25.69 | 0.7465 0.7597 | 30.07 30.16 | 0.8968 0.9004 |
| BSRGAN [43]               |         | 27.47  | 0.7913        | 25.64       | 0.6638        | 25.64       | 0.6510        | 22.35       | 0.6133        | 25.00       | 0.7785        |
| Real-ESRGAN [25]          |         | 26.16  | 0.7919        | 24.91       | 0.6661        | 25.11       | 0.6553        | 21.38       | 0.6002        | 24.71       | 0.7882        |
|                           |         | 32.27  | 0.8923        | 28.70       |               | 27.98       | 0.7536        | 25.62       | 0.7583        | 30.27       |               |
| LabNet (Ours)             |         |        |               |             | 0.7769        |             |               |             |               |             | 0.9011        |

TABLE III

THE ABLATION EXPERIMENTS OF THE STEA MODULE. WE COMPARED THE EFFECTS OF FIRST-ORDER TAYLOR EXPANSION APPROXIMATION (FTEA) AND SECOND-ORDER TAYLOR EXPANSION APPROXIMATION (STEA) ON THE PERFORMANCE, PARAMETERS, AND MFLOPS OF SR (WITH SCALE ×4). THE TEST DATASETS WERE SET5, URBAN100, AND MANGA109. THE BEST AND SECOND-BEST RESULTS ARE HIGHLIGHTED AND UNDERLINED.

TABLE IV COMPARISON OF MFLOPS AND PARAMETERS IN DIFFERENT CONFIGURATIONS.

| DWC(V)   | FTEA   | STEA   | Set5   | Set5   | Urban100   | Urban100   | Manga109   | Manga109   |
|----------|--------|--------|--------|--------|------------|------------|------------|------------|
|          |        |        | PSNR   | SSIM   | PSNR       | SSIM       | PSNR       | SSIM       |
| %        | !      | %      | 31.97  | 0.8886 | 25.19      | 0.7417     | 29.94      | 0.8946     |
| !        | !      | %      | 32.04  | 0.8891 | 25.20      | 0.7429     | 29.93      | 0.8954     |
| !        | %      | !      | 32.07  | 0.8896 | 25.31      | 0.7462     | 29.98      | 0.8964     |

TABLE V IN THE CASE OF USING DWC(V) AND STEA, COMPARISON OF MFLOPS AND PARAMETERS FOR MLFR IN DIFFERENT CONFIGURATIONS.

| DWC(V)   | FTEA   | STEA   | TTEA   | # Params   |   MFLOPs |
|----------|--------|--------|--------|------------|----------|
| %        | !      | %      | %      | 10.6K      |       33 |
| !        | !      | %      | %      | 16.4K      |       63 |
| !        | %      | !      | %      | 19.1K      |       71 |
| !        | %      | %      | !      | 21.8K      |       80 |

the MLFR strikes a favorable balance between efficiency and visual result, processing an input image in 0.53 seconds.

| MLFR v1   | MLFR v2   | MLFR v3   | # Params   |   MFLOPs |
|-----------|-----------|-----------|------------|----------|
| !         | %         | %         | 50.5K      |      198 |
| %         | !         | %         | 64.1K      |      251 |
| %         | %         | !         | 89.9K      |      358 |

## G. Details of adjustable parameters in RealNet

Adjusting a single parameter: Users can adjust the values of the denoising and deblurring parameters in RealNet to achieve the desired effects. In the visualization shown in Fig. 7, Increasing α 1 slightly enhances image details, but

LR (No GT)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-d96b9eea-e688-484f-8939-234608f2d8d4.jpg)

frog

LR (No GT)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-44fb190c-262c-42e5-b78c-39125ff0e046.jpg)

DANv1

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-3cae8aed-0fac-4891-8003-071bc4f80901.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-ba331673-dacc-45a5-9c66-0a8405ccd374.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-175af926-a316-4b8f-af51-fcd72b1f4de4.jpg)

NLCUNet

Real-ESRGAN

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-2851f9cc-436d-4ad1-bc82-d84a2719c115.jpg)

DANv1

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-f6ccf9da-d333-4e22-a139-6d2dc49abf5b.jpg)

NLCUNet

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-4ef14ec2-1357-47f9-901c-7516fafa4dea.jpg)

Real-ESRGAN

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-4a22e28c-871c-4496-b6f6-d37625d3755c.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-bb6b4407-3a20-4cce-a63a-1bf6fbc34094.jpg)

RealNet (Ours)

RealNet (Ours)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-1cd7bed2-404f-46ce-b039-64631b6077a9.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-07dffa7d-0954-44fb-860a-26b130c93648.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-b5e07863-2350-40ed-be95-ffa49637c148.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-8abf8d39-b06d-49e7-8007-0bb059875763.jpg)

BSRGAN

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-1898755d-0f6b-4c87-a6cd-3dc504db0852.jpg)

Bicubic

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-d524e497-8c90-450a-911f-62a2117729b9.jpg)

DCLS

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-f47daf69-b57f-4d10-928a-2cc1999f51ee.jpg)

BSRGAN

Ours

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-a22a2522-9199-449e-bc38-eacd74515a96.jpg)

Ours

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00470v1-figure-304e93ee-e7b6-4c9b-904b-890f8c6cc276.jpg)

OST\_009

Fig. 10. The visualization results consist of various methods, including the non-deep learning approach Bicubic, as well as the DANv1, DANv2, DCLS, and NLCUNet, which utilize only Gaussian blurring and downsampling during training. Additionally, the visualization includes the Real-ESRGAN, BSRGAN, and our RealNet, which involve training on datasets with complex degradations.

## TABLE VI

THE ABLATION EXPERIMENTS WERE CONDUCTED ON THE MLFR MODULE, USING DWC(V) AND STEA, WHERE MLFR V1 USES DEPTH-WISE CONVOLUTION WITH A KERNEL SIZE OF 9 AND A DILATION RATE OF 6, MLFR V2 USES THE SAME DEPTH-WISE CONVOLUTION AS MLFR V1, BUT ALSO INCLUDES AN ADDITIONAL DEPTH-WISE CONVOLUTION WITH A KERNEL SIZE OF 9 AND A DILATION RATE OF 4, AND MLFR V3 USES THE SAME CONFIGURATIONS AS MLFR V2, BUT ALSO INCLUDES AN ADDITIONAL DEPTH-WISE CONVOLUTION WITH A KERNEL SIZE OF 9 AND A DILATION RATE OF 2. THE BEST AND SECOND-BEST RESULTS ARE HIGHLIGHTED AND UNDERLINED, RESPECTIVELY.

TABLE VII

| MLFR v1   | MLFR v2   | MLFR v3   | Set5   | Set5   | Urban100   | Urban100   | Manga109   | Manga109   |
|-----------|-----------|-----------|--------|--------|------------|------------|------------|------------|
|           |           |           | PSNR   | SSIM   | PSNR       | SSIM       | PSNR       | SSIM       |
| !         | %         | %         | 32.09  | 0.8907 | 25.53      | 0.7539     | 30.31      | 0.8998     |
| %         | !         | %         | 32.23  | 0.8920 | 25.50      | 0.7527     | 30.25      | 0.8998     |
| %         | %         | !         | 32.27  | 0.8923 | 25.62      | 0.7583     | 30.27      | 0.9011     |

ABLATION EXPERIMENTS ON THE COMBINED AND INDIVIDUAL EFFECTS OF STEA AND MLFR. THE BEST AND THE SECOND BEST RESULTS ARE HIGHLIGHTED AND UNDERLINED.

| STEA   | MLFR   | Set5   | Set5   | Set14   | Set14   | BSD100   | BSD100   | Urban100   | Urban100   | Manga109   | Manga109   |
|--------|--------|--------|--------|---------|---------|----------|----------|------------|------------|------------|------------|
|        | MLFR   | PSNR   | SSIM   | PSNR    | SSIM    | PSNR     | SSIM     | PSNR       | SSIM       | PSNR       | SSIM       |
| !      | %      | 32.07  | 0.8896 | 28.51   | 0.7713  | 27.89    | 0.7481   | 25.31      | 0.7462     | 29.98      | 0.8964     |
| %      | !      | 32.10  | 0.8898 | 28.58   | 0.7734  | 27.93    | 0.7489   | 25.43      | 0.7485     | 30.14      | 0.8972     |
| !      | !      | 32.27  | 0.8923 | 28.70   | 0.7769  | 27.98    | 0.7536   | 25.62      | 0.7583     | 30.27      | 0.9011     |

TABLE VIII

COMPARISON OF INFERENCE TIME, NUMBER OF PARAMETERS, AND GFLOPS FOR REALNET IN DIFFERENT CONFIGURATIONS.

| Scale: × 4       | Infer time on GPU   | # Params   |   GFLOPs |
|------------------|---------------------|------------|----------|
| Without NLA      | 0.40 s              | 4 M        |       17 |
| With NLA         | 4.21 s              | 5.3 M      |      194 |
| With NLSA        | 5.15 s              | 5.4 M      |       69 |
| With STEA        | 0.48 s              | 5.8 M      |       25 |
| With STEA + MLFR | 0.53 s              | 11.5 M     |       47 |

increasing α 3 improves the distinction between objects and the background, but setting it too high can negatively impact the image. α 2 progressively enhances image visualization as it increases, but setting it to extremely large values can damage the image. Careful adjustment of α 4 is necessary to prevent color distortion, as it is related to the deep layer. It is advisable to avoid setting β 1 to β 4 to excessively large values to prevent color distortion in the resulting image.

excessive increase may lead to distorted edges. Gradually

Joint modification of multiple parameters: Users can still fine-tune the denoising and deblurring ratios (i.e., α i and β i ) according to their preferences, even after the training process.

In the visualization shown in Fig. 8, the second and third rows display the restored results using different ratios for denoising and deblurring. This demonstrates that RealNet is capable of restoring rich texture information from LR images, catering to different user preferences.

## IV. CONCLUSION

In this paper, we presented a second-order Taylor expansion approximation (STEA) to handle the high computational complexity of self-similarity techniques. Then, we designed a multi-scale large-field reception (MLFR) to compensate for the super-resolution (SR) performance loss caused by STEA. To confirm the superiority of our designed STEA and MLFR, we respectively constructed the LabNet for testing in laboratory scenarios and the RealNet for testing in real-world scenarios. Extensive experimental results demonstrated the effectiveness of our method in blind SR tasks. The ablation study further validates their contribution to the overall framework.

## REFERENCES

- [1] D. Glasner, S. Bagon, and M. Irani, 'Super-resolution from a single image,' in IEEE Conference on Computer Vision , 2009, pp. 349-356.
- [2] C. Dong, C. Loy, K. He, and X. Tang, 'Learning a deep convolutional network for image super-resolution,' in European Conference on Computer Vision , 2014, pp. 184-199.
- [3] J. Kim, J. Lee, and K. Lee, 'Accurate image superresolution using very deep convolutional networks,' in IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 1646-1654.
- [4] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, 'Enhanced deep residual networks for single image superresolution,' in IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2017, pp. 136-144.
- [5] D. Zhang, J. Shao, Z. Liang, L. Gao, and H. Shen, 'Large factor image super-resolution with cascaded convolutional neural networks,' IEEE Transactions on Multimedia , vol. 23, pp. 2172-2184, 2020.
- [6] K. Park, J. Soh, and N. Cho, 'A dynamic residual selfattention network for lightweight single image superresolution,' IEEE Transactions on Multimedia , vol. 25, pp. 907-918, 2021.
- [7] X. Zhang, P. Gao, S. Liu, K. Zhao, G. Li, L. Yin, and C. Chen, 'Accurate and efficient image super-resolution via global-local adjusting dense network,' IEEE Transactions on Multimedia , vol. 23, pp. 1924-1937, 2020.
- [8] Y. Liu, S. Wang, J. Zhang, S. Wang, S. Ma, and W. Gao, 'Iterative network for image super-resolution,' IEEE Transactions on Multimedia , vol. 24, pp. 2259-2272, 2021.
- [9] W. Li, J. Li, G. Gao, W. Deng, J. Zhou, J. Yang, and G. Qi, 'Cross-receptive focused inference network for lightweight image super-resolution,' IEEE Transactions on Multimedia , vol. 26, pp. 864-877, 2023.
- [10] W. Shi, J. Caballero, F. Husz´ ar, J. Totz, A. Aitken, R. Bishop, D. Rueckert, and Z. Wang, 'Real-time single
- image and video super-resolution using an efficient subpixel convolutional neural network,' in IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 1874-1883.
- [11] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, 'Image super-resolution using very deep residual channel attention networks,' in European Conference on Computer Vision , 2018, pp. 286-301.
- [12] K. Zhang, W. Zuo, and L. Zhang, 'Learning a single convolutional super-resolution network for multiple degradations,' in IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 3262-3271.
- [13] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, 'Plug-and-play image restoration with deep denoiser prior,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 10, pp. 63606376, 2021.
- [14] K. Zhang, L. Gool, and R. Timofte, 'Deep unfolding network for image super-resolution,' in IEEE Conference on Computer Vision and Pattern Recognition , 2020, pp. 3217-3226.
- [15] J. Liang, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, 'Mutual affine network for spatially variant kernel estimation in blind image super-resolution,' in IEEE International Conference on Computer Vision , 2021, pp. 4096-4105.
- [16] J. Gu, H. Lu, W. Zuo, and C. Dong, 'Blind superresolution with iterative kernel correction,' in IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 1604-1613.
- [17] Y. Huang, S. Li, L. Wang, T. Tan, et al., 'Unfolding the alternating optimization for blind super resolution,' Advances in Neural Information Processing Systems , pp. 5632-5643, 2020.
- [18] Z. Luo, Y. Huang, S. Li, L. Wang, and T. Tan, 'End-toend alternating optimization for blind super resolution,' arXiv preprint arXiv:2105.06878 , 2021.
- [19] K. Zhang, W. Zuo, and L. Zhang, 'Deep plug-andplay super-resolution for arbitrary blur kernels,' in IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 1671-1681.
- [20] J. Feng, Y. Wang, and F. Xing, 'Nlcunet: Single-image super-resolution network with hairline details,' in IEEE International Conference on Multimedia and Expo , 2023, pp. 1277-1282.
- [21] Y. Mei, Y. Fan, and Y. Zhou, 'Image super-resolution with non-local sparse attention,' in IEEE Conference on Computer Vision and Pattern Recognition , 2021, pp. 3517-3526.
- [22] X. Wang, R. Girshick, A. Gupta, and K. He, 'Nonlocal neural networks,' in IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7794-7803.
- [23] D. Han, X. Pan, Y. Han, S. Song, and G. Huang, 'Flatten transformer: Vision transformer using focused linear attention,' arXiv preprint arXiv:2308.00442 , 2023.
- [24] S. Zamir, A. Arora, S. Khan, M. Hayat, F. Khan, and M. Yang, 'Restormer: Efficient transformer for highresolution image restoration,' in IEEE Conference on

- Computer Vision and Pattern Recognition , 2022, pp. 5728-5739.
- [25] X. Wang, L. Xie, C. Dong, and Y. Shan, 'Realesrgan: Training real-world blind super-resolution with pure synthetic data,' in IEEE International Conference on Computer Vision , 2021, pp. 1905-1914.
- [26] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, 'Loss functions for image restoration with neural networks,' IEEE Transactions on Computational Imaging , vol. 3, no. 1, pp. 47-57, 2016.
- [27] J. Johnson, A. Alahi, and L. Fei-Fei, 'Perceptual losses for real-time style transfer and super-resolution,' in European Conference on Computer Vision , 2016, pp. 694-711.
- [28] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial networks,' Communications of the ACM , vol. 63, no. 11, pp. 139-144, 2020.
- [29] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' arXiv preprint arXiv:1409.1556 , 2014.
- [30] E. Agustsson and R. Timofte, 'Ntire 2017 challenge on single image super-resolution: Dataset and study,' in IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2017, pp. 126-135.
- [31] R. Timofte, E. Agustsson, L. Van Gool, M. Yang, and L. Zhang, 'Ntire 2017 challenge on single image superresolution: Methods and results,' in IEEE Conference on Computer Vision and Pattern Recognition Workshops , 2017, pp. 114-125.
- [32] M. Bevilacqua, A. Roumy, C. Guillemot, and M. AlberiMorel, 'Low-complexity single-image super-resolution based on nonnegative neighbor embedding,' in British Machine Vision Conference , 2012, pp. 1-10.
- [33] R. Zeyde, M. Elad, and M. Protter, 'On single image scale-up using sparse-representations,' in International Conference on Curves and Surfaces , 2010, pp. 711-730.
- [34] D. Martin, C. Fowlkes, D. Tal, and J. Malik, 'A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,' in IEEE International Conference on Computer Vision , 2001, pp. 416-423.
- [35] J. Huang, A. Singh, and N. Ahuja, 'Single image superresolution from transformed self-exemplars,' in IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 5197-5206.
- [36] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, 'Sketch-based manga retrieval using manga109 dataset,' Multimedia Tools and Applications , vol. 76, no. 20, pp. 21811-21838, 2017.
- [37] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The unreasonable effectiveness of deep features as a perceptual metric,' in IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 586-595.
- [38] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, 'Image quality assessment: from error visibility to structural similarity,' IEEE Transactions on Image Processing , vol.
- 13, no. 4, pp. 600-612, 2004.
- [39] N. Ahn, B. Kang, and K. Sohn, 'Fast, accurate, and lightweight super-resolution with cascading residual network,' in European Conference on Computer Vision , 2018, pp. 252-268.
- [40] A. Shocher, N. Cohen, and M. Irani, 'zero-shot: superresolution using deep internal learning,' in IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 3118-3126.
- [41] J. Pan, D. Sun, H. Pfister, and M. Yang, 'Deblurring images via dark channel prior,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 10, pp. 2315-2328, 2017.
- [42] Z. Luo, H. Huang, L. Yu, Y. Li, H. Fan, and S. Liu, 'Deep constrained least squares for blind image superresolution,' in IEEE Conference on Computer Vision and Pattern Recognition , 2022, pp. 17642-17652.
- [43] K. Zhang, J. Liang, L. Van Gool, and R. Timofte, 'Designing a practical degradation model for deep blind image super-resolution,' in IEEE International Conference on Computer Vision , 2021, pp. 4791-4800.
- [44] D. P Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014.
Jiancong Feng Yuan-Gen Wang Mingjie Li Fengchuang Xing
Image Super-Resolution with Taylor Expansion Approximation and Large Field Reception
Self-similarity techniques are booming in blind super-resolution (SR) due to
accurate estimation of the degradation types involved in low-resolution images.
However, high-dimensional matrix multiplication within self-similarity
computation prohibitively consumes massive computational costs. We find that
the high-dimensional attention map is derived from the matrix multiplication
between Query and Key, followed by a softmax function. This softmax makes the
matrix multiplication between Query and Key inseparable, posing a great
challenge in simplifying computational complexity. To address this issue, we
first propose a second-order Taylor expansion approximation (STEA) to separate
the matrix multiplication of Query and Key, resulting in the complexity
reduction from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$. Then, we design a
multi-scale large field reception (MLFR) to compensate for the performance
degradation caused by STEA. Finally, we apply these two core designs to
laboratory and real-world scenarios by constructing LabNet and RealNet,
respectively. Extensive experimental results tested on five synthetic datasets
demonstrate that our LabNet sets a new benchmark in qualitative and
quantitative evaluations. Tested on the RealWorld38 dataset, our RealNet
achieves superior visual quality over existing methods. Ablation studies
further verify the contributions of STEA and MLFR towards both LabNet and
RealNet frameworks.