text:
\section{Introduction}
\IEEEPARstart{S}{ingle} image super-resolution (SISR) is an important branch in the field of image processing \cite{wang2020deep}. It also aims to recover a high-resolution (HR) image over a low-resolution (LR) image \cite{yang2019deep}, leading to in its wide applications in medical diagnosis \cite{isaac2015super}, video surveillance \cite{zhang2010super} and disaster relief \cite{zha2015bayesian} etc. For instance, in the medical field, obtaining higher-quality images can help doctors accurately detect diseases \cite{huang2017simultaneous}. Thus, studying SISR is very meaningful to academia and industry.

To address SISR problem, researchers have developed a variety of methods based on degradation models of low-level vision tasks \cite{zhou2014local}. There are three categories for SISR in general, i.e., image itself information, prior knowledge and machine learning. In the image itself information, directly amplifying resolutions of all pixels in a LR image through an interpolation way to obtain a HR image was a simple and efficient method in SISR \cite{park2003super}, i.e., nearest neighbor interpolation \cite{rukundo2012nearest}, bilinear interpolation \cite{li2001new} and bicubic interpolation \cite{keys1981cubic}, etc. It is noted that in these interpolation methods, high-frequency information is lost in the up-sampling process \cite{park2003super}. Alternatively, reconstruction-based methods were developed for SISR, according to optimization methods \cite{hardeep2013survey}. That is, mapping a projection into a convex set to estimate the registration parameters can restore more details of SISR \cite{stark1989high}. Although the mentioned methods can overcome the drawbacks of image itself information methods, they still suffered the following challenges: non-unique solution, slow convergence speed and higher computational costs. To prevent this phenomenon, the priori knowledge and image itself information were integrated into a frame to find an optimal solution to improve the quality of the predicted SR images \cite{elad1997restoration,irani1991improving}. Using maximum a posteriori (MAP) can regularize a loss function to obtain a maximum probability for improving the efficiency \cite{cheeseman1996super}. Besides, machine learning methods can be presented to deal with SISR, according to relation of data distribution \cite{sun2008image}. On the basis of ensuring the image SR effect, sparse-neighbor-embedding-based (SpNE) method via partition the training data set into a set of subsets to accelerate the speed of SR reconstruction \cite{gao2012image}. There are also many other SR methods \cite{sun2008image,yan2015single} that often adopt sophisticated prior knowledge to restrict the possible solution space with an advantage of generating flexible and sharp detail. However, the performance of these methods rapidly degrades  when the scale factor is increased, and these methods tend to be time-consuming \cite{nasrollahi2014super}.

To obtain a better and more efficient SR model, a variety of deep learning methods were applied to a large-scale image dataset to solve the super-resolution tasks. For instance, Dong et al. proposed a super-resolution convolutional neural network (SRCNN) based pixel mapping that used only three layers to obtain stronger learning ability than these of some popular machine learning methods on image super-resolution \cite{dong2015image}. Although the SRCNN had a good SR effect, it still faced problems in terms of shallow architecture and high complexity. To overcome challenges of shallow architectures, Kim et al. \cite{kim2016accurate} designed a deep architecture by stacking some small convolutions to improve performance of image super-resolution. Tai et al. \cite{tai2017image} relied on recursive and residual operations in a deep network to enhance learning ability of a SR model. To further improve the SR effect, Lee et al. \cite{lim2017enhanced} used weights to adjust residual blocks to achieve better SR performance. To extract robust information, the combination of traditional machine learning methods and deep networks can restore more detailed information for SISR \cite{wang2015deep}. For instance, Wang et al. \cite{wang2015deep} embedded sparse coding method into a deep neural network to make a tradeoff between performance and efficiency in SISR. To reduce the complexity, an up-sampling operation is used in a deep layer in a deep CNN to increase the resolution of low-frequency features and produce high-quality images \cite{dong2016accelerating}. For example, Dong et al. \cite{dong2016accelerating} directly exploited the given low-resolution images to train a SR model for improving training efficiency, where the SR network used a deconvolution layer to reconstructing HR images. There are also other effective SR methods. For example, Lai et al. \cite{lai2017deep} used Laplacian pyramid technique into a deep network in shared parameters to accelerate the training speed for SISR. Zhang et al. \cite{zhang2018image} guided a CNN by attention mechanisms to extract salient features for improving the performance and visual effects in image SISR.

Although the mentioned SR methods have obtained excellent effect in SISR, the obtained damaged images are insufficient in the real world, which limits the application of the above SR methods on real cameras. To address problem of small samples, generative adversarial nets (GANs) used generator and discriminator in a game-like manner to obtain good performance on image applications \cite{goodfellow2014generative}. Specifically, the generator can generate new samples, according existing samples \cite{gui2021review}. The discriminator is used to  distinguish the samples from generator \cite{gui2021review}. Due to their strong learning abilities, GANs become popular image super-resolution methods \cite{bell2019blind}. However, there are few studies summarizing these GANs for SISR. In this paper, we conduct a comprehensive  overview of 193 papers to show their performance, pros, cons, complexity, challenges and potential research points, etc. First, we show the effects of GANs for image applications. Second, we present popular architectures for GANs in big and small samples for image applications. Third, we analyze motivations, implementations and differences of GANs based optimization methods and discriminative learning for image super-resolution in terms of supervised, semi-supervised and unsupervised manners. Fifth, we compare these GANs using experimental setting, quantitative analysis (i.e., PSNR, SSIM, complexity and running time) and qualitative analysis. Finally, we report on potential research points and existing challenges of GANs for image super-resolution. The overall architecture of this paper is shown in Fig. 1.

The remainder of this survey is organized as follows. Section \uppercase\expandafter{\romannumeral2} resents the developments of GANs. Section \uppercase\expandafter{\romannumeral3} gives a brief introduction of basic GANs for image processing tasks. Section \uppercase\expandafter{\romannumeral4} focuses on introduction of existing GANs via three ways on SISR. Section \uppercase\expandafter{\romannumeral5} compares performance of mentioned GANs from Section \uppercase\expandafter{\romannumeral4} for SISR. Section \uppercase\expandafter{\romannumeral6} offers potential directions and challenges of GANs in image super-resolution. Section \uppercase\expandafter{\romannumeral7} concludes the overview.

\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig1.pdf}
\label{fig_second_case}}
\caption{Outline of this overview. It mainly consists of basic frameworks, categories (i.e., supervised, semi-supervised and unsupervised GANs), performance comparison, challenges and potential directions.}
\label{fig:1}
\end{figure}

\section{Developments of GANs}
Traditional machine learning methods prefer to use prior knowledge to improve performance of image processing applications \cite{sun2010gradient}. For instance, Sun et al. \cite{sun2010gradient} proposed a gradient profile to restore more detailed information for improving performance of image super-resolution. Although machine learning methods based prior knowledge has fast execution speed, they have some drawbacks. First, they required manual setting parameters to achieve better performance on image tasks. Second, they required complex optimization methods to find optimized parameters. According to mentioned challenges, deep learning methods are developed \cite{bird2022fruit}. Deep learning methods used deep networks, i.e., CNNs to automatically learn features rather than manual setting parameters to obtain effective effects in image processing tasks, i.e., image classification \cite{bird2022fruit}, image inpainting \cite{liu2021pd} and image super-resolution \cite{wang2020deep}. Although these methods are effective big samples, they are limited for image tasks with small samples \cite{goodfellow2014generative}.

To address problems above, GANs are presented in image processing \cite{goodfellow2014generative}. GANs consist of generator network and discriminator network. The generator network is used to generate new samples, according to given samples. The discriminator network is used to determine truth of obtained new samples. When generator and discriminator is balance, a GAN model is finished. The work process of GAN can be shown in Fig. 2, where G and D denote a generator network and discriminator network. To better understand GANs, we introduce several basic GANs as follows.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig2_GAN.pdf}
\label{fig_second_case}}
\caption{Architecture of generative adversarial network (GAN).}
\label{fig:2}
\end{figure}

To obtain more realistic effects, conditional information is fused into a GAN (CGAN) to randomly generate images, which are closer to real images \cite{mirza2014conditional}. CGAN improves GAN to obtain more robust data, which has an important reference value to GANs for computer vision applications. Subsequently, increasing the depth of GAN instead of the original multilayer perceptron in a CNN to improve expressive ability of GAN is developed for complex vision tasks \cite{radford2015unsupervised}. To mine more useful information, the bidirectional generative adversarial network (BiGAN) used dual encoders to collaborate a generator and discriminator to obtain richer information for improving performance in anomaly detection, which is shown in Fig. 3 \cite{donahue2016adversarial}. In Fig. 3, x denotes a feature vector, E is an encoder and y expresses an image from discriminator.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig3_BiGAN.pdf}
\label{fig_second_case}}
\caption{Architecture of bidirectional generative adversarial network (BiGAN).}
\label{fig:3}
\end{figure}

It is known that pretrained operations can be used to accelerate the training speed of CNNs for image recognition \cite{hinterstoisser2018pre}. This idea can be treated as an energy drive. Inspired by that, Zhao et al. proposed an energy-based generative adversarial network (EBGAN) by using a pretraining operation into a discriminator to improve the performance in image recognition \cite{zhao2016energy}. To keep consistency of obtained features with original images, cycle-consistent adversarial network (CycleGAN) relies on a cyclic architecture to achieve an excellent style transfer effect \cite{zhu2017unpaired} as illustrated in Fig. 4.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig4_CycleGAN.pdf}
\label{fig_second_case}}
\caption{Architecture of cycle-consistent adversarial network (CycleGAN).}
\label{fig:4}
\end{figure}

Although pretrained operations are useful for training efficiency of network models, they may suffer from mode collapse. To address this problem, Wasserstein GAN (WGAN) used weight clipping to enhance importance of Lipschitz constraint to improve the stability of training a GAN \cite{arjovsky2017wasserstein}. WGAN used weight clipping to perform well. However, it is easier to cause gradient vanishing or gradient exploding \cite{gulrajani2017improved}. To resolve this issue, WGAN used a gradient penalty (treated as WGAN-GP) to break the limitation of Lipschitz for pursuing good performance in computer vision applications \cite{brock2018large}. To further improve results of image generation, GAN enlarged batch size and used truncation trick as well as BIGGAN can make a tradeoff between variety and fidelity \cite{brock2018large}. To better obtained features of different parts of an image (i.e., freckles and hair), style-based GAN (StyleGAN) uses feature decoupling to control different features and finish style transfer for image generation \cite{karras2019style}. The architecture of StyleGAN and its generator are shown in Fig. 5 and Fig. 6.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig5_StyleGAN.pdf}
\label{fig_second_case}}
\caption{Architecture of StyleGAN.}
\label{fig:5}
\end{figure}

\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig6.pdf}
\label{fig_second_case}}
\caption{The structure of generator in the StyleGAN.}
\label{fig:6}
\end{figure}

In recent years, GANs with good performance have been applied in the fields of image processing, natural language processing (NLP) and video processing. Also, there are other variants based on GANs for multimedia applications, such as Laplacian pyramid of GAN (LAPGAN) \cite{denton2015deep}, coupled GAN (CoupleGAN) \cite{liu2016coupled}, self-attention GAN (SAGAN) \cite{zhang2019self}, loss-sensitive GAN (LSGAN) \cite{qi2020loss}. These methods emphasize how to generate high-quality images through various sampling mechanisms. However, researchers focused applications of GANs from 2019, i.e., FUNIT \cite{liu2019few}, SPADE \cite{park2019semantic} and U-GAT-IT \cite{kim2019u}. Illustrations of more GANs are shown in Table \uppercase\expandafter{\romannumeral1}.
\begin{table*}[htbp!]
\caption{Introduction of many GANs.}
\label{tab:1}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{ccccc}
\hline
Models &Methods &Applications &Key words\\
\hline\\
GAN \cite{goodfellow2014generative} &GAN &Image generation &GAN in a semi-supervised way for image generation &\\
DICCGAN \cite{bird2022fruit} & CGAN & Image classification & Conditional GAN for image classification & \\
PD-GAN \cite{liu2021pd} & GAN     & Image inpainting                     & GAN for image   inpainting and image restoration                           & \\
CGAN \cite{mirza2014conditional}     & GAN     & Image generation                     & GAN in a   supervised way for image generation                             & \\
DCGAN \cite{radford2015unsupervised}    & GAN     & Image generation                     & GAN in an   unsupervised way for image generation                          & \\
BiGAN \cite{donahue2016adversarial}    & GAN     & Image generation                     & GAN with encoder in   an unsupervised way for image generation             & \\
EBGAN \cite{zhao2016energy}     & GAN     & Image generation   and training nets & GAN based energy   for image generation                                    & \\
CycleGAN \cite{zhu2017unpaired} & GAN     & Image generation                     & GAN with   cycle-consistent for image generation                           & \\
WGAN-GP \cite{gulrajani2017improved}   & GAN     & Image generation                     & GAN with gradient   penalty for image generation                           & \\
BIGGAN \cite{brock2018large}    & GAN     & Image   super-resolution             & GAN with big channels   of image super-resolution                          & \\
StyleGAN \cite{karras2019style}  & GAN     & Image generation                     & GAN with stochastic   variation for image generation                       & \\
LAPGAN \cite{denton2015deep}    & CGAN    & Image   super-resolution             & GAN with Laplacian   pyramid for image super-resolution                    & \\
CoupleGAN \cite{liu2016coupled} & GAN     & Image generation                     & GAN for both up-sampling   and image generation                            & \\
SAGAN \cite{zhang2019self}     & GAN     & Image generation                     & Unsupervised GAN with   self-attention for image generation                & \\
FUNIT \cite{liu2019few}    & GAN     & Image translation                    & GAN in an unsupervised   way for image-to-image translation                & \\
SPADE \cite{park2019semantic}    & GAN     & Image generation                     & GAN with spatially-adaptive   normalization for image generation           & \\
U-GAT-IT \cite{kim2019u}  & GAN     & Image translation                    & GAN with attention   in an unsupervised way for image-to-image translation &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\section{Popular GANs for image applications}
According to mentioned illustrations, it is known that variants of GANs based on properties of vision tasks are developed in Section \uppercase\expandafter{\romannumeral2}. To further know GANs, we show different GANs on training data, i.e., big samples and small samples for different high- and low-level computer vision tasks as shown in Fig. 7.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig7.pdf}
\label{fig_second_case}}
\caption{Frame of popular GANs for image applications.}
\label{fig:7}
\end{figure}

\subsection{Popular GANs on big samples for image applications}
\subsubsection{GANs on big samples for image generation}
Good performance of image generation depends on rich samples. Inspired by that, GANs are improved for image generation \cite{gui2021review}. That is, GANs use generator to produce more samples from high-dimensional data to cooperate discriminator for promoting results of image generation. For instance, boundary equilibrium generative adversarial networks (BEGAN) used obtained loss from Wasserstein to match loss of auto-encoder in the discriminator and achieve a balance between a generator and discriminator, which can obtain more texture information than that of common GANs in image generation \cite{berthelot2017began}. To control different parts of a face, StyleGAN decoupled different features to form a feature space for finishing transfer of texture information \cite{karras2019style}. Besides, texture synthesis is another important application of image generation \cite{li2016precomputed}. For instance, Markovian GANs (MGAN) can quickly capture texture date of Markovian patches to achieve function of real-time texture synthesis \cite{li2016precomputed}, where Markovian patches can be obtained Ref. \cite{gui2021review}. Periodic spatial GAN (PSGAN) \cite{bergmann2017learning} is a variant of spatial GAN (SGAN) \cite{jetchev2016texture}, which can learn periodic textures of big datasets and a single image. These methods can be summarized in Table \uppercase\expandafter{\romannumeral2}.
\begin{table}[htbp!]
\caption{GANs on big samples for image generation.}
\label{tab:2}
\centering
\scalebox{0.8}[0.85]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
StyleGAN \cite{karras2019style} & GAN & GAN with stochastic variation for image generation &\\
BEGAN \cite{berthelot2017began}    & GAN     & GAN with upsampling for image generation           &\\
MGAN \cite{li2016precomputed}      & GAN     & GAN with Markovian   for texture synthesis           &\\
PSGAN \cite{bergmann2017learning}     & GAN     & Periodic GAN for   texture synthesis                 &\\
SGAN \cite{jetchev2016texture}      & GAN     & GAN with spatial   tensor for texture synthesis      &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsubsection{GANs on big samples for object detection}
Object detection has wide applications in the industry, i.e., smart transportation \cite{wang2021real} and medical diagnosis \cite{gerdprasert2021object}, etc. However, complex environments have huge challenges for pursuing good performance of object detection methods \cite{zou2019object}. Rich data is important for object detection. Existing methods used a data-driven strategy to collect a large-scale dataset including different object examples under different conditions to obtain an object detector. However, the obtained dataset does not contain all kinds of deformed and occluded objects, which limits effects of object detection methods. To resolve the issue, GANs are used for object detection \cite{ehsani2018segan, li2017perceptual}. Ehsani et al. used segmentation and generation in a GANs from invisible parts in the objects to overcome occluded objects \cite{ehsani2018segan}. To address a challenge of small object detection on low-resolution and noisy representation, a perceptual GAN (Perceptual GAN) reduced differences of small objects and big objects to improve performance in small object detection \cite{li2017perceptual}. That is, its generator converted poor perceived representation from small objects to high-resolution big objects to fool a discriminator, where mentioned big objects are similar to real big objects \cite{li2017perceptual}. To obtain sufficient information of objects, an end-to-end multi-task generative adversarial network (SOD-MTGAN) used a generator to recover detailed information for generating high-quality images for achieving accurate detection \cite{bai2018sod}. Also, a discriminator transferred classification and regression losses in a back-propagated way into a generator \cite{bai2018sod}. Two operations can extract objects from backgrounds to achieve good performance in object detection [59]. More detailed information is shown in Table \uppercase\expandafter{\romannumeral3}.
\begin{table}[htbp!]
\caption{GANs on big samples for object detection.}
\label{tab:3}
\centering
\scalebox{0.7}[0.75]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
SeGAN \cite{ehsani2018segan}           & GAN     & GAN with segmentor   for object detection                     &\\
Perceptual GAN   \cite{li2017perceptual} & GAN     & GAN with super-resolved representation for object detection &\\
SOD-MTGAN \cite{bai2018sod}      & GAN     & Multi-task GAN for   object detection                         &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsection{GANs on small samples for image applications}
\subsubsection{GANs on small samples for image style transfer}
Makeup has important applications in the real world \cite{yuan2022ramt}. To save costs, visual makeup software is developed, leading to image style transfer (i.e., image-to-image) translation becoming a research hotspot in the field of computer vision in recent years \cite{gui2021review}. GANs are good tools for style transfer on small samples, which can be used to establish mappings between given images and object images \cite{gui2021review}. The obtained mappings are strongly related to aligned image pairs \cite{isola2017image}. However, we found that the above mappings do not match  our ideal models in terms of transfer effects \cite{zhu2017unpaired}. Motivated by that, CycleGAN used two pairs of a generator and discriminator in a cycle consistent way to learn two mappings for achieving style transfer \cite{zhu2017unpaired}. CycleGAN had two phases in style transfer. In the first phase, an adversarial loss \cite{park2019semantic} was used to ensure the quality of generated images. In the second phase, a cycle consistency loss  \cite{zhu2017unpaired} was utilized to guarantee that predicted images to fell into the desired domains \cite{chang2018generating}. CycleGAN had the following merits. It does not require paired training examples \cite{chang2018generating}. And it does not require that the input image and the output image have the same low-dimensional embedding space \cite{zhu2017unpaired}. Due to its excellent properties, many variants of CycleGAN have been conducted for many vision tasks, i.e., image style transfer \cite{zhu2017unpaired, chen2021arcyclegan}, object transfiguration \cite{kim2018u} and image enhancement \cite{you2019fundus}, etc. More GANs on small samples for image style transfer can be found in Table \uppercase\expandafter{\romannumeral4}.
\begin{table}[htbp!]
\caption{GANs on small samples for image style transfer.}
\label{tab:4}
\centering
\scalebox{0.55}[0.60]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
RAMT-GAN \cite{yuan2022ramt}  & GAN      & GAN for image   style transfer on makeup &\\
CycleGAN \cite{zhu2017unpaired}  & GAN      & Cycle-consistent   GAN for image-to-image translation &\\
CATVGAN \cite{Xie2020Correlation} & GAN      & Correlation   alignment GAN for image style transfer &\\
ITCGAN \cite{isola2017image}     & CGAN     & CGAN with U-net   for image-to-image translation &\\
ArCycleGAN \cite{chen2021arcyclegan} & GAN      & GAN with attribute   registration for image-to-image translation &\\
URCycleGAN \cite{kim2018u} & CycleGAN & CycleGAN with U-net   for image-to-image translation &\\
ECycleGAN \cite{you2019fundus} & CycleGAN & CycleGAN with convolutional   block attention module (CBAM) for image-to-image translation &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsubsection{GANs on small samples for image inpainting}
Images have played important roles in human–computer interaction in the real world \cite{cowie2001emotion}. However, they may be damaged when they were collected by digital cameras, which has a negative impact on high-level computer vision tasks. Thus, image inpainting had important values in the real world \cite{guillemot2013image}. Due to missing pixels, image inpainting suffered from enormous challenges \cite{elharrouss2020image}. To overcome shortcoming above, GANs are used to generate useful information to repair damaged images based on the surrounding pixels in the damaged images \cite{demir2018patch}. For instance, GAN used a reconstruction loss, two adversarial losses and a semantic parsing loss to guarantee pixel faithfulness and local-global contents consistency for face image inpainting \cite{li2017generative}. Although this method can generate useful information, which may cause boundary artifacts, distorted structures and blurry textures inconsistent with surrounding areas \cite{zhang2022gan, yu2018generative}. To resolve this issue, Zhang et al. embedded prior knowledge into a GAN to generate more detailed information for achieving good performance in image inpainting \cite{zhang2022gan}. Yu et al. exploited a contextual attention mechanism to improve a GAN for obtaining excellent visual effect in image inpainting \cite{yu2018generative}. Typical GANs on small samples for image inpainting is summarized in Table \uppercase\expandafter{\romannumeral5}.
\begin{table}[htbp!]
\caption{GANs on small samples for image inpainting.}
\label{tab:5}
\centering
\scalebox{0.75}[0.8]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
PGGAN \cite{demir2018patch} & GAN      & GAN based patch for image inpainting &\\
DE-GAN \cite{zhang2022gan} & GAN      & GAN with prior knowledge for face inpainting &\\
GFC \cite{li2017generative} & GAN      & GAN with autoencoder for image inpainting &\\
GIICA \cite{yu2018generative} & WGAN     & WGAN with attention model for image inpainting &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\section{GANs for image super-resolutions}
According to mentioned illustrations, it is clear that GANs have many important applications in image processing. Also, image super-resolution is crucial for high-level vision tasks, i.e., medical image diagnosis and weather forecast, etc. Thus, GANs in image super-resolution have important significance in the real world. However, there are few summaries about GANs for image super-resolution. Inspired by that, we show GANs for image super-resolution, according to supervised GANs, semi-supervised GANs and unsupervised GANs for image super-resolution as shown in Fig. 8. Specifically, supervised GANs in image super-resolution include supervised GANs based improved architectures, supervised GANs based prior knowledge, supervised GANs with improved loss functions and supervised GANs based multi-tasks for image super-resolution. Semi-supervised GANs for image super-resolution contain semi-supervised GANs based improved architectures, semi-supervised GANs with improved loss functions and semi-supervised GANs based multi-tasks for image super-resolution.

Unsupervised GANs for image super-resolution consists of unsupervised GANs based improved architectures, unsupervised GANs based prior knowledge, unsupervised GANs with improved loss functions and unsupervised GANs based multi-tasks in image super-resolution. More information of GANs on image super-resolution can be illustrated as follows.

\subsection{Deep CNNs based group convolutions for SISR}
\subsubsection{Supervised GANs based improved architectures for image super-resolution}

GANs in a supervised way to train image super-resolution models are very mainstream. Also, designing GANs via improving network architectures are very novel. Thus, improved GANs in a supervised way for image super-resolution are very popular. That can improve GANs by designing novel discriminator networks, generator networks, attributes of image super-resolution task, complexity and computational costs. For example, Laplacian pyramid of adversarial networks (LAPGAN) fused a cascade of convolutional networks into Laplacian pyramid network in a coarse-to-fine way to obtain high-quality images for assisting image recognition task \cite{denton2015deep}. To overcome the effect of big scales, curvature and highlight compact regions can be used to obtain a local salient map for adapting big scales in image-resolution \cite{mahapatra2017image}. More research on improving discriminators and generators is shown as follows.

In terms of designing novel and discriminators and generators, progressive growing generative adversarial networks (PGGAN or ProGAN) utilized different convolutional layers to progressively enlarge low-resolution images to improve image qualities for image recognition \cite{karras2017progressive}. An enhanced SRGAN (ESRGAN) used residual dense blocks into a generator without batch normalization to mine more detailed information for image super-resolution \cite{wang2018esrgan}. To eliminate effects of checkerboard artifacts and the unpleasing high-frequency, multi-discriminators were proposed for image super-resolution \cite{lee2019multi}. That is, a perspective discriminator was used to overcome checkerboard artifacts and a gradient perspective was utilized to address unpleasing high-frequency question in image super-resolution. To improve the perceptual quality of predicted images, ESRGAN+ fused two adjacent layers in a residual learning way based on residual dense blocks in a generator to enhance memory abilities and added noise in a generator to obtain stochastic variation and obtain more details of high-resolution images \cite{rakotonirina2020esrgan+}.

Restoring detailed information may generate artifacts, which can seriously affect qualities of restored images \cite{zhang2020supervised}. In terms of face image super-resolution, Zhang et al. used a supervised pixel-wise GAN (SPGAN) to obtain higher-quality face images via given low-resolution face images of multiple scale factors to remove artifacts in image super-resolution \cite{zhang2020supervised}. In terms of remote sensing image super-resolution, Gong et al. used enlighten blocks to make a deep network achieve a reliable point and used self-supervised hierarchical perceptual loss to overcome effects of artifacts in remote sensing image super-resolution \cite{gong2021enlighten}. Dharejo et al. used Wavelet Transform (WT) characteristics into a transferred GAN to eliminate artifacts to improve quality of predicted remote sensing images \cite{dharejo2021twist}. Moustafa et al. embedded squeeze-and-excitation blocks and residual blocks into a generator to obtain more high-frequency details \cite{moustafa2021satellite}. Besides, Wasserstein distance is used to enhance the stability of training a remote sensing super-resolution model \cite{moustafa2021satellite}. To address pseudo-textures problem, a saliency analysis is fused with a GAN to obtain a salient map that can be used to distinguish difference between a discriminator and a generator \cite{ma2019sd}.

To obtain more detailed information in image super-resolution, a lot of GANs are developed \cite{ko2021multi}. Ko et al. used Laplacian idea and edge in a GAN to obtain more useful information to improve clarities of predicted face images \cite{ko2021multi}. Using tensor structures in a GAN can facilitate texture information for SR \cite{ding2019tgan}. Using multiple generators in a GAN can obtain more realistic texture details, which was useful to recover high-quality images \cite{zareapoor2019diverse, xing2021deep}. To obtain better visual effects, a gradually GAN used gradual growing factors in a GAN to improve performance in SISR \cite{shamsolmoali2019g}.

To reduce computational costs and memory, Ma et al. used two-stage generator in a supervision way to extract more effective features of cytopathological images, which can reduce the cost of data acquisition and save cost \cite{ma2020pathsrgan}. Cheng et al. designed a generator by multi-scale feature aggregation and a discriminator via a PatchGAN to reduce memory consumption for a GAN on SR \cite{cheng2021mfagan}. Besides, distilling a generator and discriminator can accelerate the training efficiency of a GAN model for SR \cite{cheng2021mfagan}. More supervised GANs for image super-resolution are shown in Table \uppercase\expandafter{\romannumeral6}.
\begin{figure}[!htbp]
\centering
\subfloat{\includegraphics[width=3.5in]{fig8.pdf}
\label{fig_second_case}}
\caption{Frame of GANs for image super-resolution.}
\label{fig:8}
\end{figure}

\begin{table*}[htbp!]
\caption{Supervised GANs for image super-resolution in section \uppercase\expandafter{\romannumeral4} part A.}
\label{tab:6}
\centering
\scalebox{0.87}[1.0]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
LAPGAN \cite{denton2015deep} & CGAN           & CGAN with Laplacian   Pyramid for image super-resolution                                                         &\\
LSMGAN \cite{mahapatra2017image} & CGAN           & CGAN with local   saliency maps for retinal image super-resolution                                               &\\
PGGAN \cite{karras2017progressive} & GAN            & Progressive   growing GAN for image super-resolution                                                             &\\
ESRGAN \cite{wang2018esrgan} & SRGAN \cite{ledig2017photo} & SRGAN with Residual-in-Residual   Dense Block (RRDB) and relativistic discriminator for image super-resolution   &\\
MPDGAN \cite{lee2019multi} & GAN            & GAN with multi-discriminators   for image super-resolution                                                       &\\
ESRGAN+ \cite{rakotonirina2020esrgan+} & ESRGAN         & ESRGAN with Residual-in-Residual   Dense Residual Block (RRDRB) for image super-resolution                       &\\
SPGAN \cite{zhang2020supervised} & GAN            & GAN with identity-based   discriminator for face image super-resolution                                          &\\
MLGE \cite{ko2021multi} & LAPGAN         & LAPGAN with edge   information for face image super-resolution                                                   &\\
SD-GAN \cite{ma2019sd} & GAN            & GAN for remote   sensing image super-resolution                                                                  &\\
PathSRGAN \cite{ma2020pathsrgan} & SRGAN          & SRGAN with RRDB   for cytopathology image super-resolution                                                       &\\
Enlighten-GAN   \cite{gong2021enlighten} & GAN            & GAN with enlighten   block for remote sensing image super-resolution                                             &\\
TWIST-GAN \cite{dharejo2021twist} & GAN            & GAN with wavelet   transform (WT) for remote sensing image super-resolution                                      &\\
SCSE-GAN \cite{moustafa2021satellite} & GAN            & GAN with SCSE   block for image super-resolution                                                                 &\\
MFAGAN \cite{cheng2021mfagan} & GAN            & GAN with multi-scale   feature aggregation net for image super-resolution                                        &\\
TGAN \cite{ding2019tgan} & GAN            & GAN with visual   tracking and attention networks for image super-resolution                                     &\\
DGAN \cite{zareapoor2019diverse} & GAN            & GAN with disentangled   representation learning and anisotropic BRDF reconstruction for image   super-resolution &\\
DMGAN \cite{xing2021deep} & GAN            & GAN with two same   generators for image super-resolution                                                        &\\
G-GANISR \cite{shamsolmoali2019g} & GAN            & GAN with gradual learning for image   super-resolution                                                           &\\
SRGAN \cite{ledig2017photo} & GAN            & GAN with deep ResNet   for image super-resolution                                                                &\\
RaGAN \cite{jolicoeur2018relativistic} & GAN            & GAN with relativistic   discriminator for image super-resolution                                                 &\\
LE-GAN \cite{2021arXiv211108685s} & GAN            & GAN with a latent   encoder for realistic hyperspectral image super-resolution                                   &\\
NCSR \cite{kim2021noise} & GAN            & GAN with a noise   conditional layer for image super-resolution                                                  &\\
Beby-GAN \cite{li2021best} & GAN            & GAN with a   region-aware adversarial learning strategy for image super-resolution                               &\\
MA-GAN \cite{xu2021multi} & GAN            & GAN with pyramidal   convolution for image super-resolution                                                      &\\
CMRI-CGAN \cite{xia2021super} & CGAN           & CGAN with optical   flow component for magnetic resonance image super-resolution &                      \\
D-SRGAN \cite{demiray2021d} & SRGAN          & SRGAN for image   super-resolution &                      \\
LMISR-GAN \cite{ma2021medical} & GAN            & GAN with residual   channel attention block for medical image super-resolution                                   &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\subsubsection{Supervised GANs based prior knowledge for image super-resolution}
It is known that combination of discriminative method and optimization can make a tradeoff between efficiency and performance \cite{zhang2017learning}. Guan et al. used high-resolution image to low-resolution image network and low-resolution image to high-resolution image network with nearest neighbor down-sampling method to learn detailed information and noise prior for image super-resolution \cite{guan2019srdgan}. Chan et al. used rich and diverse priors in a given pretrained to mine latent representative information for generating realistic textures for image super-resolution \cite{chan2021glean}. Liu et al. used a gradient prior into a GAN to suppress the effect of blur kernel estimation for image super-resolution \cite{liu2019infrared}.

\subsubsection{Supervised GANs with improved loss functions for image super-resolution}
Loss function can affect performance and efficiency of a trained SR model. Thus, we analyze the combination of GANs with different loss functions in image super-resolution \cite{zhang2019ranksrgan}. Zhang et al. trained a Ranker to obtain representation of perceptual metrics and used a rank-content loss in a GAN to improve visual effects in image super-resolution \cite{zhang2019ranksrgan}. To eliminate effect of artifacts, Zhu et al. used image quality assessment metric to implement a novel loss function to enhance the stability for image super-resolution \cite{zhu2020gan}. To decrease complexity of GAN model in image super-resolution, Fuoli et al. used a Fourier space supervision loss to recover lost high-frequency information to improve predicted image quality and accelerate training efficiency in SISR \cite{fuoli2021fourier}. To enhance stability of a SR model, using residual blocks and a self-attention layer in a GAN enhances robustness of a trained SR model. Also, combining improved Wasserstein gradient penalty and perceptual Loss enhances stability of a SR model \cite{shahidi2021breast}. To extract accurate features, fusing a measurement loss function into a GAN can obtain more detailed information to obtain clearer images \cite{shahsavari2021proposing}.

\subsubsection{Supervised GANs based multi-tasks for image super-resolution}

Improving image quality is important for high-level vision tasks, i.e., image recognition \cite{talab2019super}. Besides, devices often suffer from effects of multiple factors, i.e., device hardware, camera shakes and shooting distances, which results in collected images are damaged. That may include noise and low-resolution pixels. Thus, addressing the multi-tasks for GANs are very necessary \cite{hu2019rtsrgan}. For instance, Adil et al.
exploited SRGAN and a denoising module to obtain a clear image. Then, they used a network to learn unique representative information for identifying a person \cite{adil2020multi}. In terms of image super-resolution and object detection, Wang et al. used multi-class cyclic super-resolution GAN to restore high-quality images, and used a YOLOv5 detector to finish object detection task \cite{wang2022remote}. Zhang et al. used a fully connected network to implement a generator for obtaining high-definition plate images and a multi-task discriminator is used to enhance super-resolution and recognition tasks \cite{zhang2018joint}.
The use of an adversarial learning was a good tool to simultaneously address text recognition and super-resolution \cite{xu2020srr}.

In terms of complex damaged image restoration, GANs are good choices \cite{li2022multi}. For instance, Li et al. used a multi-scale residual block and an attention mechanism in a GAN to remove noise and restore detailed information in CTA image super-resolution \cite{li2022multi}. Nneji et al. improved a VGG19 to fine-tune two sub-networks with a wavelet technique to simultaneously address COVID-19 image denoising and super-resolution problems \cite{nneji2022fine}. More information
is shown in Table \uppercase\expandafter{\romannumeral7}.
\begin{table}[htbp!]
\caption{Supervised GANs for image super-resolution in section \uppercase\expandafter{\romannumeral4} part A para.2 to section \uppercase\expandafter{\romannumeral4} part A para.4.}
\label{tab:7}
\centering
\scalebox{0.55}[0.60]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words\\
\hline\\
SRDGAN \cite{guan2019srdgan} & GAN    & GAN with GMSR for   image super-resolution                                                      &\\
GLEAN \cite{chan2021glean} & GAN    & GAN with   pre-trained models for image super-resolution                                        &\\
I-SRGAN \cite{liu2019infrared} & GAN    & GAN with infrared prior   knowledge for image super-resolution on infrared image                &\\
RankSRGAN \cite{zhang2019ranksrgan} & SRGAN  & SRGAN with ranker   for image super-resolution                                                  &\\
GMGAN \cite{zhu2020gan} & GAN    & GAN with a novel   quality loss for image super-resolution                                      &\\
FSLSR \cite{fuoli2021fourier} & GAN    & GAN with fourier   space losses for image super-resolution                                      &\\
I-WAGAN \cite{shahidi2021breast} & GAN    & GAN with improved   wasserstein gradient penalty and perceptual loss for image super-resolution &\\
CESR-GAN \cite{shahsavari2021proposing} & GAN    & GAN with a   feature-based measurement loss function for image super-resolution                 &\\
RTSRGAN \cite{hu2019rtsrgan} & SRGAN  & SRGAN for real   time image super-resolution                                                    &\\
MSSRGAN \cite{adil2020multi} & ESRGAN & ESRGAN with denoising module for image super-resolution                                         &\\
RSISRGAN \cite{wang2022remote} & GAN    & GAN for image   super-resolution on RSI                                                         &\\
JPLSRGAN \cite{zhang2018joint} & GAN    & GAN for license plate recognition and   image super-resolution                                  &\\
SRR-GAN \cite{xu2020srr} & GAN    & GAN for image super-resolution   on text images                                                 &\\
MRD-GAN \cite{li2022multi} & GAN    & GAN with attention   mechanism for image super-resolution and denoising                        & \\
MESRGAN+ \cite{nneji2022fine} & ESRGAN & ESRGAN with siamese   network for image super-resolution and denoising.                        &\\
RealESRGAN \cite{wang2021real} & ESRGAN & ESRGAN with pure synthetic   data for blind image super-resolution &\\
SNPE-SRGAN \cite{tampubolon2021snpe} & SRGAN  & SRGAN with SPNE   for image super-resolution                                                    &\\
SOUP-GAN \cite{zhang2022soup} & GAN    & GAN with 3D MRI   for image super-resolution &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsection{Semi-supervised GANs for image super-resolution}
\subsubsection{Semi-supervised GANs based improved architectures for image super-resolution}

For real problems with less data, semi-supervised techniques are developed. For instance, asking patients takes multiple CT scans with additional radiation doses to conduct paired CT images for training SR models in clinical practice is not realistic. Motivated by that, GANs in semi-supervised ways are used for image super-resolution \cite{you2019ct}. For instance, by maintaining the cycle-consistency of Wasserstein distance, a mapping from noisy low-resolution images to high-resolution images was built \cite{you2019ct}. Besides, combining a convolutional neural network, residual learning operations in a GAN can facilitate more detailed information for image super-resolution \cite{you2019ct}. To resolve super-resolution and few labeled samples, Xia et al. used soft multi-labels to implement a semi-supervised super-resolution method for person re-identification \cite{xia2021real}. That is, first, a GAN is used to conduct a SR model. Second, a graph convolutional network
is exploited to construct relationship of local features from a person. Third, some labeled samples are used to train unlabeled samples via a graph convolutional network.

\subsubsection{Semi-supervised GANs with improved loss functions and semi-supervised GANs based multi-tasks for image super-resolution}
The combinations of semi-supervised GANs and loss functions are also effective in image super-resolution \cite{jiang2020novel}. For example, Jiang et al. combined an adversarial loss, a cycle-consistency loss, an identity loss and a joint sparsifying transform loss into a GAN in a semi-supervised way to train a CT image super-resolution model \cite{jiang2020novel}. Although this model made a significantly progress on some evaluation criteria, it was still disturbed by artifacts and noise.

In terms of multi-tasks, Nicolo et al. proposed to use a mixed adversarial Gaussian domain adaptation in a GAN in a semi-supervised way to obtain more useful information for implementing a 3D super-resolution and segmentation \cite{savioli2021joint}. More information of semi-supervised GANs in image super-resolution can be illustrated in Table \uppercase\expandafter{\romannumeral8}.
\begin{table*}[htbp!]
\caption{Semi-supervised GANs for image super-resolution in section \uppercase\expandafter{\romannumeral4} part B.}
\label{tab:8}
\centering
\scalebox{0.9}[0.95]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words \\
\hline\\
GAN-CIRCLE \cite{you2019ct} & GAN & GAN with cycle-consistency   of Wasserstein distance in a semi-supervised way for noisy image   super-resolution            &\\
MSSR \cite{xia2021real} & GAN & GAN with soft   multi-labels in a semi-supervised way for image super-resolution                                            &\\
CTGAN \cite{jiang2020novel} & GAN & GAN with four   losses in a semi-supervised way for image super-resolution                                                  &\\
Gemini-GAN \cite{savioli2021joint} & GAN & GAN with mixed   adversarial Gaussian domain adaptation in a semi-supervised way for 3D   super-resolution and segmentation &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\subsection{Unsupervised GANs for image super-resolution}
Collected images in the real world have less pairs. To address this phenomenon, unsupervised GANs are presented \cite{yuan2018unsupervised}. It can be divided into four types, i.e., improved architectures, prior knowledge, loss functions and multi-tasks in GANs in unsupervised ways for image super-resolution as follows.

\subsubsection{Unsupervised GANs based improved architectures for image super-resolution}
CycleGANs have obtained success in unsupervised ways in image-to-image translation applications \cite{zhu2017unpaired}. Accordingly, the CycleGANs are extended into SISR to address unpair images (i.e., low-resolution and high-resolution) in the datasets in the real world \cite{yuan2018unsupervised}. Yuan et al. used a CycleGAN for blind super-resolution over the following phases \cite{yuan2018unsupervised}. The first phase removed noise from noisy and low-resolution images. The second phase resorted to an up-sampled operation in a pre-trained deep network to enhance the obtained low-resolution images. The third phase used a fine-tune mechanism for a GAN to obtain high-resolution images. To address blind super-resolution, bidirectional structural consistency was used into a GAN in an unsupervised way to train a blind SR model and construct high-quality images \cite{zhao2018unsupervised}. Alternatively, Zhang et al. exploited multiple GANs as basis components to implement an improved CycleGAN for train an unsupervised SR model \cite{zhang2019multiple}. To eliminate checkerboard artifacts, an upsampling module containing a bilinear interpolation and a transposed convolution was used in an unsupervised CycleGAN to improve visual effects of restored images in the real world \cite{kim2020unsupervised}.

There are also other popular methods that use GANs in unsupervised ways for image super-resolution \cite{prajapati2020unsupervised}. To improve the learning ability of a SR model in the real world, it combines an unsupervised learning and a mean opinion score in a GAN to improve perceptual quality in the real-world image super-resolution \cite{prajapati2020unsupervised}. To recover more natural image characteristics, Lugmayr et al. combined unsupervised and supervised ways for blind image super-resolution \cite{lugmayr2019unsupervised}. The first step learned to invert the effects of bicubic down sampling operation in a GAN in an unsupervised way to extract useful information from natural images \cite{lugmayr2019unsupervised}. To generate image pairs in the real world, the second step used a pixel-wise network in a supervised way to obtain high-resolution images \cite{lugmayr2019unsupervised}. To break fixed downscaling kernel, Sefi et al. used KernelGAN \cite{bell2019blind} and Internal-GAN \cite{shocher2018ingan} to obtain an internal distribution of patches in the blind image super-resolution. To accelerate the training speed, a guidance module was used in a GAN to quickly seek a correct mapping from a low-resolution domain to a high-resolution domain in unpaired image super-resolution \cite{lian2019fg}. To improve the accuracy of medical diagnosis, Song et al. used dual GANs in a self-supervised way to mine high dimensional information for PET image super-resolution \cite{mahapatra2017image}. Besides, other SR methods can have an important reference value for unsupervised GANs with for image super-resolution. For example, Wang et al. used an unsupervised method to translate real low-resolution images to real low-resolution images \cite{wang2020transformation}. Chen et al. resorted to a supervised super-resolution method to convert obtained real low-resolution images into real high-resolution images \cite{chen2020unsupervised}. More information of mentioned unsupervised GANs for image super-resolution can be shown in Table \uppercase\expandafter{\romannumeral9} as follows.
\begin{table*}[htbp!]
\caption{Unsupervised GANs based improved architectures for image super-resolution.}
\label{tab:9}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words \\
\hline\\
CinCGAN \cite{yuan2018unsupervised} & GAN          & Unsupervised GAN   for image super-resolution                                                 &\\
DNSR \cite{zhao2018unsupervised} & GAN          & Unsupervised GAN   with bidirectional structural consistency for blind image super-resolution &\\
MCinCGAN \cite{zhang2019multiple} & CycleGAN     & Unsupervised GAN   for image super-resolution                                                 &\\
RWSR-CycleGAN   \cite{kim2020unsupervised} & CycleGAN     & Unsupervised GAN   for image super-resolution                                                 &\\
USISResNet \cite{prajapati2020unsupervised} & GAN          & Unsupervised GAN   with USISResNet for image super-resolution                                 &\\
ULRWSR \cite{lugmayr2019unsupervised} & GAN          & Unsupervised GAN   with pixel wise supervision for image super-resolution                     &\\
KernelGAN \cite{bell2019blind} & Internal-GAN & Unsupervised GAN   for blind image super-resolution                                           &\\
InGAN \cite{shocher2018ingan} & GAN          & Unsupervised GAN   for image super-resolution                                                 &\\
FG-SRGAN \cite{lian2019fg} & SRGAN        & Unsupervised GAN   with a guided block for image super-resolution                             &\\
PETSRGAN \cite{mahapatra2017image} & GAN          & Unsupervised GAN   with a self-supervised   way for PET image super-resolution                &\\
TrGAN \cite{wang2020transformation} & GAN          & Unsupervised GAN   for image synthesis and super-resolution                                   &\\
CycleSR \cite{chen2020unsupervised} & GAN          & Unsupervised GAN with   an indirect supervised path for image super-resolution                &\\
UGAN-Circle \cite{guha2021unsupervised} & GAN-Circle   & Unsupervised   GAN-Circle for image super-resolution on CT images &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\subsubsection{Unsupervised GANs based prior knowledge for image super-resolution}

Combining unsupervised GANs and prior knowledge in unsupervised GANs can better address unpair image super-resolution \cite{lin2018deep}. Lin et al. combined data error, a regular term and an adversarial loss to guarantee consistency of local-global content and pixel faithfulness in a GAN in an unsupervised way to train an image super-resolution model \cite{lin2018deep}. To better support medical diagnosis, Das et al. combined adversarial learning in a GAN, cycle consistency and prior knowledge, i.e., identity mapping prior to facilitate more useful information i.e., spatial correlation, color and texture information for obtaining cleaner high-quality images \cite{das2020unsupervised}. In terms of remoting sensing super-resolution, a random noise is used in a GAN to reconstruct satellite images \cite{wang2021enhanced}. Then, authors conducted image prior by transforming the reference image into a latent space \cite{wang2021enhanced}. Finally, they updated the noise and latent space to transfer obtained structure information and texture information for improving resolution of remote sensing images \cite{wang2021enhanced}.

\subsubsection{Unsupervised GANs with improved loss functions for image super-resolution}

Combining loss functions and GANs in an unsupervised way is useful for training image super-resolution models in the real world \cite{zhang2020unsupervised}. For instance, Zhang et al. used a novel loss function based image quality assessment in a GAN to obtain accurate texture information and more visual effects \cite{zhang2020unsupervised}. Besides, an encoder-decoder architecture is embedded in this GAN to mine more structure information for pursuing high-quality images of a generator from this GAN \cite{zhang2020unsupervised}. Han et al. depended on SAGAN and L1 loss in a GAN in an unsupervised manner to act multi-sequence structural MRI for detecting braining anomalies \cite{han2021madgan}. Also, Zhang et al. fused a content loss into a GAN in an unsupervised manner to improve SR results of hyperspectral images  \cite{zhang2021degradation}. Unsupervised GANs based prior knowledge and improved loss functions for image super-resolution can be summarized in Table \uppercase\expandafter{\romannumeral10}.
\begin{table}[htbp!]
\caption{Unsupervised GANs based prior knowledge and improved loss functions for image super-resolution.}
\label{tab:10}
\centering
\scalebox{0.56}[0.63]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words \\
\hline\\
DULGAN \cite{lin2018deep} & GAN   & GAN with prior for   image super-resolution                                               &\\
USROCTGAN \cite{das2020unsupervised} & GAN   & GAN with cycle   consistency and identity mapping priors for image super-resolution       &\\
EIPGAN \cite{wang2021enhanced} & GAN   & GAN with remote   sensing image prior for image super-resolution                          &\\
URSGAN \cite{zhang2020unsupervised} & GAN   & GAN with prior   based image quality assessment for remote sensing image super-resolution &\\
MADGAN \cite{han2021madgan} & SAGAN & SAGAN with L1 loss   for medical image super-resolution                                   &\\
DLGAN \cite{zhang2021degradation} & GAN   & GAN with a content   loss for hyperspectral image super-resolution &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsubsection{Unsupervised GANs based multi-tasks for image super-resolution}

Unsupervised GANs are good tools to address multi-tasks, i.e., noisy low-resolution image super-resolution. For instance, Prajapati et al. transferred a variational auto-encoder and the idea of quality assessment in a GAN to deal with image denoising and SR tasks \cite{prajapati2021unsupervised}. Cui et al. relied on low-pass-filter loss and weighted MR images in a GAN in an unsupervised GAN to mine texture information for removing noise and recovering resolution of MRI images \cite{cui2022unsupervised}. Cai et al. presented a pipeline that optimizes a periodic implicit GAN to obtain neural radiance fields for image synthesis and image super-resolution based on 3D \cite{cai2022pix2nerf}. More unsupervised GANs based multi-tasks for image super-resolution can be presented in Table \uppercase\expandafter{\romannumeral11}.
\begin{table*}[htbp!]
\caption{Unsupervised GANs based multi-tasks for image super-resolution.}
\label{tab:11}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{cccc}
\hline
Models &Methods &Key words \\
\hline\\
VAEGAN \cite{prajapati2021unsupervised} & GAN & GAN with variational   auto-encoder and quality assessment idea for image super-resolution and   denoising &\\
ASLGAN \cite{cui2022unsupervised} & GAN & GAN with   low-pass-filter loss and weighted MR images for MRI image super-resolution   and denoising      &\\
Pix2NeRF \cite{cai2022pix2nerf} & GAN & Optimizing a   periodic implicit GAN for 3D-aware image synthesis and super-resolution                     &\\
Pi-GAN \cite{chan2021pi} & GAN & GAN with periodic   activation functions for 3D-aware image synthesis and image super-resolution &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\section{Comparing performance of GANs for image super-resolution}
To make readers conveniently know GANs in image super-resolution, we compare
super-resolution performance of these GANs from datasets and experimental settings to quantitative and qualitative analysis in this section. More information can be shown as follows.

\subsection{Datasets}
Mentioned GANs can be divided into three kinds: supervised methods, semi-supervised methods and unsupervised methods for image super-resolution, which make datasets have three categories, training datasets and test datasets for supervised methods, semi-supervised methods and unsupervised methods. These datasets can be summarized as follows.

(1)	Supervised GANs for image-resolution

Training datasets: CIFAR10 \cite{krizhevsky2009learning}, STL \cite{wang2016unsupervised}, LSUN \cite{zhang2015large}, ImageNet \cite{russakovsky2015imagenet}, Celeb A \cite{liu2015deep}, DIV2K \cite{agustsson2017ntire}, Flickr2K \cite{wang2019flickr1024}, OST \cite{wang2018recovering}, CAT \cite{zhang2008cat} Market-1501 \cite{zheng2015scalable}, Duke MTMC-reID \cite{ristani2016performance, zheng2017unlabeled}, GeoEye-1 satellite dataset \cite{ma2019sd}, Whole slide images (WSIs) \cite{ma2020pathsrgan}, MNIST \cite{lecun1998gradient} and PASCAL2 \cite{everingham2010pascal}.

Test datasets: CIFAR10 \cite{krizhevsky2009learning}, STL \cite{wang2016unsupervised}, LSUN \cite{zhang2015large}, Set5 \cite{bevilacqua2012low}, Set14 \cite{zeyde2010single}, BSD100 \cite{martin2001database}, CELEBA \cite{liu2015deep}, OST300 \cite{zhang2015large}, CAT \cite{zhang2008cat}, PIRM datasets \cite{blau20182018}, Market-1501 \cite{zheng2015scalable}, GeoEye-1 satellite dataset \cite{ma2019sd}, WSIs \cite{ma2020pathsrgan} MNIST \cite{lecun1998gradient} and PASCAL2 \cite{everingham2010pascal}.

(2)	Semi-supervised GANs for image-resolution

Training datasets: Market-1501 \cite{zheng2015scalable}, Tibia Dataset \cite{chen2018quantitative}, Abdominal Dataset \cite{mccollough2020low}, CUHK03 \cite{li2014deepreid}, MSMT17 \cite{wei2018person}, LUNA \cite{setio2017validation}, Data Science Bowl 2017 (DSB) \cite{kuan2017deep}, UKDHP \cite{woodbridge2013mridb}, SG \cite{woodbridge2013mridb} and UKBB \cite{bycroft2018uk}.

Test datasets: Tibia Dataset \cite{chen2018quantitative}, Abdominal Dataset \cite{mccollough2020low}, CUHK03 \cite{li2014deepreid}, Widerface \cite{yang2016wider}, LUNA \cite{setio2017validation}, DSB \cite{kuan2017deep}, SG \cite{woodbridge2013mridb} and UKBB \cite{bycroft2018uk}.

(3)	Unsupervised GANs for image-resolution

Training datasets: CIFAR10 \cite{krizhevsky2009learning}, ImageNet \cite{russakovsky2015imagenet}, DIV2K \cite{agustsson2017ntire}, DIV2K random kernel (DIV2KRK) \cite{agustsson2017ntire}, Flickr2K \cite{wang2019flickr1024}, Widerface \cite{yang2016wider}, NTIRE 2020 Real World SR challenge \cite{lugmayr2020ntire}, KADID-10K \cite{lin2019kadid}, DPED \cite{ignatov2017dslr}, DF2K \cite{wang2018esrgan}, NTIRE' 2018 Blind-SR challenge \cite{timofte2018ntire}, LS3D-W \cite{bulat2017far}, CELEBA-HQ \cite{karras2017progressive}, LSUN-BEDROOM \cite{yu2015lsun}, ILSVRC2012 \cite{russakovsky2015imagenet, nguyen2017plug}, NTIRE 2020 \cite{lugmayr2020ntire}, 91-images \cite{yang2010image}, Berkeley segmentation \cite{martin2001database}, BSDS500 \cite{yang2016object}, Training datasets of USROCTGAN \cite{fang2013fast, fang2012sparsity}, SD-OCT dataset \cite{fang2012sparsity}, UC Merced dataset \cite{yang2010bag}, NWPU-RESIS45 \cite{cheng2017remote} and WHU-RS19 \cite{dai2010satellite}.

Test datasets: CIFAR10 \cite{krizhevsky2009learning}, ImageNet \cite{russakovsky2015imagenet}, Set5 \cite{bevilacqua2012low}, Set14 \cite{zeyde2010single}, BSD100 \cite{martin2001database}, DIV2K \cite{agustsson2017ntire}, DIV2KRK \cite{agustsson2017ntire}, Urban100 \cite{huang2015single}, Widerface \cite{yang2016wider}, NTIRE 2020 \cite{lugmayr2020ntire}, NTIRE 2020 Real-world SR Challenge \cite{lugmayr2020ntire}, NTIRE 2020 Real World SR challenge validation dataset \cite{lugmayr2020ntire}, DPED \cite{ignatov2017dslr}, CELEBA-HQ \cite{karras2017progressive}, LSUN-BEDROOM \cite{yu2015lsun}, Test datasets of USROCTGAN \cite{fang2013fast, fang2012sparsity}, and Test datasets of USRGAN \cite{zhang2020unsupervised}.

These mentioned datasets about GANs for image super-resolution can be shown in Table \uppercase\expandafter{\romannumeral12}. To make readers easier understand datasets of different methods via different GANs for different training ways on image super-resolutions, we conduct Table 13 to show their detailed information.
\begin{table*}[htbp!]
\caption{Datasets (i.e., training datasets and test datasets) of GANs for image super-resolution.}
\label{tab:12}
\centering
\scalebox{0.75}[0.7]{
\begin{tabular}{cccll}
\hline
Training ways &Training datasets &Test datasets\\
\hline\\
Supervised ways & \begin{tabular}[c]{@{}l@{}}
CIFAR10 \cite{krizhevsky2009learning}, \\    \\ STL \cite{wang2016unsupervised}, \\    \\ LSUN \cite{zhang2015large}, \\    \\ ImageNet \cite{russakovsky2015imagenet}, \\    \\ Celeb A \cite{liu2015deep},\\    \\ DIV2K \cite{agustsson2017ntire},\\    \\ Flickr2K \cite{wang2019flickr1024},\\    \\ OST \cite{wang2018recovering},\\    \\ CAT \cite{zhang2008cat},\\    \\ Market-1501 \cite{zheng2015scalable},   \\    \\ Duke MTMC-reID   \cite{ristani2016performance, zheng2017unlabeled},\\    \\ GeoEye-1 satellite   dataset \cite{ma2019sd},\\    \\ Whole slide images   (WSIs) \cite{ma2020pathsrgan},\\    \\ MNIST \cite{lecun1998gradient}, \\    \\ PASCAL2  \cite{everingham2010pascal},\\    \\ Set5 \cite{bevilacqua2012low}, \\    \\ Set14 \cite{zeyde2010single}, \\    \\ BSD100 \cite{martin2001database}, \\    \\ Urban100 \cite{huang2015single}.
\end{tabular}
& \begin{tabular}[c]{@{}l@{}}
CIFAR10 \cite{krizhevsky2009learning}, \\    \\ STL \cite{wang2016unsupervised}, \\    \\ LSUN \cite{zhang2015large},\\    \\ Set5 \cite{bevilacqua2012low}, \\    \\ Set14 \cite{zeyde2010single}, \\    \\ BSD100 \cite{martin2001database},\\    \\ CELEBA \cite{liu2015deep}, \\    \\ CAT \cite{zhang2008cat},\\    \\ OST300 \cite{wang2018recovering}, \\    \\ the PIRM datasets   \cite{blau20182018},\\    \\ Market-1501 \cite{zheng2015scalable},\\    \\ GeoEye-1 satellite   dataset \cite{ma2019sd},\\    \\ WSIs \cite{ma2020pathsrgan},\\    \\ MNIST \cite{lecun1998gradient}, \\    \\ PASCAL2 \cite{everingham2010pascal}.
\end{tabular} &\\ \hline\\

Semi-supervised ways & \begin{tabular}[c]{@{}l@{}}
Tibia Dataset   \cite{chen2018quantitative}, \\    \\ Abdominal Dataset   \cite{mccollough2020low},\\    \\ Market-1501 \cite{zheng2015scalable},   \\    \\ CUHK03 \cite{li2014deepreid}, \\    \\ MSMT17 \cite{wei2018person},\\    \\ LUNA \cite{setio2017validation}, \\    \\ Data Science Bowl   2017 (DSB) \cite{kuan2017deep},\\    \\ UKDHP \cite{woodbridge2013mridb}, \\    \\ SG \cite{woodbridge2013mridb}, \\    \\ UKBB \cite{bycroft2018uk}.
\end{tabular}
& \begin{tabular}[c]{@{}l@{}}
Tibia Dataset   \cite{chen2018quantitative}, \\    \\ Abdominal Dataset   \cite{mccollough2020low},\\    \\ Market-1501 \cite{zheng2015scalable},   \\    \\ CUHK03 \cite{li2014deepreid}\\    \\ LUNA \cite{setio2017validation}, \\    \\ DSB \cite{kuan2017deep},\\    \\ SG \cite{woodbridge2013mridb}, \\    \\ UKBB \cite{bycroft2018uk}.
\end{tabular} &\\ \hline\\

Unsupervised ways & \begin{tabular}[c]{@{}l@{}}
DIV2K \cite{agustsson2017ntire},\\    \\ Flickr2K \cite{wang2019flickr1024}, \\    \\ Widerface \cite{yang2016wider},\\    \\ NTIRE-2020 Real-world   SR Challenge validation dataset \cite{lugmayr2020ntire}, \\    \\ KADID-10K \cite{lin2019kadid},\\    \\ DPED \cite{ignatov2017dslr}, \\    \\ DF2K \cite{wang2018esrgan}, \\    \\ NTIRE'2018   Blind-SR challenge \cite{timofte2018ntire}, \\    \\ DIV2K random   kernel (DIV2KRK) \cite{agustsson2017ntire},\\    \\ LS3D-W \cite{bulat2017far},\\    \\ CIFAR10 \cite{krizhevsky2009learning}, \\    \\ ImageNet \cite{russakovsky2015imagenet}, \\    \\ CELEBA-HQ \cite{karras2017progressive}, \\    \\ LSUN-BEDROOM \cite{yu2015lsun},   \\    \\ NTIRE 2020 \cite{lugmayr2020ntire},\\    \\ 91-images \cite{yang2010image}, \\    \\ Berkeley   segmentation \cite{martin2001database}, BSDS500 \cite{yang2016object},\\    \\ Training datasets   of USROCTGAN \cite{fang2013fast, fang2012sparsity}, \\    \\ SD-OCT dataset   \cite{fang2012sparsity},\\    \\ UC Merced dataset   \cite{yang2010bag}, \\    \\ NWPU-RESIS45   \cite{cheng2017remote}, \\    \\ WHU-RS19 \cite{dai2010satellite}.
\end{tabular}
& \begin{tabular}[c]{@{}l@{}}
DIV2K \cite{agustsson2017ntire},\\    \\ Set5 \cite{bevilacqua2012low}, \\    \\ Set14 \cite{bevilacqua2012low}, \\    \\ Urban100 \cite{huang2015single}, \\    \\ BSD100 \cite{martin2001database}, \\    \\ NTIRE 2020 Real   World SR challenge \cite{lugmayr2020ntire},\\    \\ NTIRE-2020 Real-world   SR Challenge validation dataset \cite{lugmayr2020ntire},\\    \\ DPED \cite{ignatov2017dslr}, \\    \\ DIV2KRK \cite{agustsson2017ntire},\\    \\ Widerface \cite{yang2016wider},\\    \\ CIFAR10 \cite{krizhevsky2009learning}, \\    \\ ImageNet \cite{russakovsky2015imagenet}, \\    \\ CELEBA-HQ \cite{karras2017progressive}, \\    \\ LSUN-BEDROOM \cite{yu2015lsun},\\    \\ NTIRE 2020 \cite{lugmayr2020ntire},\\    \\ Tests datasets of   USROCTGAN \cite{fang2013fast},\\    \\ Test datasets of USRGAN   \cite{zhang2020unsupervised}.
\end{tabular} &\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\begin{table*}[htbp!]
\caption{Different GANs on image super-resolutions for different training ways.}
\label{tab:13}
\centering
\scalebox{0.52}[0.65]{
\begin{tabular}{cccc}
\hline
Training ways &Methods &Training datasets &Test datasets\\
\hline
\multirow{12}{*}{Supervised ways} \\& LAPGAN \cite{denton2015deep} & CIFAR10 \cite{krizhevsky2009learning}, STL   \cite{wang2016unsupervised}, LSUN \cite{zhang2015large} & CIFAR10 \cite{krizhevsky2009learning}, STL   \cite{wang2016unsupervised}, LSUN \cite{zhang2015large} \\
                                        \\& SRGAN \cite{ledig2017photo} & ImageNet \cite{russakovsky2015imagenet} & Set5 \cite{bevilacqua2012low}, Set14   \cite{bevilacqua2012low} and BSD100 \cite{martin2001database} \\
                                        \\& PGGAN \cite{karras2017progressive} & CIFAR10 \cite{krizhevsky2009learning},   Celeb A \cite{liu2015deep} & CELEBA \cite{liu2015deep}, LSUN   \cite{zhang2015large}, CIFAR10 \cite{krizhevsky2009learning} \\
                                        \\& ESRGAN \cite{wang2018esrgan} & DIV2K \cite{agustsson2017ntire},   Flickr2K \cite{wang2019flickr1024}, OST \cite{wang2018recovering} & Set5 \cite{bevilacqua2012low}, Set14   \cite{zeyde2010single}, BSD100 \cite{martin2001database}, Urban100 \cite{huang2015single} \\
                                        \\& RaGAN \cite{jolicoeur2018relativistic} & CIFAR10 \cite{krizhevsky2009learning}, CAT   \cite{zhang2008cat} & CAT \cite{zhang2008cat} \\
                                        \\& ESRGAN+ \cite{rakotonirina2020esrgan+} & DIV2K \cite{agustsson2017ntire} & BSD100 \cite{martin2001database},   Urban100 \cite{huang2015single}, OST300 \cite{wang2018recovering}, Set5 \cite{bevilacqua2012low}, Set14   \cite{bevilacqua2012low} and the PIRM datasets \cite{blau20182018} \\
                                        \\& SPGAN \cite{zhang2020supervised} & Market-1501 \cite{zheng2015scalable},   Duke MTMC-reID \cite{ristani2016performance, zheng2017unlabeled} & Market-1501 \cite{zheng2015scalable} \\
                                       \\& SD-GAN \cite{ma2019sd} & GeoEye-1 satellite   dataset \cite{ma2019sd} & GeoEye-1 satellite   dataset \cite{ma2019sd} \\
                                        \\& PathSRGAN \cite{ma2020pathsrgan} & Whole slide images   (WSIs) \cite{ma2020pathsrgan} & WSIs \cite{ma2020pathsrgan} \\
                                        \\& TGAN \cite{ding2019tgan} & MNIST \cite{lecun1998gradient},   PASCAL2 \cite{everingham2010pascal}, CIFAR10 \cite{krizhevsky2009learning} & MNIST \cite{lecun1998gradient},   PASCAL2 \cite{everingham2010pascal}, CIFAR10 \cite{krizhevsky2009learning} \\
                                        \\& DGAN \cite{zareapoor2019diverse} & DIV2K \cite{agustsson2017ntire} & Set5 \cite{bevilacqua2012low}, Set14   \cite{zeyde2010single}, CIFAR-10 \cite{krizhevsky2009learning}, BSD100 \cite{martin2001database} \\
                                        \\& G-GANISR \cite{shamsolmoali2019g}         & Set5 \cite{bevilacqua2012low}, Set14   \cite{zeyde2010single}, BSD100 \cite{martin2001database}, and Urban100 \cite{huang2015single} & Set5 \cite{bevilacqua2012low}, Set14   \cite{zeyde2010single}, and Urban100 \cite{huang2015single} \\ \\\hline
\multirow{4}{*}{Semi-supervised ways} \\& GAN-CIRCLE \cite{you2019ct} & Tibia Dataset   \cite{chen2018quantitative}, Abdominal Dataset \cite{mccollough2020low} & Tibia Dataset   \cite{chen2018quantitative}, Abdominal Dataset \cite{mccollough2020low} \\
                                        \\& MSSR \cite{xia2021real} & Market-1501 \cite{zheng2015scalable},   CUHK03 \cite{li2014deepreid}, MSMT17 \cite{wei2018person} & Market-1501 \cite{zheng2015scalable},   CUHK03 \cite{li2014deepreid} \\
                                        \\& CTGAN \cite{jiang2020novel}            & LUNA \cite{setio2017validation}, Data   Science Bowl 2017 (DSB) \cite{kuan2017deep} & LUNA \cite{setio2017validation}, DSB \cite{kuan2017deep} \\
                                        \\& Gemini-GAN \cite{savioli2021joint}       & UKDHP \cite{woodbridge2013mridb}, SG   \cite{woodbridge2013mridb}, UKBB \cite{bycroft2018uk} & SG \cite{woodbridge2013mridb}, UKBB   \cite{bycroft2018uk} \\\\\hline
\multirow{14}{*}{Unsupervised ways} \\& CinCGAN \cite{yuan2018unsupervised} & DIV2K \cite{agustsson2017ntire} & DIV2K \cite{agustsson2017ntire} \\
                                        \\& DNSR \cite{zhao2018unsupervised}            & DIV2K \cite{agustsson2017ntire},   Flickr2K \cite{wang2019flickr1024}, Widerface \cite{yang2016wider} & Set5 \cite{bevilacqua2012low}, Set14   \cite{zeyde2010single}, Urban100 \cite{huang2015single}, BSD100 \cite{martin2001database}, DIV2K \cite{agustsson2017ntire}                                   \\
                                        \\& MCinCGAN \cite{zhang2019multiple}        & DIV2K \cite{agustsson2017ntire}                                                                                & DIV2K \cite{agustsson2017ntire} \\
                                        \\& RWSR-CycleGAN   \cite{kim2020unsupervised} & NTIRE 2020 Real   World SR challenge \cite{lugmayr2020ntire}                                                                                & NTIRE 2020 Real   World SR challenge \cite{lugmayr2020ntire}                                                                             \\
                                        \\& USISResNet \cite{prajapati2020unsupervised}      & NTIRE-2020 Real-world   SR Challenge validation dataset \cite{lugmayr2020ntire}, DIV2K \cite{agustsson2017ntire}, Flickr2k \cite{wang2019flickr1024}, KADID-10K   \cite{lin2019kadid} & NTIRE-2020 Real-world   SR Challenge validation dataset \cite{lugmayr2020ntire}                                                          \\
                                        \\& ULRWSR \cite{lugmayr2019unsupervised}          & DPED \cite{ignatov2017dslr}, DF2K   \cite{wang2018esrgan}, DIV2K \cite{agustsson2017ntire}, Flickr2K \cite{wang2019flickr1024} & DPED \cite{ignatov2017dslr}, DIV2K   \cite{agustsson2017ntire} \\
                                        \\& KernelGAN \cite{bell2019blind}       & NTIRE'2018   Blind-SR challenge {[}153{]}, DIV2K random kernel (DIV2KRK) \cite{agustsson2017ntire} & DIV2KRK \cite{agustsson2017ntire} \\
                                        \\& FG-SRGAN \cite{lian2019fg}        & LS3D-W \cite{bulat2017far} & Widerface \cite{yang2016wider} \\
                                        \\& TrGAN \cite{wang2020transformation}           & CIFAR10 \cite{krizhevsky2009learning}, ImageNet   \cite{russakovsky2015imagenet}, CELEBA-HQ \cite{karras2017progressive}, LSUN-BEDROOM \cite{yu2015lsun} & CIFAR10 \cite{krizhevsky2009learning}, ImageNet   \cite{russakovsky2015imagenet}, CELEBA-HQ \cite{karras2017progressive}, LSUN-BEDROOM \cite{yu2015lsun} \\
                                        \\& PDCGAN \cite{miyato2018cgans} & ImageNet \cite{russakovsky2015imagenet},   ILSVRC2012 \cite{russakovsky2015imagenet, nguyen2017plug}, CIFAR10 \cite{krizhevsky2009learning}, CIFAR-100 \cite{krizhevsky2009learning} & ImageNet \cite{russakovsky2015imagenet} \\
                                        \\& CycleSR \cite{chen2020unsupervised}         & DIV2K \cite{agustsson2017ntire}, NTIRE   2020 \cite{lugmayr2020ntire} & DIV2K \cite{agustsson2017ntire}, NTIRE   2020 \cite{lugmayr2020ntire} \\
                                        \\& DULGAN \cite{lin2018deep}          & 91-images \cite{yang2010image},   Berkeley segmentation \cite{martin2001database}, BSDS500 \cite{yang2016object} & Set5 \cite{bevilacqua2012low}, Set14 \cite{zeyde2010single} \\
                                        \\& USROCTGAN \cite{das2020unsupervised}       & Training datasets of USROCTGAN \cite{fang2013fast, fang2012sparsity}, SD-OCT dataset \cite{fang2012sparsity} & Test datasets of   USROCTGAN \cite{fang2013fast} \\
                                        \\& URSGAN \cite{zhang2020unsupervised} & UC Merced dataset   \cite{yang2010bag}, NWPU-RESIS45 \cite{cheng2017remote}, WHU-RS19 \cite{dai2010satellite} & Test datasets of   USRGAN \cite{zhang2020unsupervised} \\\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\subsection{Environment configurations}
In this section, we compare the differences of environment configurations between different GANs via different training ways (i.e., supervised, semi-supervised and unsupervised ways) for image super-resolution, which contain batch size, scaling factors, deep learning framework, learning rate and iteration. That can make readers easier to conduct experiments with GANs for image super-resolution. Their information can be listed as shown in Table \uppercase\expandafter{\romannumeral14} as follows.

\begin{table*}[htbp!]
\caption{Environment configurations of different GANs for image super-resolution.}
\label{tab:14}
\centering
\scalebox{0.9}[1.05]{
\begin{tabular}{ccccccc}
\hline
Training ways &Methods &Batchsize &Scaling factors &Framework &Learning rates &Iteration\\
\hline
\multirow{10}{*}{Supervised ways}       & LAPGAN \cite{denton2015deep}           & 16     & ×2                 & Pytorch \cite{paszke2019pytorch}                            & 0.02            & -             \\
                                        & ESRGAN \cite{wang2018esrgan} & 16     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 2×10-4,   10-4  & 2×105,   300K \\
                                        & ESRGAN+ \cite{rakotonirina2020esrgan+} & 16     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 10-4            & 300K          \\
                                        & SPGAN \cite{zhang2020supervised} & 16     & ×2, ×4, ×8 and ×16 & TensorFlow \cite{goldsborough2016tour}                         & 10-3            & -             \\
                                        & SD-GAN \cite{ma2019sd} & 9      & ×3                 & TensorFlow \cite{goldsborough2016tour}                         & 10-3            & 2,000         \\
                                        & TGAN \cite{ding2019tgan} & 32     & ×2                 & TensorFlow \cite{goldsborough2016tour}                         & 10-4            & 500           \\
                                        & DGAN \cite{zareapoor2019diverse}             & -      & ×4, ×6 and ×8      & TensorFlow \cite{goldsborough2016tour}                         & 0.125           & 25,000        \\
                                        & G-GANISR \cite{shamsolmoali2019g}         & -      & ×4, ×6 and ×8      & PyTorch \cite{paszke2019pytorch} and   TensorFlow \cite{goldsborough2016tour} & 10-4            & 8×105         \\
                                        & GMGAN \cite{zhu2020gan}           & 16     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 2×10-4          & 105           \\ \hline
\multirow{3}{*}{Semi-supervised   ways} & MSSR \cite{xia2021real} & 64     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 0.001, 0.01     & -             \\
                                        & PSSR \cite{maeda2020unpaired}             & 16     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 10-4            & 3×105         \\
                                        & CTGAN \cite{jiang2020novel}            & 16     & ×4                 & TensorFlow \cite{goldsborough2016tour}                         & 10-4            & 6×104         \\ \hline
\multirow{12}{*}{Unsupervised ways}     & CinCGAN \cite{yuan2018unsupervised}         & 16     & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 10-4            & 400,000       \\
                                        & DNSR \cite{zhao2018unsupervised}            & 16     & ×2, ×4             & TensorFlow \cite{goldsborough2016tour}                         & 10-4            & 106           \\
                                        & MCinCGAN \cite{zhang2019multiple}        & 1      & ×2, ×4, ×8         & -                                            & 2×10-4          & 500,000       \\
                                        & RWSR-CycleGAN   \cite{kim2020unsupervised} & 16     & ×4                 & TensorFlow \cite{goldsborough2016tour}                         & 10-4            & 300,000       \\
                                        & USISResNet \cite{prajapati2020unsupervised} & 32     & ×4                 & PyTorch \cite{paszke2019pytorch} & 10-4            & 120,000       \\
                                        & ULRWSR \cite{lugmayr2019unsupervised} & - & ×4 & TensorFlow \cite{goldsborough2016tour} & 10-4 ,   2×10-4 & 50,000        \\
                                        & KernelGAN \cite{bell2019blind} & -      & ×2, ×4             & PyTorch \cite{paszke2019pytorch}                            & 2×10-4          & 3,000         \\
                                        & InGAN \cite{shocher2018ingan} & 1      & ×2                 & -                                            & 10-4            & 20,000        \\
                                        & PETSRGAN \cite{mahapatra2017image} & 10, 20 & ×4                 & PyTorch \cite{paszke2019pytorch}                            & 2×10-4          & -             \\
                                        & PDCGAN \cite{miyato2018cgans} & -      & ×4                 & TensorFlow \cite{goldsborough2016tour}                         & 2×10-4          & 850,000       \\
                                        & USROCTGAN \cite{fang2013fast} & 64     & ×4                 & TensorFlow \cite{goldsborough2016tour}                         & 2×10-4          & 68,000        \\
                                        & URSGAN \cite{zhang2020unsupervised} & 64     & ×2, ×4             & PyTorch \cite{paszke2019pytorch}                            & 5×10-4          & -\\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table*}

\subsection{Experimental results}
To make readers understand the performance of different GANs on image super-resolution, we use quantitative analysis and qualitative analysis to evaluate super-resolution effects of these GANs. Quantitative analysis is PSNR and SSIM of different methods via three training ways on different datasets for image super-resolution, running time and complexities of different GANs on image super-resolution. Qualitative analysis is used to evaluate qualities of recovered images.

\subsubsection{Quantitative analysis of different GANs for image super-resolution}
We use SRGAN \cite{ledig2017photo}, PGGAN \cite{karras2017progressive}, ESRGAN \cite{wang2018esrgan}, ESRGAN+ \cite{rakotonirina2020esrgan+}, DGAN \cite{zareapoor2019diverse}, G-GANISR \cite{shamsolmoali2019g}, GMGAN \cite{zhu2020gan}, SRPGAN \cite{ledig2017photo}, DNSR \cite{zhao2018unsupervised}, DULGAN \cite{lin2018deep}, CinCGAN \cite{yuan2018unsupervised},MCinCGAN \cite{zhang2019multiple}, USISResNet\cite{prajapati2020unsupervised}, ULRWSR \cite{lugmayr2019unsupervised}, KernelGAN \cite{bell2019blind}  and  CycleSR \cite{chen2020unsupervised} in one training way from supervised, semi-supervised and unsupervised ways on a public dataset from Set14 \cite{zeyde2010single},  BSD100  \cite{martin2001database} and DIV2K \cite{agustsson2017ntire} to test performance for different scales in image super-resolution as shown in Table \uppercase\expandafter{\romannumeral15}. For instance, ESRGAN \cite{wang2018esrgan} outperforms SRGAN \cite{ledig2017photo} in terms of PSNR and SSIM in a supervised ways on Set 14 for ×2, which shows that ESRGAN has obtained better super-resolution performance for ×2. More information of these GANs can be shown in Table \uppercase\expandafter{\romannumeral15}.
\begin{table}[htbp!]
\caption{PSNR and SSIM of different GANs via different training ways on Set14, BSD100 and DIV2K for image super-resolution.}
\label{tab:15}
\centering
\scalebox{0.75}[0.75]{
\begin{tabular}{cccccc}
\hline
Training ways &Methods &Datasets &Scale &PSNR &SSIM\\
\hline
\multirow{32}{*}{Supervised ways}   & \multirow{2}{*}{SRGAN \cite{ledig2017photo} } & \multirow{16}{*}{Set14 \cite{zeyde2010single}}  & ×2 & 32.14  & 0.8860 \\
                                    &                                      &                                    & ×4 & 26.02  & 0.7379 \\
                                    & \multirow{2}{*}{PGGAN \cite{karras2017progressive}}      &                                    & ×6 & 29.54  & 0.8301 \\
                                    &                                      &                                    & ×8 & 28.14  & 0.8094 \\
                                    & \multirow{2}{*}{ESRGAN \cite{wang2018esrgan}}     &                                    & ×2 & 33.62  & 0.9150 \\
                                    &                                      &                                    & ×4 & 30.50  & 0.7620 \\
                                    & ESRGAN+ \cite{rakotonirina2020esrgan+}                     &                                    & ×4 & 19.79  & -      \\
                                    & \multirow{3}{*}{DGAN \cite{zareapoor2019diverse}}       &                                    & ×4 & 31.62  & 0.9166 \\
                                    &                                      &                                    & ×6 & 28.62  & 0.9003 \\
                                    &                                      &                                    & ×8 & 26.85  & 0.8911 \\
                                    & \multirow{3}{*}{G-GANISR \cite{shamsolmoali2019g}}   &                                    & ×4 & 29.67  & -      \\
                                    &                                      &                                    & ×6 & 30.56  & 0.8881 \\
                                    &                                      &                                    & ×8 & 28.07  & 0.8803 \\
                                    & GMGAN \cite{zhu2020gan}                      &                                    & ×4 & 26.37  & 0.7055 \\
                                    & \multirow{2}{*}{SRPGAN \cite{ledig2017photo}}    &                                    & ×6 & 26.19  & 0.8187 \\
                                    &                                      &                                    & ×8 & 29.17  & 0.8733 \\\cline{2-6}
                                    & \multirow{2}{*}{SRGAN \cite{ledig2017photo}}      & \multirow{13}{*}{BSD100 \cite{martin2001database}} & ×2 & 31.89  & 0.8760 \\
                                    &                                      &                                    & ×4 & 25.16  & 0.6688 \\
                                    & \multirow{2}{*}{ESRGAN \cite{wang2018esrgan}}     &                                    & ×2 & 31.99  & 0.8870 \\
                                    &                                      &                                    & ×4 & 27.69  & 0.7120 \\
                                    & \multirow{3}{*}{DGAN \cite{zareapoor2019diverse}}       &                                    & ×4 & 31.53  & 0.9105 \\
                                    &                                      &                                    & ×6 & 29.62  & 0.8937 \\
                                    &                                      &                                    & ×8 & 27.85  & 0.8811 \\
                                    & \multirow{3}{*}{G-GANISR \cite{shamsolmoali2019g}}   &                                    & ×4 & 28.12  & -      \\
                                    &                                      &                                    & ×6 & 31.23  & 0.9273 \\
                                    &                                      &                                    & ×8 & 29.18  & 0.9065 \\
                                    & GMGAN \cite{zhu2020gan}                      &                                    & ×4 & 25.46  & 0.6592 \\
                                    & \multirow{2}{*}{SRPGAN \cite{ledig2017photo}}    & & ×6 & 27.48  & 0.8652 \\
                                    &                                      &                                    & ×8 & 23.18  & 0.8504 \\\cline{2-6}
                                    & \multirow{2}{*}{SRGAN \cite{ledig2017photo}}      & \multirow{3}{*}{DIV2K \cite{agustsson2017ntire}}   & ×2 & 25.08  & 0.7007 \\
                                    &                                      &                                    & ×4 & 28.09  & 0.8210 \\
                                    & ESRGAN \cite{wang2018esrgan}                      &                                    & ×4 & 28.68  & 0.8530 \\ \hline
Semi-supervised   ways              & PSSR \cite{maeda2020unpaired}                        & DIV2K \cite{agustsson2017ntire}                    & ×4 & 21.32  & 0.5541 \\\hline
\multirow{17}{*}{Unsupervised ways} & \multirow{2}{*}{DNSR \cite{zhao2018unsupervised}}      & \multirow{3}{*}{Set14 \cite{zeyde2010single}}   & ×2 & 33.83  & 0.9220 \\
                                    &                                      &                                    & ×4 & 31.76  & 0.8910 \\
                                    & DULGAN \cite{lin2018deep}                     &                                    & ×4 & 27.39  & 0.7412 \\\cline{2-6}
                                    & \multirow{2}{*}{DNSR \cite{zhao2018unsupervised}}      & \multirow{2}{*}{BSD100 \cite{martin2001database}}  & ×2 & 32.24  & 0.9010 \\
                                    &                                      &                                    & ×4 & 25.69  & 0.7880 \\\cline{2-6}
                                    & \multirow{3}{*}{CinCGAN \cite{zhang2019multiple}}   & \multirow{12}{*}{DIV2K \cite{agustsson2017ntire}}  & ×2 & 25.61  & 0.6957 \\
                                    &                                      &                                    & ×4 & 25.32  & 0.6705 \\
                                    &                                      &                                    & ×8 & 24.58  & 0.6581 \\
                                    & DNSR \cite{zhao2018unsupervised}                       &                                    & ×4 & 28.87  & 0.8650 \\
                                    & \multirow{3}{*}{MCinCGAN \cite{zhang2019multiple}}  &                                    & ×2 & 26.15  & 0.7020 \\
                                    &                                      &                                    & ×4 & 25.51  & 0.6878 \\
                                    &                                      &                                    & ×8 & 24.79  & 0.6618 \\
                                    & USISResNet \cite{prajapati2020unsupervised}                 &                                    & ×4 & 21.22  & 0.5760 \\
                                    & ULRWSR \cite{lugmayr2019unsupervised}                     &                                    & ×4 & 23.30  & 0.6200 \\
                                    & \multirow{2}{*}{KernelGAN \cite{bell2019blind}} &                                    & ×2 & 30.363 & 0.8669 \\
                                    & & & ×4 & 26.810 & 0.7316 \\
                                    & CycleSR \cite{chen2020unsupervised} & & ×4 & 23.807 & 0.5930 \\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

Running time and complexity are important indexes to evaluate performance of image super-resolution techniques in the real devices \cite{tian2020deep}. According to that, we conduct experiments of four GANs (i.e., ESRGAN \cite{wang2018esrgan}, PathSRGAN \cite{ma2020pathsrgan}, RankSRGAN \cite{zhang2019ranksrgan} and KernelGAN \cite{bell2019blind}) on two low-resolution images with sizes  and  for ×4 to test running time and compute parameters of different GANs. The conducted experiments have the following experimental environments. They can run on Ubuntu of 20.04.1, CPU of AMD EPYC ROME 7502P with 32 cores and Memory of 128G via PyTorch of 1.10.1 \cite{paszke2019pytorch}. Besides, they depend on a NVIDIA GeFore RTX 3090 with cuda of 11.1 and cuDNN of. 8.0.4. In Table \uppercase\expandafter{\romannumeral16}, we can see that ESRGAN \cite{wang2018esrgan} has slower speed than that of PathSRGAN for ×4 on image super-resolution. However, it uses less parameters than that of PathSRGAN for ×4 on image super-resolution. Thus, ESRGAN is competitive with PathSRGAN for image super-resolution. More information of different GANs for image super-resolution in terms of running time and parameters can be shown in Table \uppercase\expandafter{\romannumeral16}.
\begin{table}[htbp!]
\caption{Running time and parameters of different GANs for ×4.}
\label{tab:16}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{ccc}
\hline
Methods &Testing time (s) &Parameters\\
\hline
ESRGAN \cite{wang2018esrgan}     & 9.7607 & 1.670×107 \\
PathSRGAN \cite{ma2020pathsrgan}  & 9.1608 & 2.433×107 \\
RankSRGAN \cite{zhang2019ranksrgan}  & 7.3818 & 1.554×106 \\
KernelGAN \cite{bell2019blind} & 251.00 & 1.816×105 \\
\hline
\end{tabular}}
\label{tab:booktabs}
\end{table}

\subsubsection{Qualitative analysis of different GANs for image super-resolution}
To test visual effects of different GANs for image super-resolution, we choose Bicubic, ESRGAN \cite{wang2018esrgan}, RankSRGAN \cite{zhang2019ranksrgan}, KernelGAN \cite{bell2019blind} and PathSRGAN \cite{ma2020pathsrgan} to conduct experiments to obtain high-quality images for ×4. To further observe these images, we choose an area of predicted images from these GANs to amplify it as an observation area. Observation area is clearer, corresponding method has good superior SR performance. For example, ESRGAN \cite{wang2018esrgan} is clearer than that of PathSRGAN \cite{ma2020pathsrgan} on an image from the BSDS100 in Table \uppercase\expandafter{\romannumeral17} and Set14 in Table  \uppercase\expandafter{\romannumeral18} for ×4, which show that the ESRGAN is more effective in image super-resolution.

\begin{table}[htbp!]
\caption{Visual images of different GANs on an image of BSDS100 for ×4: (a) original image, (b)
Bicubic, (c) ESRGAN, (d) RankSRGAN, (e) KernelGAN, and (f) PathSRGAN.}
\label{tab:17}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{ccc}
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{119082_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{bicubic_119082_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{ESRGAN_119082_zoom.pdf}}
\end{minipage} \\
(a) & (b) & (c) \\
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{RankSRGAN_119082_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{KernelGAN_119082_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{PathSRGAN_119082_zoom.pdf}}
\end{minipage} \\
(d) & (e) & (f) \\
\end{tabular}}
\label{tab:booktabs}
\end{table}

\begin{table}[htbp!]
\caption{Visual images of different GANs on an image of Set14 for ×4: (a) original image, (b)
Bicubic, (c) ESRGAN, (d) RankSRGAN, (e) KernelGAN, and (f) PathSRGAN.}
\label{tab:18}
\centering
\scalebox{1.0}[1.0]{
\begin{tabular}{ccc}
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{flowers_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{bicubic_flowers_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{ESRGAN_flowers_zoom.pdf}}
\end{minipage} \\
(a) & (b) & (c) \\
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{RankSRGAN_flowers_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{KernelGAN_flowers_zoom.pdf}}
\end{minipage} &
\begin{minipage}[b]{0.3\columnwidth}
\raisebox{-.5\height}{\includegraphics[width=\linewidth]{PathSRGAN_flowers_zoom.pdf}}
\end{minipage} \\
(d) & (e) & (f) \\
\end{tabular}}
\label{tab:booktabs}
\end{table}

\section{Challenges and directions of GANs for image super-resolution}
Variations of GANs have achieved excellent performance in image super-resolution. Accordingly, we provide an overview of GANs for image super-resolution to offer a guide for readers to understand these methods. In this section, we analyze challenges of current GANs for image super-resolution and give corresponding solutions to facilitate the development of GANs for image super-resolution.

Although GANs perform well in image super-resolution, they suffer from the following challenges.

1) Unstable training. Due to the confrontation between generator and discriminator, GANs are unstable in the training process.

2) Large computational resources and high memory consumption. A GAN is composed of a generator and discriminator, which may increase computational costs and memory consumption. This may lead to a higher demand on digital devices.

3) High-quality images without references. Most of existing GANs relied on paired high-quality images and low-resolution images to train image super-resolution models, which may be limited by digital devices in the real world.

4) Complex image super-resolution. Most of GANs can deal with a single task, i.e., image super-resolution and synthetic noisy image super-resolution, etc. However, collected images by digital cameras in the real world suffer from drawbacks, i.e., low-resolution and dark-lighting images, complex noisy and low-resolution images. Besides, digital cameras have higher requirement on the combination of image low-resolution and image recognition. Thus, existing GANs for image super-resolution cannot effectively repair low-resolution images of mentioned conditions.

5) Metrics of GANs for image super-resolution. Most of existing GANs used PSNR and SSIM to test super-resolution performance of GANs. However, PSNR and SSIM cannot fully measure restored images. Thus, finding effective metrics is very essential about GANs for image super-resolution.

To address these problems, some potential research points about GANs for image super-resolution are stated below.

1) Enhancing a generator and discriminator extracts salient features to enhance     stabilities of GANs on image super-resolution. For example, using attention mechanism (i.e., Transformer \cite{han2021transformer}), residual learning operations, concatenation operations act a generator and discriminator to extract more effective features to enhance stabilities for  accelerating GAN models in image super-resolution.

2) Designing lightweight GANs for image super-resolution. Reducing convolutional kernels, group convolutions, the combination of prior and shallow network architectures can decrease the complexities of GANs for image super-resolution.

3) Using self-supervised methods can obtain high-quality reference images.

4) Combining attributes of different low-level tasks, decomposing complex low-level tasks into a single low-level task via different stages in different GANs repairs complex low-resolution images, which can help high-level vision tasks.

5) Using image quality assessment techniques as metrics evaluates quality of precited images from different GANs.

\section{Conclusion}
In this paper, we analyze and summarize GANs for image super-resolution. First, we introduce developments of GANs. Then, we present GANs in big samples and small samples for image applications, which can make readers easier GANs. Next, we give differences of GANs based optimization methods and discriminative learning for image super-resolution in terms of supervised, semi-supervised and unsupervised manners. Subsequently, we compare the performance of these popular GANs on public datasets via quantitative and qualitative analysis in SISR. Finally, we highlight challenges of GANs and potential research points on SISR.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv