Integrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis

This paper describes our proposed model for the Chinese Grammatical Error Diagnosis (CGED) task in NLPTEA2020. The goal of CGED is to use natural language processing techniques to automatically diagnose Chinese grammatical errors in sentences. To this end, we design and implement a CGED model named BERT with Score-feature Gates Error Diagnoser (BSGED), which is based on the BERT model, Bidirectional Long Short-Term Memory (BiLSTM) and conditional random field (CRF). In order to address the problem of losing partial-order relationships when embedding continuous feature items as with previous works, we propose a gating mechanism for integrating continuous feature items, which effectively retains the partial-order relationships between feature items. We perform LSTM processing on the encoding result of the BERT model, and further extract the sequence features. In the final test-set evaluation, we obtained the highest F1 score at the detection level and are among the top 3 F1 scores at the identification level.

Introduction
Recently, with the continuous development of China, more and more people have begun to learn Chinese as their second language. Due to the many complexities of Chinese, such as the differences in how tenses are formed in Chinese and English, many learners mistakenly write many Chinese sentences with grammatical errors when they first learn Chinese. Therefore, it is necessary to develop a CGED system, which can not only improve the learning efficiency of Chinese learners, but also serve many downstream tasks based on Chinese corpora. Compared with English grammatical error diagnosis, Chinese grammatical error correction has received limited interest in the research community. English grammar error detection models began being developed as early as the 1980s, such as the early Writer's Workbench system (Macdonald NH, 1983) for detecting punctuation errors and style errors. Later, a series of tasks for English grammatical error detection and correction were proposed, such as CoNLL-2013(Ng et al., 2013 and CoNLL-2014(Ng et al., 2014. With the release of the CGED task in the NLPTEA workshop in recent years, grammar diagnosis models for Chinese have also begun to be developed. The goal of the CGED task is to use natural language processing techniques to diagnose grammatical errors in Chinese sentences written by learners who use Chinese as a second language. The CGED task allows researchers to exchange experiences and ultimately promote the development of this shared task. It defines four types of Chinese grammatical errors, which are: redundant words (denoted as a capital "R"), missing words ("M"), word selection errors ("S"), and word ordering errors ("W"). The system developed for this task needs to identify the type and location of the errors in the input sentence.
Most recent solutions to the CGED shared task convert the problem into a sequence labeling problem and use a BiLSTM-CRF-based architecture as a basic framework to train the model. However, in previous work, feature engineering for the input sequence has become more and more complex. In addition, for some score-based features which exhibit partial-order relationships, such as the commonly used PMI Score features, previous works usually learn their embedding matrix after discretizing the scores. Through this process, the partial-order relationships between items will be lost, and the dimensionality of the feature embedding matrix will gradually increase as the granularity of the score discretization becomes finer, increasing the number of parameters needed to be trained. In response to the above problems, we design and implement BERT with Score-feature Gates Error Diagnoser (BSGED), and integrate score-based features through the use of a gating mechanism, which not only greatly reduces the workload of feature engineering, but also retains the original partial-order relationships for score-based features. Experiments verify that the BSGED model achieves excellent results with less feature engineering.
In summary, our contributions are as follows: • We propose a novel model BSGED for the CGED task, which achieves better results with fewer prior features and greatly reduces the workload of feature engineering. • We propose a gating mechanism for integrating score-based features, which not only preserves the partial-order relationships between feature items, but also greatly reduces the amount of model training parameters. • Through ablation experiments, we verify the effectiveness of adding a BiLSTM layer to further improve the model's ability to capture long-term dependencies of input sequences.

Related Work
Grammatical error diagnosis models appeared as early as the 1980s. Early grammatical error diagnosis models used rule-based methods to check and correct grammatical errors (Naber D, 2003). However, because the design of matching rules requires rich linguistic knowledge, it has become more and more difficult as well as time-consuming to design rules for such models.
In order to deal with more complex error types, a series of grammatical error detection and correction models based on machine translation technology have been proposed. Brockett et al. (2006) proposed a model that uses Statistical Machine Tranalation (SMT) techniques to detect and correct grammatical errors, which deal with mass/count noun confusions by translating the incorrect phrases as a whole. Felice et al. (2014) proposed a model for grammatical error diagnosis which combines rule-based and SMT systems in a pipeline. The model first uses rules to detect errors and generate candidates. After the candidates are roughly screened by the n-gram language model, they are sent to the SMT model for further screening. In the end, candidates will be further selected through language models and filtering rules. In order to solve the CGED2018 shared task, Hu et al. (2018) proposed a sequence-to-sequence network to model the problem, and used a semi-supervised method to generate pseudo-grammatical error data for training the model. Models based on machine translation require a large-scale training corpus to train the model. Inspired by the powerful capabilities of Neural Machine Translation (NMT) in grammatical error diagnosis, Zheng et al. (2016) regarded CGED as a sequence labeling problem, and used the powerful feature learning ability of an LSTM network to model the input sequence, and achieved better results. Yang et al. (2017) incorporated more grammatical features into the model based on the BiLSTM-CRF framework. Based on the LSTM-CRF error detection model, Li et al. (2018) combined three error correction models: a rule-based model, an NMT GEC model, and an SMT GEC model. The three GEC models aid the BiLSTM-CRF model in marking possible error locations during the detection phase. Fu et al. (2018) designed a model that incorporates richer features and added a template matcher and probability fusion mechanism.

Baseline Model
Similar to most previous models for CGED shared tasks, we treat the CGED problem as a sequence labeling problem, and use BiLSTM-CRF as the basic framework of BSGED. Specifically, for a given input sequence , which consists of a character sequence [ 1 , 2 , … , ], BSGED will output an equal-length sequence , which is composed of a label sequence [ 1 , 2 , … , ] composition. We adopt the BIO marking strategy, that is, for characters without grammatical errors, we mark them as 'O', and for a subsequence of grammatical errors, such as word selection errors, the initial characters will be marked as 'B-S', The remaining single characters will be marked as 'I-S'.
Inspired by previous work, we use a BiLSTM network as the RNN unit to obtain the input character encoding sequence. The BiLSTM network has a strong ability to capture long-term dependencies of the input sequence. CRFs are widely used in a large number of natural language processing tasks, especially sequence-annotation tasks. With the addition of a CRF, the BiLSTM-CRF model can predict the input sequence more accurately. For example, the BiLSTM-CRF model can avoid incorrect sequence predictions beginning with "I-X". In terms of feature selection, we select some simple features, such as the POS tag sequence, POS Score, and PMI Score. Different from previous work, BSGED adopts the BERT model as the character encoder of the input sequence, and uses a novel fusion mechanism to incorporate score-based features. The model details are introduced in the next section. The framework of the base model adopted by BSGED is shown in Figure 1.

BERT-Encoder and Gating mechanism
Unlike previous models based on the BiLSTM-CRF architecture, BSGED does not utilize overly complex feature engineering, but uses the novel BERT model to obtain a token embedding representation of the input sequence. As a pre-trained language model, BERT has been successfully applied to many natural language understanding tasks, such as Chinese spelling error correction (Zhang et al. 2020). Due to its powerful semantic extraction capabilities, we utilize BERT as a semantic feature extractor, converting characters into vector representations. In order to preserve the long-term dependencies on the input sequence better, BSGED takes the final layer output of the BERT model as part of the BiLSTM input, instead of concatenating it with the output results of the other features through the BiLSTM network. Experiments verify that this operation can further improve the overall performance of BSGED.
1 https://github.com/HIT-SCIR/ltp We use prior knowledge to calculate the POS features of the input sequence and the PMI features between adjacent words. Specifically, we first use the LTP word segmentation tool 1 to perform word segmentation processing on the input sequence, and then perform part-of-speech tagging on the segmented sequence. This step also makes use of the LTP library. We also integrate location information into the POS tags. For example, for a Chinese sequence 1 2 3 1 2 1 , the segmented sequence should be 1 2 3 -1 2 -1 . Assuming the POS information of word A, word B, and word C are , , respectively, then the result of POS labeling should be --. For the score-based features, we use the news corpus provided by SogouCS 2 as a large corpus to obtain prior-knowledge statistics. Similar to the approach of Yang et al. (2017), for the POS Score feature, we first count the discrete probability distribution of the POS feature of each word, and use the probability value as its POS Score. Similarly, we count the co-occurrence frequency between every two words on the same large corpus, and use the normalized co-occurrence frequency score as the PMI Score of two adjacent words. It should be noted that we also merge the character position information in the vocabulary into these feature items.
We propose a novel fusion mechanism for scorebased features. For continuous score features, traditional models usually discretize them first, and then embed the discretized score into a low-dimensional space. However, this embedding method will lose the partial-order relationships between the scores. In addition, the size of the feature space will change with the discretization granularity and the original value range of the score, and the model will have more parameters to be trained. Our approach differs in that we retain the continuity of 2 https://www.sogou.com/labs/resource/cs.php score features and train a matrix ∈ ℝ 2 * for each score-based feature, where is the preset embedding matrix dimension. For the -th character, the final score embedding vector is as follows: Where is the final embedding representation and is the position information of the character, = 0 for a "B-Word", and = 1 for an "I-Word". At this point, the role of score-based features is similar to an input gate (Hochreiter and Schmidhuber, 1997). This strategy not only preserves the partial-order relationship of score features, but also greatly reduces the size of the parameter matrix. The composition structure of the features for the input sequence is shown in Figure  2.

Ensemble mechanism
Following our experiments, we find that for different initialization parameters, the prediction results of the model are highly variable. This observation is consistent with that of Yang et al. (2017). In order to further improve the performance, we train multiple single models and use an ensemble mechanism to fuse them together. We adopt a simple and effective voting mechanism as our ensemble method, which improves the precision of the model while preserving the recall value.
In our final version, we use a total of four parameter groups, and we select 4 random factors for each group, so we finally merge 16 single models.

Post-Processing
The ensemble mechanism may produce conflicting prediction results. To solve this problem, we perform post-processing operations on the results of the ensemble model. We adopt some rule-base schemes, which integrate prior knowledge simply and effectively. The main processing methods are as follows: First, in cases when some single models predict a sentence to be correct and other single models predict it to be incorrect, the conflict is resolved by retaining the prediction 'incorrect'. The 'correct' label is only output when all models predict the sentence as 'correct'.
Second, we resolve 'incorrect' predictions with overlaps, such as when the following two predictions are output for sentence : Where is the starting position of the prediction, is the ending position, and is the predicted error type. When Equation 3 is established, BSGED believes that the two prediction results overlap.
When overlapping occurs, the model uses the segmentation boundary of the original sentence to filter. Suppose that the word segmentation boundary of sentence is = [ 1 , 2 , … , , … ] , that is, [ −1 : ] represents a word of the sentence. The model will retain the prediction result of < , , > which is more suitable for .

Data Preparation
We use all the data from the CGED2015-CGED2018 training and test sets, as well as the training data from CGED2020. More specifically, our training data consists of the following parts: all data from the CGED2015-2016 training set and test set, all data from the CGED2017-2018 training set, 50% of the CGED2017-2018 test set, and 20% of the CGED2020 training set. The validation set consists of 50% of the CGED2017-2018 test set and 80% of the CGED2020 training set.
Since the training set of CGED2020 has the same data as the test set from CGED2017-2018, in order to prevent data leakage, we de-duplicate the training set. Following de-duplication, the training set contains 43925 samples, and the validation set contains 3843 samples.

BERT Selection
Since richer model initialization parameters result in more diverse predictions, thereby further improving the recall rate of the model after ensembling, we therefore choose two different BERT pretraining parameters to initialize our model.
In addition to using the BERT-Base-Chinese version released by Google (Devlin et al., 2018), we also use another version of Chinese BERT. In order to further promote the research and development of Chinese information processing, the HFL team released the Chinese pre-training model BERT-wwm (Cui et al., 2019), which uses a Whole Word Masking technique, as well as models closely related to this technology: BERT-wwm-ext.
BERT-wwm is trained on Chinese Wikipedia (including simplified and traditional characters) and LTP is used to perform word segmentation before masking is carried out on all Chinese characters that make up the same word. Similar to other BERT-based models, it has 12 layers, 768 hidden size, and 12 self-attention heads.

Validation Results
We select the model parameters through the validation set results, which mainly include the selection of the filtering threshold during model integration. Since BSGED uses a total of 16 single models for integration, we first simply set the max filtering threshold to 10, and explore the performance of the model after integration within this range. The performance of the model on the validation set is shown in Table 3 and Figure 3. It should be noted that when selecting parameters, we only paid attention to the performance of the model at the position level.
It can be seen that as the filtering threshold increases, as does the precision, and the resulting predictions are more reliable; and as the filtering threshold decreases, the recall rate of the results will increase, enabling the model to be able to cover more actual errors. A low threshold will encourage retention of a large number of over-detection errors, while a high threshold will filter out partially correct results during post-processing. When the filter threshold is in the middle of the range, the model can achieve a higher F1 value.
Finally, we select three fusion models by selecting the parameter group with the highest precision   Table 2: The performance of the three submissions on the official evaluation test data set. The scores in bold represent the best scores we obtained among all the participating teams. The "Best Team" row records the best scores among all participating teams for each task-specific evaluating metric.   rate, the parameter group with the highest recall rate, and the parameter group with the highest F1 value. The results on the validation set are shown in Table 1.

Testing Results
The final version of BSGED obtained the top F1 score at the detection level and was among the top 3 F1 scores at the identification level on the test set released by CGED2020. In addition, BSGED obtained the highest precision rate and recall rate among all error diagnosis evaluation levels except the precision rate at the Detection Level. The specific results are shown in Table 2.

Ablation Experiment
In order to evaluate the novel components of our approach, we conduct two sets of ablation experiments.
The first set of ablation experiments focuses on the gating mechanism. We use 7 parameter groups from the 16 parameter groups from our original experiments. 7 single models use the traditional discretized embedding method for score-based features, and 7 single models used the novel gating approach we propose. The final comparison results are shown in Table 4. The results show that the control group that uses the gating mechanism achieves higher F1 values at each level of error detection; the performance improvements at the detection level, identification level, and position level are 0.0173, 0.0371 and 0.0348 respectively, demonstrating the effectiveness of the gating mechanism.
The second set of ablation experiments shows the performance improvement brought about by the addition of the BiLSTM layer compared to the BERT-only model. Through connecting the encoded output of the BERT model to the BiLSTM layer, the model can further improve its ability to capture the long-term dependencies of the input sequence. We conduct an experimental comparison of the model with and without the connected BiLSTM layer. For this experiment, 4 single models use a BERT-CRF architecture, and 4 single models connect the BERT output to a BiLSTM (BSGED). The two single model groups use the same parameter settings. The comparison result is shown in Table 5. As can be seen, the control group with the addition of the BiLSTM achieves F1 value improvements of 0.0159, 0.0316, and 0.0265 at the detection level, identification level, and position level, demonstrating the effectiveness of the BiLSTM layer.

Case Study
We found that different optimizations enable BSGED to solve different types of errors better. Among them, the gating mechanism directly retains the partial-order relationships of the original score-based features, so it has an improved ability for recognizing errors at character-or word-level. Some examples are shown in Table 6. For example, in the first sentence in Table 6 Table 5: The influence of the BiLSTM layer in the BSGED on the model's results on the validation set. The value is the average of 4 models "much love") should be identified as being incorrect, with the correct phrase being "最爱" (meaning "favorite"). Similarly, "沿" (meaning "along") and "没" (meaning "no") are words formed with similar strokes. In the second example sentence, " 沿" should be replaced with "没", because the PMI score of "沿有" is extremely low. In the third sentence, "速度减速" is a word-level error, and the correct expression should be "速度减慢".
The addition of the BiLSTM layer enables the model to better capture the long-term dependencies of the input sequence so that the model has stronger processing capabilities for error samples that rely on semantic understanding and long-term dependencies. Some examples are shown in Table 7. For example, in the first sentence, "在…去" should be identified as an incorrect expression in Chinese, with the correct structure being "到…去". Identifying this error that requires judging long-term dependencies of the text. Finally, the phrase "首歌" in the second example is a common collocation, but in the example, through the semantic understanding of the last clause, "首" should be identified as an R type error.

Conclusion and Future Work
This paper describes our novel BSGED model for the CGED2020 shared task, which uses only a few and simple features, greatly reducing the workload of feature engineering for CGED; a gating mechanism is also proposed to retain the original partialorder relationships between score-based features and at the same time reduce the amount of model training parameters. In addition, we connect the sequence encoding result of the BERT model to the BiLSTM layer, which improves the BSGED model's ability to capture long-term dependencies of the input sequence. BSGED achieves the best F1 score at the detection level and the third highest F1 score at the identification level.
In the future, we intend to use the MLM model to build a model that includes grammatical error correction, and apply the natural language generation capabilities of the pre-trained language model to the task of correcting Chinese grammatical errors. In addition, we will also integrate more explicit grammatical rules, which will also greatly help the improvement of model performance. 12, 13, S (减速) 世界里有很多挑选新能源。最主要的生态学的能源有是：太阳能，风能，潮 汐能，还有地热能。


SEECR: Secure Energy Efficient and Cooperative Routing Protocol for Underwater Wireless Sensor Networks

Underwater wireless sensor networks (UWSNs) is an emerging technology for exploration of underwater resources. Security plays an important role in the UWSNs environment because the environment of UWSNs is prone to different security attacks. This research proposes SEECR: Secure Energy Efficient and Cooperative Routing protocol for UWSNs. SEECR comprised of energy efficient and strong defense mechanism for combatting attacks in underwater environment. SEECR exploits cooperative routing for enhancing the performance of network. Considering the resource constrained UWSNs environment minimum computation is employed for implementing security so that SEECR remains suitable for underwater environment. In order to evaluate the performance of SEECR, this research compares the performance of SEECR with AMCTD: Adaptive Mobility of Courier Nodes in Threshold-optimized DBR - a well-known routing protocol for UWSNs environment. The performance of SEECR and AMCTD protocols are evaluated using different performance evaluation parameters such as number of alive nodes, transmission loss, throughput, energy tax and end-to-end delay. The results suggest an improved performance of SEECR over AMCTD. SEECR shows an improvement of 9% in terms of number of alive nodes, over 50% reduction in terms of transmission loss, up to 9% increase in throughput, up to 23% reduction in energy tax, and 25% reduction in end-to-end delay. Further, we observe that attack significantly degrades the performance of AMCTD whereas due to the embedded defense mechanism in SEECR the impact of attack is negligible.

I. INTRODUCTION
Underwater wireless sensor networks (UWSNs) consist of sensor nodes which are deployed under the water in order to sense the properties such as temperature, pressure and quality etc. [1]. The environment of UWSNs is very dissimilar from terrestrial wireless sensor networks (WSNs) in many ways. The electromagnetic signals cannot travel for long distance in UWSNs due to the high attenuation, scattering and the absorption effect [2]. In order to address this issue acoustic waves are used in UWSNs environment [3]. The propagation speed while using the acoustic waves in UWSNs environment is 1500m/sec which is much slower as compared to the radio The associate editor coordinating the review of this manuscript and approving it for publication was Longxiang Gao . waves based communication. In acoustic communication the end-to-end delay is high along with long propagation latency. The available bandwidth is very limited in acoustic communication (less than 100kHz) [2], [4], [5]. The sensor nodes in UWSNs environment are mostly considered static but due to the underwater activities these sensor nodes can move from 1 to 3m/sec. The sensor nodes in UWSNs environment are large in size and they consume more power due to which efficient utilization of energy is a critical factor in UWSNs communication. Charging and replacing of batteries in UWSNs environment is a challenging task [2], [6], [7]. The basic architecture of UWSNs is shown in Fig. 1.
The sensor nodes are deployed in UWSNs environment as shown in Fig. 1. The communication among sensor nodes under the water is through acoustic waves whereas the sink VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ node communicates with on-shore server through radio communication.
Applications of UWSNs include pollution monitoring, temperature and pressure measurements, environmental monitoring, seismic monitoring, assisted navigation, unmanned underwater exploration, submarine navigation, underwater disaster detection and prevention, underwater exploration such as corals, minerals and rare metals etc. [8]- [12]. The challenges in UWSNs environment are deployment, fabrication, maintenance and recovery cost is very high as compared to the terrestrial WSNs. Other challenges include power harvesting and power consumption, localization, deployment of sensor nodes, time synchronization, battery lifetime, underwater data collection, replacing or repairing sensor nodes, capacity of communication channels and security etc. [8], [13].
The basic security requirements in UWSNs are authentication, confidentiality, integrity and availability [14], [15]. There are different attacks possible in UWSNs which can abrupt the normal operation of the networks. The most common attacks include black hole attack, wormhole attack and sinkhole attack. Attacks in UWSNs environment can be grouped into the following two categories. i) Attack on the sensor node in UWSNs ii) Attack on the protocol in UWSNs. The first type of attack damages the sensor node but such type of attack is least likely in the real environment because it is very difficult to physically access the sensor nodes in UWSNs environment. The second type of attack target the communication protocols used in UWSNs. Once the communication protocol is compromised then it affects the entire network [13]. There are security requirements and countermeasure mechanisms to combat these attacks in UWSNs [15].
In this research work, SEECR protocol is proposed that is secure, energy efficient and cooperative in nature. SEECR efficiently utilizes the energy of sensor nodes for maximizing their life time. Cooperation technique used in SEECR has vital role in efficient utilization of energy. Moreover, security mechanism has been incorporated in SEECR for combating security attacks. Security mechanism involves minimum computation considering the resource constrained environment of UWSNs. SEECR detects and eliminates all those active attacks which drop the packets.
The rest of the research paper is organized as follows: Section II discusses the current state of the art. Section III includes motivation and contributions. Section IV discusses underwater channel. Section V includes detail of the proposed routing protocol (SEECR) whereas Section VI includes details of simulation environment and performance evaluation parameters. Results are presented and discussed in Section VII. Section VIII concludes the work and presents directions for future work.

II. LITERATURE REVIEW
Energy efficiency is a key consideration in wide range of applications [16]- [18]. The authors in [16] proposed a novel energy efficient key agreement mechanism for UWSNs. For reducing the communication overhead between sensor nodes UWSNs are gathered into the clusters. The key agreement mechanism proposed in this research can resist different security attacks which include replay attacks, sybil attack, spoofed attacks and node replication attacks etc. In the proposed mechanism geographical information and identity are added to the public as well as the private key of the sensor node in UWSNs in order to boost the capability of the mechanism to resist against different attacks. The results of the simulation revealed that the key agreement mechanism proposed in this research produced better results in terms of security as well as networks performance. Using the proposed mechanism the UWSNs can have high network connectivity and ensures that the energy consumption of the sensor node and cluster-head node is under 10J and 20J respectively.
The authors in [19] proposed key distribution scheme for peer to peer communication in mobile UWSNs environment. Two different mobility models are used in this research which are meandering and nomadic mobility model. In both the mobility models, structure used is hierarchical structure and communication is managed through distribution scheme known as Blom's key distribution scheme. Simulation results of this research indicated some minor connectivity issue due to mobility but the proposed scheme resolves connectivity issues on time. The key distribution scheme proposed in this research revealed better resiliency performance when some nodes are captured by the adversary and in that case very few links become compromised. Simulation results also revealed better performance of the proposed scheme in terms of security and energy consumption.
The authors in [20] proposed cryptographic mechanism for UASNs. The authors proposed encryption algorithm which is efficient in nature for protecting the integrity and confidentiality while considering the unique characteristics of UASNs environment. In this research some modifications are made in traditional AES-128 by using alternate approach instead of using S-Box. The proposed algorithm can resist against brute force as well as other attacks. It has been concluded in this research that the round key is not breakable using brute force attack. The performance of proposed cryptographic scheme is compared with AES-128, Blowfish and PRESENT. Simulation results revealed that as compared to AES-128 and other cryptographic schemes the proposed cryptographic approach is secure and energy efficient.
The authors in [21] proposed a design of secure routing approach for UASNs. The researchers in this research proposed an efficient signature schemes with no need of the online trusted third party. The scheme proposed in this research can tackle forgery attacks and improves the overall security. Moreover, a trapdoor scheme is presented which is used in routing messages for achieving anonymity among communicating nodes. The trap door design in the proposed scheme evades the overhead involved in maintaining huge number of pre-shared keys. The proposed routing scheme is compared with geographical information routing protocol based on partial network coding (GPNC) and level based adaptive geo-routing (LB-AGR). The performance parameters used are energy consumption, throughput and PDR. The results obtained from the simulation in this research revealed that the proposed routing scheme progresses security and the performance of the network become moderate.
The research conducted in [22] proposed distributed approach for combating routing attacks in UWSNs. The proposed approach can detect sinkhole attack and wormhole attack in UWSNs environment. Two different phases are used in the proposed approach such as silent monitoring phase and detection phase. According to the proposed approach of detection and mitigation each sensor node in UWSNs has to keep track of its neighbor by overhearing the messages sent and received by the neighbor. An analytical model is presented in this research in order to capture the interaction among different parameters. Simulation has been used for implementing the idea presented in this research and the obtained simulation results revealed that the suggested approach is efficient and correct.
The authors in [23] conducted detail study on denial of service (DOS) attack. In mobile UWSNs the DOS attack can be categorized into man in the middle attack, flooding attack and demolishing attack. The possible attacks in UWSNs environment are sybil attack, wormhole attack and selective forwarding attack. The influence of mentioned attacks on UWSNs is analyzed. They surveyed secure localization techniques for terrestrial WSNs. Simulation results revealed that there is dissimilarity in the performance of mobile UWSNs and mobile WSNs which indicates that the security mechanism suitable for WSNs are not appropriate for UWSNs environment.
The authors in [24] considered the requirements as well as security related issues of UWASNs. In this research it is proposed that same key should be utilized by both the sender and the receiver for the purpose of encryption and decryption as the size of the symmetric key is less as compared to the size of the asymmetric key. When all headers are added the entire message is entered in the algorithm known as message-authenticated code having a secret key which is shared. The message integrity code (MIC) which is an output value is attached to the entire message at its end. The purpose of the MIC is to ensure that each and every one bit of the complete message as well as the shared key is authentic. The encryption algorithm encrypts the message as well as the MIC. Receiver side calculates its MIC and compares it with the MIC received from sender side. If both the MIC are same the receiver accepts the message otherwise discards the entire message. The authors in this research revealed that minimum amount of overhead should be added to the data when applying security in UWASNs. They proposed to use cryptographic module validation program (CMVP) algorithm for the said purpose.
The research conducted in [25] proposed a secure neighbor discovery wormhole resilient routing scheme for UWSNs. This research used RIPEMID-160 and direction of arrival estimation as authentication mechanism for the purpose of secure neighbor discovery. The proposed scheme can resist against wormhole attack, impersonation attack and route poisoning attack. The scheme proposed in this research uses four agencies such as security agency, routing agency, underwater gateway agency and vehicle agency. Simulation results revealed that the secure routing scheme proposed in this research produced better results as compared to basic neighbor discovery schemes in terms of route maintenance overhead, consumption of energy, PDR and detection of failure.
The authors in [26] offered a secure MAC protocol for UASNs environment in order to efficiently manage the environment of UASNs regarding energy efficiency, data reliability, authenticity, confidentiality of data, and the prevention against attacker. In this research cryptography algorithm is used in MAC protocol in order to provide security services such as data authenticity, confidentiality of data and protection against replay attack. CCM-UW mode based on AES and ARIA algorithms are used in order to carry out this research. The proposed MAC protocol is efficient in terms of consumption of energy and transmission time. The proposed MAC protocol for UWSNs is compared with existing MAC protocols. The results revealed that the proposed MAC protocol is secure and efficient as compared to the existing solutions.
The authors in [27] evaluated different schemes of digital signature for the purpose of end-to-end authentication in UWSNs environment in terms of consumption of energy. The results obtained in this research revealed that the schemes which perform better in WSNs do not necessarily perform better in UWSNs due to the exceptional characteristics of UWSNs environment. Different characteristics of digital signatures schemes are identified in order to make them suitable for UWSNs. Three different digital signature schemes are evaluated for different UWSNs scenarios in this research in terms of power consumption. These schemes are: i) zhangsafavi-naini-susilo (ZSS), ii) elliptic curve digital signature algorithm (ECDSA) iii) boneh-lynn-shacham (BLS). The signature generation time for ZSS is 229ms, ECDSA is 134ms and BLS is 302ms. The signature size of ZSS is 21 bytes, ECDSA is 40 bytes and BLS is 21 bytes. It has been revealed in this research that the use of aggregate and short signatures can effectively increase the energy efficiency in UWSNs.
The authors in [28] proposed a secure neighbor discovery scheme for UASNs. This research proposed a suite of protocols which performs secure discovery of neighbor resilient against wormhole attack in UASNs environment. The proposed protocol utilizes the direction of arrival (DoA) signals approach. The proposed protocol can resist against wormhole attack with very good probability and without the requirement of traditional hard requirements. The proposed solution in this research comprises of four protocols. Protocols such as B-NDP and MA-NDP are appropriate for UANs environment with little density and applications in which end-to-end delay and connectivity are of the main concern. On the other side protocols such as SDV-NDP and DV-NDP are appropriate for those applications in which there is high density of node and extraordinary requirements for the resilience of wormhole attack.
The research conducted in [29] proposed a security suite for UASNs comprised of both static and mobile sensor nodes. The proposed security suite is comprised of cryptographic primitives and secure routing protocols for achieving confidentiality and integrity while considering the constraints of UASNs environment. The authors proposed a protocol known as FLOOD. The FLOOD protocol was not secure and for protecting the confidentiality and integrity of the FLOOD protocol the authors proposed a secure flood (SeFLOOD) protocol. The experimental results revealed that the protocols suite proposed in this research is suitable for UASNs environment and it has affordable power consumption and communication overhead. The authors claim that so far their proposed protocol suite is the first practical, complete and efficient solution for providing confidentiality and integrity in the UASNs environment.
The authors in [30] proposed Tic-Tac-Toe AI-MINIMAX algorithm for implementing security in UWSNs. The focus of this research is to find the best and secure path for routing in UWSNs. The approaches of game theory are extensively utilized in this research in order to protect the sensor nodes in UWSNs from attacks and to protect the security of data in UWSNs communication. The implementation of Minmax algorithm includes two players such as min and max. While its implementation in UWSNs max is the sender and min is the attacker. The pre-requisite of Minmax algorithm is Tic-Tac-Toe. The Tic-Tac Toe considers all the available situations and chooses the optimal and secure move in UWSNs environment without any threat. The implementation of the proposed algorithm is done using the following procedures such as i) finding best move ii) current move better condition iii) GameOver state condition iv) Making our AI smarter. The authors predict that utilizing the AI models intelligent attacks can be mitigated up to some extent in the UWSNs environment.
The authors in [31] proposed a trust management model called a trust cloud model (TCM) for UWSNs. The purpose of the trust model is quantifying trust relationship among sensor nodes in UWSNs environment. After evaluating the quantified results the sensor nodes can determine whether the other node is trust worthy or not. Therefore, in the UWSNs environment only trust worthy sensor nodes will be selected for the transmission of data. The performance of the proposed TCM has been evaluated using the following three aspects: i) performance of detection of the malicious node: TCM performed well as compared to LCT and CBTM because both the LCT and CBTM did not take trust timelines into account. ii) performance of the calculation of trust value: It has been observed that in TCM the behavior of communication of normal nodes is good and their trust values rise with the passage of simulation time whereas in the presence of malicious nodes the trust values decreases due to the loss of packets by malicious nodes. iii) performance of the transmission of data: It has been observed that the rate of communication which is successful under the TCM is more as compared to the other two trust models.

III. MOTIVATION AND CONTRIBUTIONS
Research has been done on different issues in UWSNs such as the researchers in [32]- [38] conducted study on the recent applications, issues and challenges in UWSNs environment. Majority of the research done in UWSNs environment have energy efficiency as primary concern because the sensor nodes in UWSNs environment are operated on built-in battery having limited life time. The research conducted in [39]- [47] proposed different energy efficient routing protocol for UWSNs environment. Security is equally important to energy efficiency in the UWSNs environment. Compromising security in UWSNs cannot be tolerated in most of the cases. Therefore, this research work proposed a secure, energy efficient and cooperative routing (SEECR) protocol for UWSNs environment. The performance of SEECR is compared with AMCTD [48] a well-known routing protocol for UWSNs environment. This research implemented attack in both SEECR and AMCTD protocol in order to reveal the consequences of security attack in both the routing protocols. This research will help the research community to realize the impact of a security attack in UWSNs environment and must consider security mechanism while designing routing protocols for UWSNs environment. Secure solutions should have minimum computations considering the resource constrained environment of UWSNs.

IV. UNDERWATER WIRELESS CHANNEL
This section includes the attenuation, noise, signal to noise ratio (SNR) and path loss modeling in underwater wireless channel.

A. ATTENUATION IN UNDERWATER WIRELESS CHANNEL
The propagation speed of sound in water (c) is approximately 1500 m/s. The attenuation which takes place over the distance l, for a signal of narrow-band having carrier frequency f , is calculated in Eq.(1) [49].
where A 0 is a constant known as the normalizing constant, k is called the spreading factor, having value k = 1.5, and a(f ) is called the absorption coefficient which is modeled by using the Thorp's formula [50].
where a (f ) is calculated in dB/km and f is calculated in KHz. The value of total attenuation A (l, f ) can be calculated in Eq. (3).
10 log(A(l, f )) = k × 10 log(l) + l × 10 log(a(f )) (3) where k ×10 log(l) denotes spreading loss and l×10 log(a(f )) denotes absorption loss. where N t , N s , N w , and N th are the noise due to turbulence, shipping, wind, and thermal activities respectively. The whole spectral density of noise power is calculated in Eq. (4) [51].

C. SNR IN UNDERWATER WIRELESS CHANNEL
The signal to noise ratio (SNR) for underwater wireless channel is computed in Eq. (5) [51].

D. PATH LOSS IN UNDERWATER WIRELESS CHANNEL
Signals in underwater wireless channel practice frequency as well as link length-dependent path loss that is quite complex as compared to the radio channels and it is modeled in Eq. (6) [52].

V. SEECR: SECURE ENERGY EFFICIENT AND COOPERATIVE ROUTING PROTOCOL FOR UNDERWATER WIRELESS SENSOR NETWORKS
The proposed SEECR protocol utilizes multi-hop networking in UWSNs environment by utilizing the cooperation technique. In the proposed scheme, data packets which are generated from source node are forwarded to the destination node or sink node via hop by hop. Deployment of the relay node is done at the joint which contains two consecutive hops that accepts the arriving packets and after that amplifies them, and then retransmits these packets towards the destination. The proposed scheme detects common active routing attacks which drop packets and eliminates attacker nodes from the network efficiently.
A. NETWORK MODEL Fig. 2 shows a network model of UWSNs, where S is the source node, R1 and Rb are relay nodes, D is the destination/sink node and A1 and A2 are attacker nodes deployed in UWSNs. Rb is the best relay node selected among the available relay nodes such as R1 and Rb. The line dark in color reflected in Fig. 2 is the communication path which is direct and the dotted lines show the routes which are cooperative and these cooperative routes are used when the direct path is either unavailable or infeasible. The relay node will be checked whether it is attacker node or not, if it is attacker node then it will not be selected for the transmission of data.

B. CONFIGURATION AND INITIALIZATION
In this phase configuration and initialization is done. Each sensor node broadcast its depth as well as residual energy information to the neighbors using hello packets for the initialization of network operation. In this phase each sensor node knows about their neighbor sensor nodes. Sink node sends the hello packet to all sensor nodes. Each sensor node computes its weight using Eq. (7).
where W i is the weight of node i, T l reflects the path loss, R i represents the residual energy of node i, D w represents depth of water and D i shows the depth of node i.

C. MOVEMENT SCHEME OF COURIER NODES
For eliminating flooding process the value of depth threshold of the sensor nodes is marked to 60m. When the numbers of dead nodes is increased by 20% then for increasing the number of depth based threshold neighbors, the depth threshold is set to 40m. When the number of dead nodes in UWSNs environment increase to 75% then the depth of the courier nodes are adjusted for improving the network lifetime as follows: where N d is the number of dead nodes C 1d , C 2d , C 3d , C 4d are the depth of courier nodes 1, 2, 3 and 4 respectively, as also utilized in AMCTD.

D. ELIMINATING ATTACKER NODES
For detecting and eliminating attacker node each sensor node stores packet which are sent and received by neighbor sensor nodes. The packets are stored in Q j and Q k . The incoming packets P in of sensor node are stored in Q j and the outgoing packets P out of sensor node are stored in Q k . After the packets are stored, both the values Q j and Q k will be compared. If both the values are not equal and the sensor node is not a sink node then there are chances that the sensor node is an attacker node. The value of A i (attack indicator) will be incremented by 1. If the value of A i reaches x for a sensor node then that node will be detected as an attacker node and it will be eliminated so that it cannot contribute in the routing process. The value of x is set to 3. The value of x can be adjusted according to the environment. The value of x should be set carefully. If the value of x is very high the attacker node will be there in network for more time, if the value of x is very low then there is possibility of removing genuine sensor node as attacker node.
The process of attacker node detection and elimination has to be performed by all sensor nodes in order to ensure that the attacker node cannot participate in any activity of the network. Since the sensor nodes are positioned underwater in UWSNs and physical access to the sensor nodes is difficult and time consuming due to which after the detection of attacker node(s) it can still physically exist in the premises of UWSNs but the attacker node(s) will not be able to participate in any operation of the network.

Q j = P in
Store incoming packets P in of a sensor node in Q j Q k = P out Store outgoing packets P out of a sensor node in Q k where S k represents sink and A i is the attack indicator.  Where S represents source, Rb represents best relay node, and D represents destination.
Y sd = H s * g sd + n sd (8) where Y sd as shown in Fig.3 is the signal that is directly conveyed from the source to the destination node, H s reflects channel among the source and destination node or sink node, g sd reflects the information which is broadcasted from the source to the destination node, and n sd reflects the ambient noise which is added to the channel H s .
Y sr = H s * g sr + n sr (9) Y sr shows the signal transmitted among the source node and the relay node, g sr is the information which is conveyed from the source towards the relay node, n sr reflects the ambient noise which is added to the channel H s , Y rd = Y sr * g rd + n rd (10) During network operation when the destination node do not receive data packets from the source node then the relay node starts transmission and send data packets. Y rd represents the data which is transmitted from the relay node towards the destination node, g rd represents the information which is forwarded by the relay node towards the destination node, and n rd reflects the noise which is added on the channel H s to the information Y sr .

F. RELAY STRATEGY AND ROUTING PHASE
A source sensor node S has n encompassing sensor nodes in its vicinity and it depends on the condition of the channel in order to figure out the neighbors most suitable for transferring its data towards the sink node. The source node selects the relay node from its neighborhood by equating their weights. The sensor node having the maximum value of W i is selected for the transmission of data. If the residual energy of the source is more than or equal to the residual energy of the relay node, then direction transmission will be done otherwise the transmission will be through relay node.
where R s is residual energy of source node and R r is residual energy of relay node. The technique known as amplify and forward is considered at the relay node which uses an amplification factor on the signal received from the source and prior to forwarding the signal towards the destination node.

G. COMBINING STRATEGY
The destination sensor node D utilizes signal to noise ratio combining (SNRC) technique as combining technique in order to combine the signals which are arriving from the source S and relay R. The SNRC is where the signals combined at the receiver have a weight equal to the SNR seen at each array element. SNRC definitely works better than equal ratio combining (ERC) because it considers the small scales fluctuations and weights those signals less (low SNR) while combining. SNRC can be calculated as: (11) where Y d shows the output signal which is combined at the destination node D, X 1 represents weight of direct path and X 2 represents weight of relay path.

H. WEIGHT UPDATING PHASE
When the number of dead nodes is less than or equal to 20% then weight of sensor nodes will be same as the weight calculated in the initialization phase in Eq. (7). When the number of dead nodes in UWSNs environment is more than 20% and less than 75% then statements following if condition will be used to calculate the weight. If the numbers of dead nodes are more than or equal to 75% of the sensor nodes then statement following else condition will execute and the weight will be computed according to the statement following else condition as follows: If N d > 20 and N d < 75 The complete working of SEECR is reflected in a flowchart as shown in Fig. 4.

VI. SIMULATION ENVIRONMENT AND PERFORMANCE EVALUATION PARAMETERS
SEECR and AMCTD protocols are evaluated in two different scenarios such as with and without attack. The attack scenario contains eight attacker nodes whereas there is no attacker node deployed in without attack scenario. The numbers of sensor nodes deployed are 225 whereas the numbers of sink nodes deployed at the surface of the water is set to 10. The total number of rounds in simulation is set to 9000 as shown in Table 1.

A. PERFORMANCE EVALUATION OF SEECR
For evaluating the performance of proposed SEECR, it is compared with AMCTD using different performance evaluation parameters. The performance of SEECR and AMCTD are evaluated in different scenarios such as with and without attack.

B. PERFORMANCE EVALUATION PARAMETERS
The following parameters are used in order to evaluate the performance of SEECR and AMCTD protocols in the scenarios such as with and without attack.

1) NUMBER OF ALIVE NODES
The number of alive nodes is calculated by subtracting the number of dead nodes from the total number of nodes during the entire simulation.

2) TRANSMISSION LOSS
It is the transmission loss between the source and sink node during a single round. The transmission loss is calculated in decibels (dB).

3) THROUGHPUT
It is the entire number of packets which reach the sink node.

4) ENERGY TAX
It is the energy consumed while forwarding the data from the source node towards the sink node. It is calculated in joules.

5) END-TO-END DELAY
It is the time between packet generation and packet reach the sink. It is calculated in milliseconds.

VII. RESULTS AND DISCUSSION
The results obtained in this research from different performance evaluation parameters are discussed in this section of the research paper. The results obtained are presented in the form of both figures and tables. Fig. 5 shows the number of alive sensor nodes during the entire simulation. Initially the number of sensor nodes deployed is set to 225 which includes all the source and relay nodes. At the end of simulation the number of alive nodes in SEECR with and without attack is 111 and 112 respectively, the number of alive nodes in AMCTD with and without attack is 68 and 82 respectively. The results obtained show that attack significantly affects the energy consumption due to which the numbers of alive nodes is significantly less in AMCTD with attack scenario. Fig. 5 further revealed that stability is better in SEECR as compared to AMCTD in both the scenarios such as with and without attack. SEECR improves the overall stability of the network whereas attack causes more instability in AMCTD protocol with attack scenario. Moreover, due to the embedded security defense mechanism in SEECR there is almost negligible impact of attack on SEECR as we can see in the scenario of SEECR with attack. Table 2 shows the number of alive nodes after every 1000 rounds in different scenarios. It is clear from the results  obtained in Table 2 that the performance of SEECR is better than AMCTD in terms of number of alive nodes due to the energy efficient approach adopted in SEECR protocol. The percentage of alive nodes is calculated for the entire simulation. Moreover, it can be seen in Table 2 that the overall percentage of alive nodes in SEECR with and without attack is 60.9% and 63.2% respectively, which shows very little impact of attack and the little impact is due to the computation done for the detection as well as elimination of attacker nodes from the network. On the other side it is clear from Table 2 that the overall percentage of alive nodes in AMCTD with and without attack is 51.7% and 56.6% respectively. There is a significant degradation in the performance of AMCTD routing protocol in the presence of attack. The attack consistently degrades the performance of AMCTD protocol due to no defense mechanism. Moreover, SEECR protocol beats AMCTD protocol in both the scenarios such as with and without attack. Fig. 6 shows the transmission loss of SEECR and AMCTD protocols in different scenarios such as with and without attack. The results obtained revealed that the transmission loss of SEECR with and without attack is significantly less as compared to the transmission loss of AMCTD with and without attack. The energy efficient approach utilized VOLUME 8, 2020  by SEECR protocol significantly reduced the transmission loss in SEECR protocol. Moreover, SEECR protocol beats AMCTD routing protocol in terms of transmission loss. Table 3 shows the transmission loss of SEECR and AMCTD protocols after every 1000 rounds in different scenarios. The results mentioned in Table 3 indicates that the transmission loss in SEECR protocol with and without attack is significantly less as compared to the transmission loss of AMCTD protocol with and without attack. Moreover, the overall percentage of transmission loss in SEECR protocol with and without attack is 43.7% and 43.1% respectively, whereas the overall percentage of transmission loss in AMCTD with and without attack is 100% and 98.8% respectively. It is clear from Table 3 that as compared to AMCTD there is more than 50% reduction of transmission loss in SEECR protocol in both the scenarios such as with and without attack. Fig. 7 shows the throughput of SEECR and AMCTD protocols in different scenarios such as with and without attack. The results obtained show that the throughput of SEECR protocol is more as compared to the throughput of AMCTD protocol in both the cases such as with and without attack. The better throughput in SEECR protocol than AMCTD protocol is due to the energy efficient and secure mechanism utilized in K. Saeed et al.: SEECR Protocol for UWSNs  SEECR protocol. It is obvious from Fig. 7 that SEECR beats AMCTD in terms of throughput. Table 4 shows throughput of SEECR and AMCTD protocols after every 1000 rounds in different scenarios. The results mentioned in Table 4 revealed that the throughput of SEECR protocol is much better as compared to the throughput of AMCTD protocol. Moreover, it has been revealed in the results obtained that attack significantly degrades the performance of AMCTD protocol whereas the effect of attack is negligible on SEECR protocol due to the strong defense mechanism embedded in SEECR protocol. The overall throughput percentage of SEECR protocol with and without attack is 42.4% and 42.6% respectively, whereas the throughput percentage of AMCTD protocol with and without attack is 32.9% and 37.6% respectively, which shows significant degradation in the performance in AMCTD protocol in the presence of attack. Fig. 8 shows the energy tax of SEECR and AMCTD protocols in different scenarios such as with and without attack. The results obtained show that the energy tax of SEECR protocol is much less as compared to the energy tax of AMCTD protocol in both the scenarios such as with and without attack. Moreover, it is obvious from Fig. 8 that attack significantly  degrades the performance of AMCTD protocol in terms of energy tax whereas the effect of energy tax on SEECR protocol is negligible. Table 5 shows energy tax of SEECR and AMCTD protocols after every 1000 rounds in different scenarios. The results obtained in Table 5 revealed that energy tax of SEECR protocol is much less as compared to the energy tax of AMCTD protocol in both the cases such as with and without attack. Moreover, it is also clear from Table 5 that in attack scenario the energy tax of AMCTD protocol increases. The overall percentage of energy tax of SEECR protocol with and without attack is 77.3% and 76.1% respectively, whereas the percentage of energy tax of AMCTD protocol with and without attack is 100% and 88.3% respectively. It can be concluded from Table 5 that the energy tax in SEECR protocol is up to 23% less as compared to AMCTD protocol. Fig. 9 shows end-to-end delay of SEECR and AMCTD protocol in different scenarios such as with and without attack. Results reflected in Fig. 9 shows that there is major difference in the end-to-end delay of SEECR and AMCTD protocols. The end-to-end delay is less in SEECR protocol as compared to the end-to-end delay of AMCTD protocol. Table 6 shows end-to-end delay of SEECR and AMCTD protocols after every 1000 rounds in different scenarios.   The result obtained in Table 6 indicates that the overall percentage of end-to-end delay in SEECR protocol with and without attack is 70.2%, whereas the percentage of end-toend delay in AMCTD with and without attack is 95.6% and 100% respectively. It can be concluded from Table 6 that the end-to-end delay is 25% less in SEECR protocol as compared to that in AMCTD protocol.

VIII. CONCLUSION AND FUTURE WORKS
Routing attacks in UWSNs environment is an important factor which needs to be addressed. This research work proposed SEECR protocol for UWSNs environment. The proposed routing protocol efficiently utilizes the energy consumption and has built-in defense mechanism against common active attacks in UWSNs environment. The performance of SEECR protocol has been compared with AMCTD protocol in terms of different performance evaluation parameters. The results obtained revealed that SEECR protocol beats AMCTD protocol in terms of all performance evaluation parameters. Moreover, due to the built-in defense mechanism in SEECR protocol there is negligible impact of attack on SEECR protocol in UWSNs environment.
This research work is focused on the importance of energy efficient security mechanism in routing protocol for UWSNs environment. The results produced in this research revealed that despite of computation for attacker node detection and elimination the proposed solution is still suitable for UWSNs environment. The proposed solution in this research will encourage the research community to design secure solutions for UWSNs as well as other environments. Some possible research directions in this area can be designing other secure routing solutions and using artificial intelligence models for mitigating attacks in UWSNs environment. In the future we have a plan to introduce other energy efficient secure solutions for different environments.

IX. CONFLICTS OF INTEREST
The authors declare that there is no conflict of interest regarding the publication of this paper.
KHALID SAEED received the M.S. degree (Hons.) in computer engineering from the University of Engineering and Technology at Taxila, Pakistan, in 2011. He is currently pursuing the Ph.D. degree in computer science with the University of Engineering and Technology at Peshawar, Peshawar, Pakistan. His research interests include security in underwater wireless sensor networks, wireless sensor networks, cloud computing, mobile ad-hoc networks, delay tolerant networks, and information and communication technologies. Till date he has more than 20 publications to his credit in IEEE conferences and reputed journals.
WAJEEHA KHALIL received the Ph.D. degree from the University of Vienna. She is currently an Assistant Professor with the Department of Computer Sciences and Information Technology, University of Engineering and Technology at Peshawar, Peshawar. Her research interests include distributed computing, human-computer interaction, and parallel computing.
SHEERAZ AHMED received the Ph.D. degree in electrical engineering from COMSATS Islamabad, Pakistan, in the domain of underwater and body area sensor networks. He is currently a Professor with Iqra National University, Peshawar, Pakistan. His research interests include UWSNs, WBANs, smart grids, VANETs, FANETs, and so on. Till date he has more than 160 publications to his credit in IF journals and IEEE conferences. He has an experience of more than 20 years in teaching, research, and administrative positions.
IFTIKHAR AHMAD received the master's degree in computer science from the University of Freiburg, Germany, and the Ph.D. degree in theoretical computer science from Saarland University, Saarbrücken, Germany. He is currently an Assistant Professor with the Department of Computer Science and Information Technology, University of Engineering and Technology at Peshawar, Pakistan. He is also leading the National Center of Big Data and Cloud Computing, Machine Learning Group, University of Engineering and Technology at Peshawar. His research interests include theoretical computer science, machine learning, graph theory, and blockchain and cryptocurrencies.
MUHAMMAD NAEEM KHAN KHATTAK is currently working as the Chairman at the Department of Mechanical Engineering, University of Engineering and Technology at Peshawar, Peshawar. He has his specialization in technology and quality management. In particular, his areas of interest include total quality management and technology and innovation management. He has produced several publications internationally in the fields of his interest. VOLUME 8, 2020


AIM 2020 Challenge on Video Extreme Super-Resolution: Methods and Results

This paper reviews the video extreme super-resolution challenge associated with the AIM 2020 workshop at ECCV 2020. Common scaling factors for learned video super-resolution (VSR) do not go beyond factor 4. Missing information can be restored well in this region, especially in HR videos, where the high-frequency content mostly consists of texture details. The task in this challenge is to upscale videos with an extreme factor of 16, which results in more serious degradations that also affect the structural integrity of the videos. A single pixel in the low-resolution (LR) domain corresponds to 256 pixels in the high-resolution (HR) domain. Due to this massive information loss, it is hard to accurately restore the missing information. Track 1 is set up to gauge the state-of-the-art for such a demanding task, where fidelity to the ground truth is measured by PSNR and SSIM. Perceptually higher quality can be achieved in trade-off for fidelity by generating plausible high-frequency content. Track 2 therefore aims at generating visually pleasing results, which are ranked according to human perception, evaluated by a user study. In contrast to single image super-resolution (SISR), VSR can benefit from additional information in the temporal domain. However, this also imposes an additional requirement, as the generated frames need to be consistent along time.

Introduction
Super-resolution (SR) aims at reconstructing a high-resolution (HR) output from a given low-resolution (LR) input. Single image SR (SISR) generally focuses on restoring spatial details as the input consists of only one single image. By comparison, as the input of video SR (VSR) is usually composed of consecutive frames, it is expected to concentrate on the exploitation of the additional temporal correlations, which can help improving restoration quality over SISR methods. Making full use of the temporal associations among multiple frames and keeping the temporal consistency for VSR remain non-trivial problems. Furthermore, when moving towards more extreme settings, like higher scale factors that require to restore a large amount of pixels from severely limited information, the VSR problem will get much more challenging.
Following our first AIM challenge [9], the goal of this challenge is to super resolve the given input videos with an extremely large zooming factor of 16, with searching for the current state-of-the-art and providing a standard benchmark protocol for future research in the field. Fig.1 presents a few downscaled crops and their corresponding HR crops. In this challenge, track 1 aims at probing the state-of-the-art for the extreme VSR task, where fidelity to the ground truth is measured by peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). As a trade-off for the fidelity measurement, track 2 is designed for the production of visually pleasing videos, which are ranked by human perception opinions with a user study.

Related Work
For SR, deep learning based methods [6,21,25,41,40,4,32] have proven their superiority over traditional shallow learning methods. For example, [6] introduces convolution neural networks (CNN) to address the SISR problem. In particular, it proposes a very shallow network to deeply learn LR features, which are subsequently leveraged to generate HR images via non-linear mapping. To reduce the time complexity of the network operations in the HR space, [43] proposes an effective sub-pixel convolution network to extract and map features from the LR space to the HR space using convolutional layers instead of classical interpolations (e.g., bilinear and bicubic). [56] exploits a residual dense network block with direction connections for a more thorough extraction of local features from LR images. A comprehensive overview of SISR methods can be found in [47].
Compared with SISR, the VSR problem is considerably more complex due to the additional challenge of harnessing the temporal correlations among adjacent frames. To address this problem, a number of methods [26,34,5] are suggested to leverage temporal information by concatenating multiple LR frames to generate a single HR estimate. Following this strategy, [2] first warps consecutive frames towards the center frame, and then fuses the frames using a spatio-temporal network. [20] aggregates motion compensated adjacent frames, by computing optical flow and warping, followed by a few convolution layers for the processing on the fused frames. [29] calculates multiple HR estimates in parallel branches. In addition, it exploits an additional temporal modulation branch to balance the respective HR estimates for final aggregation. By contrast, [19] relies on implicit motion estimation. Dynamic upsampling filters and residuals are computed from adjacent LR frames with a single neural network. Finally, the dynamic upsampling filters are employed to process the center frame, which is then fused with the residuals. Similarly, [31] proposes a dynamic local filter network to perform implicit motion estimation and compensation. Besides, it suggests a global refinement neural network based on residual block and autoencoder structures to exploit non-local correlations and enhance the spatial consistency of the superresolved frames. To address large motions, [48] devises an alignment module to align frames with deformable convolutions in a coarse-to-fine manner. Besides, it suggests a fusion module, where attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration.
In addition to the aggregation strategy, some other works suggest to make use of recurrent neural networks (RNN) for better VSR. Due to the better capacity of learning temporal information on input frames, they provide a potentially more powerful alternative to address the SR problem. For instance, [45] suggests an autoencoder style network as well as an intermediate convolutional long shortterm memory (LSTM) layer. The whole network is capable of processing the preliminary HR estimate from a subpixel motion compensation layer, for better HR estimate. [15] proposes a bidirectional recurrent network, which exploits 2D and 3D convolutions with recurrent connections and combines a forward and a backward pass to produce the HR frames. To make use of temporal information, [42] designs a neural network that warps the previous HR output towards the current time step, by observing the optical flow in LR space. The warped output is concatenated with the current LR input frame and a SR network generates the HR estimate. [8] exploits a recurrent latent space propagation (RLSP) algorithm for more efficient VSR. Particularly, RLSP introduces high-dimensional latent states to propagate temporal information along frames in an implicit manner so that the efficiency can be highly improved.

Data
We use the Vid3oC [22] dataset for this challenge, which has been part of previous challenges [9,10]. The dataset is a collection of videos taken with three different cameras on a rig. This results in roughly aligned videos, which can be used for weak supervision. In this challenge, only the high-quality DSLR camera (Canon 5D Mark IV) is used to serve as HR ground truth. The corresponding LR source data is obtained by downscaling the ground truth by factor 16, using MATLAB's imresize function with standard settings, see Fig. 2. In order to retain proper pixel-alignment, the ground truth 1080 × 1920 FullHD frames are cropped to 1072 × 1920 before downscaling, to be dividable by 16. We provide 50 HR sequences to be used for training. To save bandwidth, these videos are provided as MP4 files together with scripts to extract and generate the LR source frames. Additionally, the dataset contains 16 paired sequences for validation and 16 paired sequences for testing, each consists of 120 frames in PNG format.

Challenge Phases
The challenge is hosted on CodaLab and is split up in a validation and a test phase. During the validation phase, only the validation source frames are provided and participants were asked to submit their super-resolved frames to the CodaLab servers to get feedback. Due to storage constraints on CodaLab, only a subset of frames could be submitted to the servers (every 20th frame in the sequence). In the following test phase, the final solutions had to be submitted to enter the challenge ranking. There was no feedback provided at this stage, in order to prevent overfitting to the test set. Additionally, the full set of frames had to be made accessible to the challenge organizers for the final rankings. After the submission deadline, the HR validation ground truth was released on CodaLab, for public use of our dataset.

Track 1 -Fidelity
This track aims at high fidelity restoration. For each team, the restored frames are compared to the ground truth in terms of PSNR and SSIM and can be objectively quantified by these pixel-level metrics. The focus is on restoring the data faithfully to the underlying ground truth. Commonly, methods for this task are trained with a pixel-level L1-loss or L2-loss. The final ranking among teams is determined by PSNR/SSIM exclusively, without visual assessment of the produced frames.

Track 2 -Perceptual
Super-resolution methods optimized for PSNR tend to oversmooth and often fail to restore the highest frequencies. Also, PSNR does not correlate well with human perception of quality. Therefore, the focus in the field has shifted towards generation of perceptually more pleasing results in trade-off for fidelity to the ground truth. Since the extreme scaling factor 16 and its associated large information loss prohibits high fidelity results, the only possibility to achieve realistically looking HR videos in this setting, is by hallucinating plausible high frequencies. Track 2 is aimed at upscaling the videos for highest perceptual quality. Quantitative assessment of perceptual quality is difficult and remains largely an open problem. We therefore resort to a user study in track 2, which is still the most reliable benchmark for perceptual quality evaluation.

KirinUK
Recent video super-resolution approaches [3] propose splitting the spatio-temporal attention operation in several dimensions. Their aim is to reduce the computational cost of a traditional 3D non-local block. Nevertheless, these methods still need to store the HWxHW attention matrices, which is challenging, especially when dealing with GPUs with limited amount of memory or when upscaling HR videos. To tackle this, the KirinUK team proposes to extend the VESR-Net [3] architecture by replacing the Separate Non Local (SNL) module with an Efficient Point-Wise Temporal Attention Block (EPAB). This block aggregates the spatio-temporal information with less operations and memory consumption, while still keeping the same performance. The team names this new architecture Efficient Video Enhancement and Super-Resolution Net (EVESRNet) and an overview of it can be seen in Fig. 3. It is mainly composed of Pyramid, Cascading and Deformable Convolutions (PCDs) [48], the EPAB, and Channel Attention Residual Blocks (CARBs) [3] [55].
The EPAB module is illustrated in Fig.4 and can be divided into two subblocks, the Spatio-Channel Attention (SCA) and the Spatio-Temporal Attention (STA). Both of them share the same structure, the only difference is the permutation operation at the beginning and at the end of the STA sub-block.
To perform Extreme Video SR, the team employs two 4x stages in cascade mode. Each stage was trained independently with an EVESRNet architecture. Moreover, their training does not start from scratch, they first pretrain a model with the REDS dataset [35] to initialize the networks. This helps preventing overfitting in the first stage where the amount of spatial data is limited. The REDS dataset was only utilized for pretraining. As a reference, the initialization model achieves 31.19 PSNR in the internal REDS validation set defined in [48], which corresponds to a +0.1 dB improvement with respect to EDVR [48].
Each track uses the same pipeline, the only differences are: 1) For the fidelity track, the two stages were trained using L2 loss. 2) For the perceptual track, the second stage was trained using the following combined loss: where λ 1 is 1e-3, λ 2 is 1 and λ 3 is 5e-3. They used a patch discriminator as in [18]. The rest of the hyperparameters are the same as in [49].

Team-WVU
Recently, deformable convolution [57] has been received increasingly more attention to solve low-level vision tasks such as video super-resolution. EDVR [48] and TDAN [46] have already successfully implemented deformable convolution to temporally align reference frame and its neighboring frames which can let networks better utilize both spatial and temporal information to enhance the final results. Inspired by state-of-the-art video SR method EDVR [48], the team develops the novel Multi-Frame based Deformable Kernel Convolution Network [51] to temporally align the non-reference and reference frames with deformable kernel [12] convolution alignment module and enhance the edge and texture features via deformable kernel spatial attention module.
The overall diagram of proposed network is shown in Fig. 5. It mainly includes four parts, feature extractor, DKC Align module (deformable kernel convolution alignment module), reconstruction module and upscale module. Different from PCD alignment module from EDVR, the team implemented stacked deformable kernel convolution layers instead of traditional convolution layer to extract offset. Deformable kernel can better adapt effective receptive fields than  normal convolution [12] which can better enhance the offset extraction compared with the normal convolution. On the step of reconstruction, to calibrate reconstructed feature maps before feeding into each upscaling module, the team proposes a Deformable Kernel Spatial Attention (DKSA) module (integrated to reconstruction module) to enhance the textures that can help the proposed network to reconstruct SR frames sharper and clearer. Because this challenge aims to super-resolve extreme LR videos with the scale factor of 16, to avoid generating undesired blurring and artifacts, the LR frames are super-resolved with a scale factor of 2 each time (see Fig. 5). Finally, the LR frames are super-resolved four times in total to upscale the LR frames with a magnification factor ×16. Charbonnier Loss [48,24] is used as the loss function for both track 1 and track 2 training, the loss can be expressed as follows: where ξ = 1 × 10 −3 ,X r is super-resolved frame and X r is target frame (groundtruth).

BOE-IOT-AIBD
The team proposes 3D-MGBP, a fully 3D-convolutional architecture designed to scale efficiently for the difficult task of extreme video SR. 3D-MGBP is based on the Multi-Grid Back-Projection network introduced and studied in [38,36,37].
In particular, they extend the MGBPv2 network [36] that was designed to scale efficiently for the task of extreme image SR and was successfully used in the 2019-AIM Extreme Image SR competition [33] to win the Perceptual track of that challenge. For this challenge they redesigned the MGBPv2 network to use 3D-convolutions strided in space. The network works as a video enhancer that, ignoring memory constrains, can take a whole video stream and outputs a whole video stream with the same resolution and framerate. They input a 16× Bicubic upscaled video and the network enhances the quality of the video stream. The receptive field of the network extends in space as well as time by using 3D-convolutional kernels of size 3 × 3 × 3. The overall architecture uses only 3D-convolutions and ReLU units. This is in contrast to general trends in video processing networks that often include attention, deformable convolutions, warping or other non-linear modules. Figure 6 displays the diagram of the 3D-MGBP network used in the competition. In inference it is impossible for 3D-MGBP to process the whole video stream and so they extend the idea of overlapped patches used in MGBPv2 by using overlapped spatio-temporal patches (overlapping in space and time). More precisely, to upscale arbitrarily long video sequences they propose a patch based approach in which they average the output of overlapping video patches produced by the Bicubic upscaled input. First, they divide input streams into overlapping patches (of same size as training patches) as shown in Figure 6; second, they multiply each output by weights set to a Hadamard window; and third, they average the results.
They trained the 3D-MGBP network starting from random parameters (no pre-trained models were used). For the Fidelity track they trained the model using L2 loss on the output spatio-temporal patch. For the Perceptual track they submitted the output of two different configurations of the same architecture. The first submission, labeled Smooth, was trained with L2 loss as they noticed better time-consistency and smooth edges. In their second submission, labeled Texture, they followed the loss and training strategy of G-MGBP [37], adding a noise input to activate and deactivate the generation of artificial details. The noise input consists of one channel of Gaussian noise concatenated to the Bicubic upscaled input. In this solution, although more noisy and farther from ground truth due to the perception-distortion trade-off [1], they noticed better perception of textures.

ZZX
In order to restore the high-frequency information of the video, the team designed the multi-scale aggregated upsampling based on high frequency attention (MAHA) network. The framework is illustrated in Fig. 7. The team inputs seven LR frames into the feature extraction module, and inspired by the EDVR [48], the Pyramid, Cascading and Deformable (PCD) alignment module was applied to address global motion. The non-local module was used to select valid interframe information. Then, the extracted reference frame features are concatenated with the alignment features that were utilized to perform spatial-temporal fusion in a progressive strategy, which helps to aggregate spatial-temporal information.  Next, the team also proposed an attention-guided multi-level residual feature reconstruction module to fully improve feature representation. Finally, to generate a sharp structure HR video, the team computed the gradient map of the LR image to guide the spatial attention module. The team divided the network training into two stages for both track 1 and track 2. For stage 1, the L1 loss was used. For stage 2, the team fine-tuned the results of stage 1 and using 0.15 * L L1 + 0.85 * L SSIM to train the network. However, the team applied different testing strategies for track 1 & 2 . For Track 1, they employed model fusion testing and test enhancement strategies to improve the PSNR value. For the track 2, the best validation performance was used to directly generate the test results.

sr xxx
The team employs the high-level architecture design of EDVR [1], with improvements to accommodate large upscaling factors with up to 16. The used network is illustrated in Fig.8. To explain the framework, they first start with EDVR as baseline. EDVR is a unified framework which can achieve good alignment and fusion quality in video restoration tasks. It proposed a Pyramid, Cascading and Deformable (PCD) alignment module, which for the first time uses deformable  convolutions to align temporal frames. Besides, EDVR includes a Temporal and Spatial Attention (TSA) fusion module to emphasize important features.
Their proposed network takes 5 LR frames as input and generates one HR output image frame. They first conduct feature extraction, followed by PCD alignment module, Non-local module and TSA module to align and fuse multiple frames. Right after the reconstruction module, they use adaptive weighted multiscale (AWMS) module as our upsampling layer. In the last module, they add the learned residual to a direct bicubic upsampled image to obtain the final HR outputs.
They trained two different reconstruction models and ensemble their results to obtain more stable texture reconstructions. To incorporate finer detail ensembling, they combine residual feature aggregation blocks [30] and residual channel attention blocks [55].

lyl
As shown in Fig.9, the team proposes a coarse to fine network for progressive super-resolution reconstruction. By using the suggested FineNet:lightweight upsampling module (LUM), they achieve competitive results with a modest number of parameters. Two requirements are contained in a coarse to fine network(CFN): (1) progressiveness and (2) merge the output of the LUM to correct the input in each level. Such a progressive cause-and-effect process helps to achieve the principle for image SR: high-level information can guide an LR image to recover a better SR image. In the proposed network, there are three indispensable parts to enforce the suggested CFN: (1) tying the loss at each level (2) using LUM structure and (3) providing a lower level extracted feature input to ensure the availability of low-level information.
They propose to construct their network based on the Laplacian pyramid framework, as shown in Fig.9. Their model takes an LR image as input and progressively predicts residual images at S 1 , S 2 ...S n levels where S is the scale factor. For example, the network consists of 4 sub-networks for super-reconstructing an LR image at a scale factor of 16, if the scale factor is 3, S = S 1 × S 2 , S 1 = 1.5, S 2 = 2. Their model has three branches: (1) feature extraction and (2) image reconstruction (3) loss function.

TTI
The team's base network for x16 video SR is STARnet [13] shown in Fig.10. With the idea that space and time are related, STARnet jointly optimizes three tasks (i.e., space SR, time SR, and space-time SR). In the experiments, STARnet was initially trained using three losses, i.e. space, time, and space-time losses, which evaluate the errors of images reconstructed through space SR paths (red arrows in the figure), time SR paths (blue arrows), and space-time SR paths (purple arrows), respectively. The network is then fine-tuned using only the space loss for optimizing the model, which is specialized for space SR. While space SR in STARnet is basically based on RBPN [14], this fine-tuning strategy allows them to be superior to RBPN trained only with the space loss. While the original STARnet employs pyflow [28] for optical flow computation, pyflow almost cannot estimate optical flows in LR frames in this challenge (i.e., 120 × 67 pixels). The optical flows are too small between subsequent frames. Based on an extensive survey, we chose sift-flow [27] that shows better performance on the LR images used in this challenge.

CET CVLab
The architecture used by the team is inspired from the wide activation based network in [52] and channel attention network in [55]. As shown in Fig.11, the network mainly consists of 3 blocks. A feature extraction block, a series of wide activation residual blocks and a set of progressive upsampling blocks(×2). Charbonnier loss is used for training the network as it better captures the edge information than with mean squared errorloss (MSE).

Track 1 -Fidelity
This track aims at restoring the missing high frequencies that were lost during downsampling with the highest fidelity to the underlying ground truth. Challenge track 1 has 65 registered participants, from which 12 submitted solutions to the validation server and 8 teams entered the final ranking. The ranking, along with details about the training and testing are summarized in Tab is highly challenging and requires high-complexity networks in order to restore the details from learned priors. The top teams mostly employ window based approaches, attention modules and 3D-convolutions to additionally aggregate the temporal information. Team lyl and CET CVLab do not process temporal information in their networks and instead rely only on a single frame for the upscaling process. They can not compete with the top teams, which shows the importance of temporal information for high-quality restoration. The winner in track 1 is team KirinUK with a PSNR score of 22.83dB, followed by team Team-WVU and BOE-IOT-AIBD, which share the second place due to their identical PSNR scores.
Metrics Since this track is about high fidelity restoration, we rank the teams according to PSNR, which is a pixel-level metric. Additionally, we compute SSIM scores which is a metric based on patch statistics and is considered to correlate better with human perception of image quality. PSNR does not explicitly enforce to retain smooth temporal dynamics. It is therefore possible, that a method can generate high image quality on frame level, but introduces temporal artifacts like flickering. Most window based and 3D-convolution approaches however manage to produce frames with only minimal flickering artifacts, as they have access to adjacent frames. On the other hand, the flickering is very prominent for the single frame enhancers in this challenge.
Visual Results In addition to the metrics, we also provide visual examples in Fig. 12 for all competing methods in the challenge. We also show the Bicubic baseline (MATLAB's imresize) together with the ground truth frames for reference. All methods manage to clearly outperform the Bicubic baseline, which is also reflected in the PSNR and SSIM metrics in Tab. 1. The methods improve PSNR and SSIM by 1.08dB to 2.14dB and 0.0342 to 0.0680 respectively. As expected for such a challenging task, no method is capable of restoring all the fine details present in the ground truth. Predominantly, only sharp edges and smooth textures can be recovered even by the top teams, since the information loss is so extreme. Interestingly, teams KirinUK, Team-WVU, BOE-IOT-AIBD, sr xxx, ZZX and TTI restore two windows (blue box highlight, second column) instead of a single one as present in the ground truth frame. This could indicate, the method's upscaling for such a large factor is highly dependent on image pri- ors and having access to temporal information is not sufficient to achieve high restoration quality. Still, a top ranking is only achieved by teams that leverage  temporal information, which shows its importance for video super-resolution, even in such extreme settings.

Track 2 -Perceptual
Due to the extreme information loss, it is hard to accurately restore the highfrequency content with respect to the ground truth. If deviations from the ground truth can be accepted and more visually pleasing results are desired, perceptual quality can be traded-off for fidelity. The results may not entirely reflect the underlying ground truth, but instead boost the perceptual quality considerably. We therefore do not rely on PSNR and SSIM for evaluation in track 2, but instead conduct a user study to asses human perceptual quality. Challenge track 2 has 54 registered participants, from which 7 submitted solutions to the validation server and 7 teams entered the final ranking Metrics Assessing perceptual quality quantitatively is difficult and remains largely an open problem. Attempts for such metrics have been made in the past and one of the most promising metrics is called Learned Perceptual Image Patch Similarity (LPIPS), which is proposed in [54]. This metric measures similarity to the ground truth in feature space of popular architectures, e.g. [23]. While this metric is widely adopted for perceptual quality assessment, especially on images, it still fails in some cases to reflect human perception. Like PSNR and SSIM, it does not discriminate on temporal dynamics, which are crucial for high quality videos. User Study Since quantitative metrics are not reliable, we resort to a user study to rank the participating teams in track 2. For that matter we split the evaluation in two separate user studies, a frame level study and a video level study. The frame level study is meant to judge the image level quality and is performed on randomly subsampled frames from all 16 sequences in the test set. The competing method's frames are compared side-by-side in a one vs. one setting, resulting in 28 comparisons per frame. We asked 10 users to judge the frame level quality, which results in 16 × 28 × 10 = 4480 total ratings. The detailed results are shown in a confusion matrix in Tab. 2. Each row shows the preference of the method in the first column against all other methods. Additionally, we show the total number of preferences (Wins (tot)) plus the relative preferences, which are normalized to the number of total comparisons 16×7×10 = 1120 with all teams (Wins (%)). The video level study is meant for evaluating the temporal dynamics in the videos and the overall perceptual quality when watching the videos. Again, we generated side-by-side videos between all methods for comparison in a one vs. one setting. The videos are generated by compiling all 1920 frames from 16 sequences into a single video, showing two competing methods. This results in 28 short videos of ≈ 1 minute. We also ask 10 users to perform the video level user study and get a total number of 28 × 10 = 280 ratings. In order to directly compare with the frame level study, we normalize the total wins by the total comparisons with all the teams (10 × 7 = 70) to get the relative scores. We derive the ranking from the combined relative scores of both frame level and video level user studies (see Tab. 2, lower right). Team KirinUk is the clear winner in track 2, followed by Team-WVU and ZZX on the second and third place. We also provide PSNR, SSIM and LPIPS metrics for reference. A qualitative illustration of all track 2 results is presented in Fig. 13, including the user study results. Note, the final ranking is only derived from the user study.
Visual Results To allow direct comparison to track 1, we provide the visual results on the same frames for track 2 in Fig. 14. Note that teams Team-WVU, sr xxx, lyl and CET CVLab submitted the same set of frames to both tracks, while KirinUK, BOE-IOT-AIBD and ZZX adapted their solutions to the specific requirements in both tracks. BOE-IOT-AIBD even provided two distinct solutions for track 2. One is optimized with emphasis on textures, the other is designed for temporal smoothness, abbreviated with (t) and (s) respectively in Tab. 2. Surprisingly, the texture based solution of BOE-IOT-AIBD also performs better in the video level user study. According to the users, the sharper texture details seem to have a higher impact on the quality than the flickering artifacts. The winning team KirinUK manages to not only outperform all other teams in both user studies, but also in the provided metrics PSNR, SSIM and LPIPS. However, it has to be considered, that the solution is optimized with L1 and VGG-loss, which are both closely related to these metrics. BOE-IOT-AIBD and KirinUK are the only teams that incorporate a GAN loss into their training strategy. On the other hand, Team-WVU trains its network only on the pixelbased Charbonnier Loss, and ZZX trains their perceptual solution on L1 and SSIM. Nevertheless, they outperform BOE-IOT-AIBD, which employs a GAN loss. Therefore, strong guidance from a pixel-based loss might be important for such an extreme scaling factor.

Conclusions
This paper presents the AIM 2020 challenge on Video Extreme Super-Resolution. We evaluate the performance in this challenging setting for both high fidelity restoration (track 1) and perceptual quality (track 2). The overall winner Kir-inUK manages to strike the best balance between restoration and perceptual quality. The participating teams provided innovative and diverse solutions to deal with the extreme upscaling factor of 16. Further improvements could be achieved by reducing and ideally removing the notorious flickering artifacts associated with video enhancement in general. On top of that, a more powerful generative setting could be designed for higher perceptual quality in track 2. Quantitative evaluation for perceptual quality still requires more research, especially in the video domain, where temporal consistency is important. We hope this challenge attracts more researchers to enter the area of extreme video superresolution as it offers great opportunities for innovation.

CET CVLab
Title: Video Extreme Super-Resolution using Progressive Wide Activation Net Members: Hrishikesh P S, Densen Puthussery, Jiji C V Affiliations: College of Engineering, Trivandrum, India