Meta Transferring for Deblurring

Most previous deblurring methods were built with a generic model trained on blurred images and their sharp counterparts. However, these approaches might have sub-optimal deblurring results due to the domain gap between the training and test sets. This paper proposes a reblur-deblur meta-transferring scheme to realize test-time adaptation without using ground truth for dynamic scene deblurring. Since the ground truth is usually unavailable at inference time in a real-world scenario, we leverage the blurred input video to find and use relatively sharp patches as the pseudo ground truth. Furthermore, we propose a reblurring model to extract the homogenous blur from the blurred input and transfer it to the pseudo-sharps to obtain the corresponding pseudo-blurred patches for meta-learning and test-time adaptation with only a few gradient updates. Extensive experimental results show that our reblur-deblur meta-learning scheme can improve state-of-the-art deblurring models on the DVD, REDS, and RealBlur benchmark datasets.

Introduction
Dynamic scene deblurring aims to recover sharp images from blurred ones caused by camera shakes or moving objects. It is challenging to recover images with such blur since dynamic scene blur is often non-uniform and directional. Using methods with some prior assumptions [4,20,33] or blur uniformity [7] can only achieve limited performance. Existing learning-based methods for single image deblurring [6,12,23,32,34,35,36] or video deblurring [1,8,14,15,18,21,26,29,38,39] could obtain better results by training these models in a supervised manner, learning to deblur images from training with blurred and Figure 1: Illustration of the proposed reblur-deblur meta-transferring scheme. We utilize the reblurring model to generate pseudo-blurred and pseudo-sharp pairs to facilitate metalearning for the test-time adaptation. Note that pseudo-sharp patches are selected from blurred video frames.
sharp image pairs. Although significant progress has been made using these supervisedlearning methods, they did not consider exploiting rich internal information in test data and tend to be sub-optimal during testing. Therefore, we aim to build a domain adaptation strategy that allows test-time adaptation to improve existing deblurring methods. Few deblurring works have been done using meta-learning since ground truth is not accessible during testing. Chi et al. [5] proposed a self-supervised meta-auxiliary learning strategy that meta-trains and meta-tests the model by restoring the input blurred image. However, it could only achieve sub-optimal adaptation since blur self-restoration may not help to deblur. By contrast, we observe that a scene appearing in multiple frames in a video often has different blurring degrees. Some local patches are relatively sharp, whereas some are relatively blurred. Therefore, we propose to leverage these patches to synthesize sharp and blurred pairs to enable meta-learning for dynamic scene deblurring during testing without access to the ground truth like [5] but in a different approach. More specifically, this work proposes a reblur-deblur meta-transferring scheme to generate pseudo-blurred and pseudo-sharp pairs to achieve test-time adaptation in meta-learning, as shown in Fig. 1. Our reblurring model can transfer blurred patterns homogeneous to the task to pseudo-sharp patches selected to synthesize pseudo-blurred patches, which can serve as pseudo-sharp-and-blurred pairs as the support set in meta-learning. Our contributions are three-fold. First, we propose a novel reblur-deblur meta-transferring scheme that can generate pseudo-blurred and pseudo-sharp pairs for meta-learning. Second, the proposed scheme facilitates meta-learning for dynamic scene deblurring without extra training data needed. Third, extensive experimental results show that our method improves the performance of existing deblurring models on various datasets, including DVD [28], REDS [17], and RealBlur-J [ 29,38,39]. The main difference is the number of input frames. Video deblurring utilizes consecutive frames to help deblurring, which can obtain more information than image deblurring. However, most existing methods fail to consider internal information from testing data and ignore the problem of domain gap between training and testing sets, which would be sub-optimal during testing.

Meta learning on low-level vision tasks
Previous low-level vision methods commonly train a generic model using training data without considering the rich internal information in test data. Recently, a few methods developed for low-level vision tasks have been proposed [5,24,27]. Super-resolution (SR) methods [24,27] utilized MAML [9] to efficiently adapt to test images for better SR results, where they used the patch-recurrence property intrinsic to an image itself, downsampling images to generate low-and high-resolution training pairs during testing. Since finding such an intrinsic characteristic from a blurred image is challenging, Chi et al. [5] proposed a meta-auxiliary learning method, where they meta-trained the deblurring model to learn the input blurred image in a self-supervised manner to help deblur. Thus, it can enable the model to adapt to test blurred images without using their ground-truth counterparts. Nevertheless, restoring the input blurred image and deblurring it essentially works in a contradictory fashion, which may only produce sub-optimal results. Unlike meta-auxiliary learning for deblurring proposed in [5] requires attaching additional layers to train the model, our work builds a reblurring model that can transfer blur patterns from relative-blurred to relative-sharp patches. It can synthesize pseudo-blurred and pseudo-sharp pairs for deblurring models without modifying them to facilitate meta-learning.

Reblurring for deblurring.
Some recent deblurring works [3,19,22,37] were proposed to generate additional blurred images to train the models. Zhang et al. [37] used an extra real blurred dataset to produce synthetic blurred images using generative adversarial networks. Park et al. [22] used a recurrent method to produce images with various blurring degrees. However, the generated data for training may not help upon testing. To enable test-time adaptation, Nah et al. [19] utilizes a reblurring loss to constrain deblurring results using a pre-trained reblurring model based on the observation that clean images are hard to reblur. Chen et al. [3] chose to self-supervise a model for deblurring, similar to [5]. In contrast, we leverage the internal information in testing videos to generate domain-aware pseudo-blurred and pseudo-sharp pairs, enabling meta-learning better than [3,5].

Overview
The paper proposes a reblur-deblur meta-transferring scheme for test-time adaptation, composed of meta-training and meta-testing phases.
In meta-training, we consider deblurring each video V i in the training set as a task, where V i contains N blurred frames B n and their sharp counterparts S n (n = {1, 2, ..., N}). Figure 2 depicts the proposed reblur-deblur meta-transferring training scheme, where the inner update trains a deblurring model on the support set generated by the reblurring model, and the Figure 2: Illustration of our reblur-deblur meta-transferring training scheme. In the inner update, we use an adapted reblurring model to generate a domain-aware support set, enabling meta-learning to update the deblurring model. In the outer update, we evaluate the adapted models by query set to obtain meta-learned weights θ RB and θ DB .
outer update trains the deblurring and reblurring models adapted to the query set. Like [5], not needing the ground truth at inference as well, we instead use the proposed reblurring model to generate M pseudo-blurred patchesB Ω m from pseudo-sharp patchesS Ω m selected from {B n } N n=1 . Based on our observation that a scene often has various blurring degrees in multiple frames in a blurred video, we can pick relatively sharp patches in blurred frames to be pseudo-sharp patches. To choose sharp patches, we propose to measure the blurring degree by a self-shift method, and those with the least blur are selected. These M pseudoblurred and pseudo-sharp pairs {B Ω m ,S Ω m } M m=1 are used as the support set. For the query set, we use the whole V i = {B n , S n } N n=1 to train the reblurring and deblurring models. In meta-testing, given any testing video V test i , we generate the support set as in the metatraining phase, not needing the ground truth. Next, we explain how to create the support set for meta-learning in Sec. 3.2 and the architecture of the reblurring model in Sec. 3.3. At last, Sec. 3.4 details the proposed reblur-deblur meta-transferring scheme for test-time adaptation.

Generation of the Support Set
To enable meta-learning for the test-time adaptation, we generate pseudo-blurred and pseudosharp pairs {B Ω m ,S Ω m } M m=1 from each blurred video. These pseudo-sharp patchesS Ω m are fed into a reblurring model to generate pseudo-blurred patchesB Ω m . The support set contains To search for relatively sharp patches in a blurred video, we propose to measure the blurring degree by a self-shift method. Based on our observation, sharp images usually have stronger edges with larger gradients than blurred images. Fig. 3 illustrates the self-shift method, where four images are generated by shifting the input image one pixel to the right, top, top-right, and top-bottom, in four directions. We obtain a self-shift score by averaging four PSNR values calculated by the four shifted images with the input image as the reference. A higher score indicates that the shifted images are similar to their original image ( Fig. 3 (a)), implying it is blurred. In comparison, a lower score represents that the input image is sharp. We randomly select M locations to crop image patches, where the relatively sharp patches are chosen with the lowest shift scores among all the colocated patches in all the input video frames, shown in Fig. 3 (c). Finally, we feed {S Ω m } M m=1 into the reblurring model to generate pseudo-blurred patches {B Ω m } M m=1 to generate the support set. , height, embedded feature height, width, embedded feature width, and channel dimensions, respectively. Inspired by [2], we propose a cross-frame fusion block (CFFB), containing regional self-attention (RSA), cross self-attention (CSA), and local self-attention (LSA) [2]. The CFFB divides F into non-overlapping regional tokens with a window size of r as F ∈ R T ×N×D , where N = H W r 2 and D = C × r 2 denote the number of tokens and dimensions. To reduce the channel dimensions, we use an embedding layer to embed F into F ∈ R T ×N×D , where D = 2C. We then feed F into RSA [2], which performs self-attention on regional tokens to generate F RSA ∈ R T ×N×D , where T is regarded as the number of batches. To utilize cross-frame information, we propose CSA, which considers the center frame patch tokens

Reblurring Model
where W q , W k , and W v ∈ R D ×D denote the embedding weights of the query, key, and value. We concatenate the output features followed by a linear layer with dimensions R 5D ×D to generateF CSA t ∈ R N×D as After CSA, we replace the F RSA t in F RSA withF CSA t to obtain a new F RSA ∈ R T ×N×D , wherẽ F CSA t has fused the neighboring information. Lastly, we feed F RSA and initial input features F into LSA to obtain F LSA ∈ R T ×N×D like [2]. After four CFFBs, the middle frame patch tokens F LSA t ∈ R N×D are fed into a linear layer to increase channel dimensions from D to D and folded into F O ∈ R H ×W ×C followed by a decoder to generate the pseudo-blurred patch B Ω m ∈ R H×W ×3 .
The reblurring model needs to be pre-trained, where we use blurred and sharp image pairs in the training set with the Charbonnier loss [13] and adversarial loss [10] derived from an addional discriminator, i.e., five consecutive sharp frames as input and the middle blurred counterpart as the ground truth. To address the domain gap of the pre-trained reblurring model, we also update the reblurring model and discriminator during testing. Therefore, we generate realistic pseudo pairs {B Ω m ,S Ω m } M m=1 as the support set by transferring blurred patterns at inference to enable meta-learning.

Reblur-Deblur Meta-Transferring
Here, we introduce the meta-learning process for the reblurring and deblurring models. Meta training: We consider deblurring each video V i in the training set as a task, where V i = {B n , S n } N n=1 are available during training. We initialize the reblurring model (parameterized by θ RB ), discriminator (θ D ), and deblurring model (θ DB ) with their pre-trained weights. For each video V i , we first initialize the reblurring, discriminator, and deblurring weightsθ RB i , θ D i , andθ DB i by θ RB , θ D , and θ DB , respectively. As shown in Fig. 2, the meta-process can be split into two stages: one is the inner update to obtain the reblurring and deblurring models using the support set, and the other is the outer update to evaluate the ability of adapted models by the query set.
In the inner update, we first update the reblurring model to generate the support set for the deblurring model. We choose M pseudo-sharp patches {S Ω m } M m=1 from {B n } N n=1 by the self-shift method to generate pseudo-blurred patches {B Ω m } M m=1 from the reblurring model where the five consecutive frame patches from the pseudo-sharp patch are concatenated as the input. To generate realistic and homogenous blurred patterns, we use the discriminator parameterized byθ D i to calculate the adversarial loss L adv . Here, the pseudo-blurred patch B Ω m is considered a faked blurred image, and we select a real blurred patch by the self-shift method with a high score. Additionally, we add a cycle loss based on cycle consistency from the deblurring model f DB (·) parameterized by θ DB as where L is the loss function adopted in the deblurring model. It represents that deblurring the generated pseudo-blurred patch by the reblurring model should obtain its corresponding pseudo-sharp. For updating the reblurring model, we first update the discriminatorθ D i by L adv and then update the reblurring model by the total reblurring loss as L RB = L adv + λ L cycle , where we set λ = 0.01 to focus more on the blur-transferring ability of the disciminator, i.e.,θ where L is the loss function used by the deblurring method, and β is its learning rate.
In the outer update, we verify the adaptability of adapted models for each video V i . We utilize all the blurred and sharp images {B n , S n } N n=1 in V i as the query set to perform metaupdate based on the gradient calculated using the query set as where L char is the Charbonnier loss [13]. Since we regard deblurring on each video V i as a task, the meta-training process is repeated on all the available training videos until the model converges. Through Eq. 7, we can obtain meta-learned weights θ RB and θ DB , more transferable to a new task (video) compared to the original pre-trained weights. Meta testing: Given any testing video V test i = {B n } N n=1 , we take meta-learned reblurring (θ RB ), deblurring (θ DB ) weights and pre-trained discriminator (θ D ) weights as initialization. Similar to the meta-training phase, we choose M pseudo-sharp patches {S Ω m } M m=1 from {B n } N n=1 by the self-shift method and iteratively update the reblurring model using Eq. 5 to obtain the adapted reblurring model (θ RB test ). Next, we use the reblurring model (θ RB test ) to generate the support set to update the deblurring model by Eq. 6. Lastly, we deblur the whole testing video V test i by the adapted deblurring model (θ DB test ). Because our support set is generated by pseudo-blurred and pseudo-sharp patches {B Ω m ,S Ω m } M m=1 , we can enable metalearning on the deblurring task without the ground truth. The detailed training and testing processes are shown in the supplementary. Table 1: Evaluation results on three datasets and four SOTA deblurring models. "Baseline" means the deblurring results obtained using the original models pre-trained on GoPro. "Meta" means the results using "Baseline" with our reblur-deblur meta-transferring scheme.

Experiments
In this section, we evaluate the effectiveness of the proposed method. We first explain datasets and implementation details in Sec. 4.1. We then demonstrate the quantitative and qualitative results in Sec. 4.2. In the end, we conduct ablation studies in Sec. 4.3.

Datasets and Implementation Details
We trained all the compared models on the GoPro [16] training set, containing 22 videos, totally having 2, 103 training blurred and sharp image pairs in both the pre-training and meta-training stages. At the meta-testing time, we use three datasets to demonstrate the effectiveness of the proposed test-time adaptation scheme, including DVD [28] testing set (10 videos, having 1,000 images), REDS [17] validation set (30 videos, having 3,000 images), and RealBlur-J [25] testing set (50 videos, having 980 images). First, we pre-trained the reblurring model on the GoPro training set with a batch size of eight for 1, 000 epochs by Adam optimizer [11], where the initial learning rate is set to 10 −4 , and the decay is 10 −8 with the cosine annealing strategy. We adopted random cropping, flipping, and rotating for data augmentation. Next, for meta-training, we randomly cropped 256 × 256 patches for the support and query sets. Note that the support set contains pseudo-sharp patches selected from these cropped patches and then pseudo-blurred patches generated by the pseudo-sharps using the reblurring model. We set the learning rate α to 10 −6 in the reblurring model and β to 2.5 × 10 −6 in the deblurring model. All methods are tested on an Nvidia A5000 GPU.

Experimental Results
Quantitative Analysis: We employed our scheme on four different state-of-the-art deblurring models, including MIMOUnet+ [6], MPRNet [34], Restormer [35], and CDVD-TSP [21]. The first three models are for image deblurring, and the last is for video deblurring.
In Table 1, we compare the results obtained with or without the proposed reblur-deblur metatransferring scheme. As can be seen, our scheme can improve existing deblurring methods The authors from the universities in Taiwan completed the experiments on the datasets.  by 0.27dB, 0.29dB, and 0.28dB in PSNR on average on the DVD, REDS, and RealBlur-J datasets, respectively, indicating that it achieves the test-time adaptation for better performance.
Qualitative Analysis: Fig. 5 shows visual comparisons of results obtained using the original state-of-the-art methods with or without using the proposed reblur-deblur meta-transferring scheme on public benchmark datasets. As shown, using our meta-transferring scheme restores the blurred content better, indicating that the proposed approach synthesizing images with homogenous blur to each task at inference time for test-time adaptation can effectively boost the deblurring performance.

Ablation Studies
This section analyzes the ablation studies we conducted for the proposed approach. Table 2 shows the effectiveness of different loss terms. Next, we compared meta-testing with finetuning and different numbers of pseudo pairs included in the support set, shown in Fig. 6.
Analyses of the adopted losses in reblurring model: Our approach adopts the adversarial and cycle losses to update the reblurring model, as in Eq. 5. In Table 2, we analyze the effects of using these losses for deblurring on the DVD test set. "Baseline" here denotes the performance using the original models. "Cycle loss" and "GAN loss" mean updating the reblurring model with either the cycle loss or adversarial loss used in the inner update. As observed, using either the cycle loss or GAN loss improves the performance, and adopting both losses leads to more performance gain. Meta-testing and number of pseudo pairs used: Fig. 6 compares our meta-transferring scheme with fine-tuning, which means directly fine-tuning the models with the pseudo pairs instead of meta-learning. It shows that adopting our scheme outperforms merely fine-tuning for all the state-of-the-art methods on the benchmark datasets. We can also observe that including more pseudo pairs in the support set can increase the performance gain but with diminishing marginal benefits since selecting more pseudo-sharp patches may include ones with much blur and harm the performance. In our experiment, M is set to 20 for DVD and RealBlur-J and 10 for REDS. We use more pairs in DVD [28] and RealBlur-J [25] since they have less blur for finding good pseudo-sharp patches to help test-time adaptation. Computation cost and limitation: For meta-testing, each video needs to run the inner update for ten iterations on the REDS dataset and twenty on the DVD and RealBlur-J datasets. Take MIMO-UNet+ as an example. We need additional 0.1 seconds on average for inferencing one image. A limitation of the proposed scheme is that when we deal with a strongly blurred video containing few relative-sharp patches, the reblur-deblur meta-transferring process would be less effective, possibly leading to poor performance.

Conclusion
This paper proposed a novel reblur-deblur meta-transferring scheme to facilitate meta-learning without ground truth at inference time. We combine a reblurring model to generate pseudoblurred patches from selected pseudo-sharp patches as the support set, enabling meta-learning to update deblurring models during testing. Extensive experiments have shown that our method can improve existing deblurring models on benchmark datasets, including DVD, REDS, and RealBlur-J.


A Comparative Study of Confidence Calibration in Deep Learning: From Computer Vision to Medical Imaging

Although deep learning prediction models have been successful in the discrimination of different classes, they can often suffer from poor calibration across challenging domains including healthcare. Moreover, the long-tail distribution poses great challenges in deep learning classification problems including clinical disease prediction. There are approaches proposed recently to calibrate deep prediction in computer vision, but there are no studies found to demonstrate how the representative models work in different challenging contexts. In this paper, we bridge the confidence calibration from computer vision to medical imaging with a comparative study of four high-impact calibration models. Our studies are conducted in different contexts (natural image classification and lung cancer risk estimation) including in balanced vs. imbalanced training sets and in computer vision vs. medical imaging. Our results support key findings: (1) We achieve new conclusions which are not studied under different learning contexts, e.g., combining two calibration models that both mitigate the overconfident prediction can lead to under-confident prediction, and simpler calibration models from the computer vision domain tend to be more generalizable to medical imaging. (2) We highlight the gap between general computer vision tasks and medical imaging prediction, e.g., calibration methods ideal for general computer vision tasks may in fact damage the calibration of medical imaging prediction. (3) We also reinforce previous conclusions in natural image classification settings. We believe that this study has merits to guide readers to choose calibration models and understand gaps between general computer vision and medical imaging domains.

Introduction
Confidence calibration measures the agreement between the predicted probability and the true class prevalence (expected accuracy) [13]. Confidence calibration is important across domains, especially healthcare, for trustworthy prediction [6,21,3]. A predictive model should know when its prediction is inaccurate. For example, in computer-aided diagnosis system, the model should inform its users, doctors, of the uncertainty around its prediction. 1 Our implementation is publicly available at https://github.com/MASILab/ComparativeCalibration. ? This research was supported by NSF CAREER 1452485, R01 EB017230 and R01 CA253923. This study was supported in part by U01 CA196405 to Massion.
The one-stage approach of mixup [45] includes a regularization technique for deep learning that linearly interpolates the input and label. Mukhoti et al. propose the use of focal loss [29] can train better calibrated deep learning models from theoretical and empirical perspectives [32]. Researchers [42,20] showed that training with mixup can significantly improve the model calibration. Some methods improving model calibration is under the uncertainty estimation context. Based on the assumption that a well-calibrated model should be accurate when it is certain (with high confidence) and be uncertain (with low confidence) when it is inaccurate, Krishnan et al. [24] introduced the di erentiable accuracy versus uncertainty calibration (AvUC) loss function which was shown to optimize models that have well-calibrated uncertainties. Karandikar et al. [23] introduce di erentiable soft calibration objectives (e.g., soft version of AvUC) to improve calibration.
There are also calibration models in a two-stage framework. One of the earliest works on calibrating deep learning models is discussed in [13], which o ers calibration insights into neural network learning and provide a simple but e ective way (i.e., temperature scaling) to calibrate models. Label smoothing [39] is another widely applied regularization technique that helps avoid overfitting during training. Muller et al. [33] introduced the use of label smoothing as a method to increase the confidence calibration. Zhong et al. [47] incorporate a label-aware smoothing in the calibration model for long-tail learning in the second stage. The techniques of Krishnan et al. [24] and Karandikar et al. [23] have been applied in the second stage as well.
Even though promising calibration results have been demonstrated in previous arts, most of them are under standardized (e.g., large-scale training, balanced training) or simulated classification tasks. The imbalanced-training challenges especially in clinical contexts have not been well explored. While approaches for calibrating deep learning models have been proposed recently, there has not been a comprehensive study of calibration models applied in di erent challenging contexts, especially clinical disease diagnosis. As a representative example, lung cancer has the highest mortality across all cancers [38,37]. Deep learning models have been widely applied to diagnose lung cancer using computed tomography (CT) images with satisfactory discrimination [2,28,11,12], however, confidence calibration of the prediction models is lacked.
In this work, we conduct a comparative study of four leading one-and two-stage calibration models in di erent contexts and domains for a total of 12 experiments for comparison. Specifically we compare calibration models when used in a balanced multi-class classification task with large-scale training set (i.e., original CIFAR10 [25]), a simulated imbalanced training classification context with large-scale datasets (i.e., long tail CIFAR10 following [47]). and a clinical prediction task (i.e., lung cancer diagnosis) with CT images including internal-and external-validation with three data cohorts. Our study seeks to help readers to understand big picture of calibration models with four leading approaches and demonstrate their varying e ectiveness across di erent domains.
The rest of manuscript are organized as follows: Section 2 introduces and discusses the selected four calibration methods across Stage 1 and 2, including their high impact notes. Section 4 shows our experiments in computer vision including balanced and imbalanced training. Our clinical prediction task (i.e., lung cancer diagnosis) are demonstrated in Section 5. Finally, we conclude and discuss our work with take-home messages in Section 6.

Calibration Methods
In this section, we introduce the used representative approaches for calibration in this paper. Our motivation to choose the approaches is (1) well-known and widely used, (2) cover multiple categories such as one-vs. two-stages. With the 4 selected approaches and the baseline cross-entropy loss, we have 12 combinations in total for comparison.
Baseline: cross-entropy loss (CEL). CEL is also known as negative log likelihood (NLL), which measures the quality of a probabilistic model [15]. CEL is commonly used in deep learning for classification [27], and is defined as followed: where p k is the predicted probability of class k. y k = 1 for the correct class and y k = 0 for the rest classes.
Focal Loss [29]. Focal loss was first proposed for dense object detection to deal with class imbalance problems. Let's denote the predicted probability corresponding to target class y t as p t , which ranges from 0 to 1. To prevent easily classified negatives dominating the loss and gradient, focal loss includes a modulating factor (1 * p t ) to cross entropy loss, where is a positive number as the exponent. In practice, the focal loss can be adopted in the ↵-balanced form: where ↵ t is the hyper-parameter associated to class y t . When p t is close to 1, which indicates the sample can be easily classified, the modulating factor (1 * p t ) would be small. Thus, compare with CEL, Focal loss has less penalty on easy samples. This reduces the overconfidence of training.
Label-Aware Smoothing [33]. Label smoothing (LS) [39,33] applies a transformation to ground truth labels such that y LS k = y k (1 * ↵) + ↵_K. where ↵ is the smoothing factor to relief the overconfidence in training. To improve calibration for long-tailed recognition, Zhong et al. [47] proposed the Label-aware Smoothing (LAS) where the smoothing factor of class y depends on class number N y , as below: where large N y has a larger smoothing factor f (N y ). Thus, the ground truth label in LAS is no longer a one-hot vector. Consider a 3-class classification task with training sizes of 10000, 5000, 1000 for class 1, 2, and 3 respectively. The one-hot representation of ground truth for CEL is class 0: Under this situation, the head classes (i.e., the classes with more samples) has larger regularization to avoid overfitting. Following [47], the LAS is applied in the second stage.
Mixup [45]. Mixup strategy is motivated by the Vicinal Risk Minimization [7]. As in [45], the vicinal points ( É x, É y) are generated according to the following rules: where x i and x j are randomly selected sample pairs. y i and y j are related one-hot labels of x i and x j . À [0, 1] and Ì beta(↵, ↵) for ↵ Ì (0, ÿ). The mixup strategy assumes that linear interpolation of inputs should linear interpolation of outputs. Mixup training has been shown to contribute to the robustness and generation of models [46].
Temperature Scaling [13]. Platt scaling is a parametric approach for model calibration, which learns two parameters a,b to achieve a new prediction q i = (a z i +b) given the original prediction z i . Temperature scaling is a simple and straightforward extension of Platt scaling, which only has one learnable parameter. In the multi-class classification context, given the logit vector z i , the new prediction is obtained as follows: where the parameter T is called temperature and SM is the Softmax function. Since both plat scaling and temperature scaling do not change the ranking of predictions, so they do not a ect the discrimination of the overall model.

Evaluation Methods
In this section, we introduce the quantitative metrics and illustration diagrams to evaluate the model's calibration as well as discrimination.
Expected Calibration Error (ECE). ECE [34] is one of the most popular metrics to measure calibration performance. There are two main steps to compute ECE: (1) dividing the prediction value space into equalspace bins, (2) calculating the weighted average of the di erence of accuracy and confidence. The ECE is defined as: where the n is the number of samples, B m is the m-th bin. Accuracy (ACC). Accuracy is a metric that measures the fraction of model predictions get right.
where Ç y i and y i is the predicted label and ground truth for sample i, respectively. N is the number of samples. ACC is used for evaluating the discrimination of multi-class classification.
Area Under the ROC Curve (AUC). ROC curve (receiver operating characteristic curve) is a figure showing the performance of a prediction/classification model at all thresholds [10]. The curve plots true positive rate as y-axis and false positive rate as the x-axis. The AUC represents the area of the twodimensional region under the whole ROC curve from (0,0) to (1,1). AUC is used for evaluating the discrimination of binary classification.
Reliability Diagrams. Reliability diagrams [14] are the figures of the predicted probabilities (confidence) and the observed frequency (expected frequency), which shows the frequency of predicted probabilities happened in practice (i.e., confidence calibration). Examples can be found in Figure 1, 2 and 4.
Histograms of Predicted Probability. The histogram shows the distribution of predicted probability of the corresponding class. The histograms indicate how confident (high or low) of the model in the population. Integrated with reliability diagrams, the overconfidence or under-confidence can be inferred. These are depicted in Figure 3 and 5.

Data Introduction
The CIFAR10 consists of 60,000 tiny images evenly from 10 classes. There are originally 50,000 samples in the training set and 10,000 samples in the test set. We further split the original training set into training/validation splits with the ratio of 9:1. The validation set is used for tuning hyper-parameters and selecting epoch number.
The original CIFAR10 [25] dataset is balanced, and we term it as CIFAR10-Ori in this chapter. Following the setting of [47], we create a variant of CIFAR10-Ori with long-tail distribution called CIFAR10-LT with the imbalance factor (IF) 0.01 for the training set. The number of samples in class i is defined as N i = N ori IF i_9 , where N ori is the sample number in CIFAR10-Ori. The validation and test sets of CIFAR10-LT keep the same as CIFAR10-Ori.

Implementation
We utilize the ResNet-32 [16] as our backbone network, as was previously studied in [47]. The training process has up to 2 stages. For Stage 1, the feature extraction model ResNet-32 is trained from scratch with Focal loss or cross-entropy loss (CEL), and we also have the comparison of with mixup and without mixup. For Stage 2, the feature extraction model is fixed. In this case, only the classifier (last layer) was trained, or the logits were scaled. We include the Label-aware Smoothing (LAS) [47] and Temperature Scaling (TS) [13] for comparison. The experiments are conducted on both CIFAR10-Ori and CIFAR10-LT.
We use the SGD optimizer with a momentum of 0.9 and a base learning rate (BLR) of 0.1. In the first five epochs, the learning rate is BLR < epoch index _5. The learning rate is divided by 10 at 150th and 250th epoch. The max number of epochs is 300. The batch size is set to 128 and the weight decay is 5e-4. Our experiments are conducted with PyTorch 1.6. The hyper-parameter ↵ in mixup is set to 1 for CIFAR10-LT and 0.2 for CIFAR10-Ori. The hyper-parameters in focal loss are ↵ = 1, = 2. The number of bins when computing the ECE is 15.

Results and Their Analyses
Herein, we first present the easier setting of balanced training (i.e., CIFAR10-Ori) and then the harder setting of imbalanced training (i.e., CIFAR10-LT).
Balanced CIFAR10-Ori Results. Table 1, Figure 2, and Figure 3 show the experimental results on CIFAR10-Ori. From the results of Stage 1, the discrimination performance (ACC) was similar across four settings. Compared with the baseline CEL, the strategy of mixup slightly improved the ACC while improve calibration performance with a clear margin (ECE: 3.60%->2.86%). In contrast, replacing the CEL with Focal loss slightly decreased the ACC and improved the ECE with a large margin (ECE: 3.60%->1.22%). When adding both the mixup and Focal loss, the calibration performance became distinctly worse.   Table 1. Note that if the Accuracy is larger than the Expected Accuracy, the part of light blue bar would be covered by dark blue bar.  Table 1. The histogram shows how confident the prediction is, and when integrated with reliability diagrams, we can inference if the prediction is overconfident or underconfident.   Table 2.
Note that if the Accuracy is larger than the Expected Accuracy, the part of light blue bar would be covered by dark blue bar.

Frequency (y-axis)
Predicted Confidence (x-axis)  Table 2. The histogram shows how confident the prediction is, and when integrated with reliability diagrams, we can inference if the prediction is overconfident or underconfident. When applying the temperature scaling (TS) in Stage 2 for recalibration, the class discrimination performance did not change, while the confidence calibration saw a large improvement across all the four Stage-1 settings. When using the label aware smoothing (LAS) [47], the ACCs were slightly changed. Generally, LAS improved calibration performance in Stage 2, but less so than TS.
In Stage-1 comparison, as indicated in Figure 2 and 3, when trained by CEL, the predicted risks are highly concentrated near 1 (i.e., highly confident). Here, mixup and focal loss both individually reduced the overconfidence of CEL, which improves calibration. However, when mixup and focal loss were applied together, the prediction became underconfident, which hurt calibration.
As for class discrimination, there is no clear impact found (ACC changes < 1%) from mixup and focal loss. As expected, temperature scaling did not change discrimination across all experiments as it only scales the logits and will not change the ranking of predicted risks from di erent classes. As the second stage post-processing methods, both LAS and temperature scaling work well in confidence re-calibration.
In conclusion, all the selected approaches improved calibration except the combination of focal loss + mixup in the setting of training a classifier on a large-scale balanced dataset.

Imbalanced CIFAR10-LT Results.
The experiments of training with CIFAR10-LT are shown in Table 2. Compared with CIFAR10-Ori, the discrimination performance was lower by approximately Ì 20% ACC.
In Stage 1, mixup greatly improved the calibration performance (ECE) and slightly improved the discrimination performance (ACC). The Focal loss improved the CEL of the baseline model but decreased the ECE when used with mixup (i.e., ECE: CEL + mixup (1.72) vs. Focal Loss + mixup (11.48)). The LAS improved calibration and discrimination except in the setting of CEL + mixup.

Imbalanced CIFAR10-LT Analyses.
The limited diversity of classes in the imbalanced training setting is widely known to limit a model's generalizability, decreased discrimination, and possibly lead to overfitting of overrepresented classes.
Like the analyses of CIFAR10-Ori, mixup, Focal loss and the second stage model LAS compensate for model overconfidence, which improved calibration performance. While all the selected approaches positively impacted performance in this simulated long-tail classification, LAS resulted in the best discrimination and the TS resulted in the best calibration.
In contrast to the CIFAR10-Ori setting, "Focal + mixup" had better calibration performance than the baseline "CEL". However, individually they out performed the combination of the two due to underconfidence predictions when combining Focal loss and mixup.  Table 2 Test results of training with CIFAR10-LT. (~) represents larger number is better. (ö) represents smaller number is better.

Data Introduction
We have included three datasets in this experiment: the national lung screening trial (NLST) [35], and two cohorts, UPMC and UCD, from MCL [1].
NLST. The NLST is a randomized controlled nested case:control clinical trial for lung cancer detection that was designed to evaluate whether the screening with CT reduces mortality compared to screening with X-ray.About 54,000 participants across 33 centers were enrolled between 2002 and 2004. It represents one of the largest lung CT datasets available for research. This study used subjects that 1) have a tissue-based diagnosis, and 2) have a diagnosis within 1 years of the last scan for cancer case. There are in total 606 cancer subjects out of 5344 subjects. Similar to [12], the selected subjects are all high-risk patients (all received biopsies) and the distinction between cancer / non-cancer in our cohort is more challenging than a cancer classification task on the entire NLST cohort. Our dataset was randomly split into five even folds. Four folds were used for training and one-fold for validation.
UPMC and UCD. Unlike the NLST which is a nested case:control screening cohort with low cancer rate, the UPMC and UCD datasets are matched case:control studies, with a mix of screening and incidentally detected lung cancers, and have much higher cancer rate (Ì 0.5). There are 78 cancer cases out of 155 patients in UPMC and 52 cancer cases out of 96 patients in UCD.

Implementation
We have included one calibration method (i.e., focal loss) in Stage 1 and two calibration methods (i.e., temperature scaling and LAS) in Stage 2 for a total of six combinations for comparison. Because the mixup strategy is designed for single image classification and the lung cancer diagnosis experiment is a multiinstance learning problem, we did not evaluate mixup in the lung cancer diagnosis experiment.
The neural network backbone is motivated by the image path of our previous work [12], as shown in Figure 6.
We first apply the preprocessing steps and nodule detection model of [44] to raw CT image data. Suggested by [28], the top five confidence nodule proposals are enough to cover all the nodules. To speed the training/test process, we use 2D images rather than 3D. Axial/coronal/sagittal directions of the nodule proposal are formulated as three channel data with the dimension of (3 ù 128 ù 128). We use a ResNet-18 backbone to extract image features of each nodule proposal. Then, the image features of the five nodule proposals are transferred to a single image feature vector with an attention-based multi-instance learning layer [18].
We use the SGD optimizer with the momentum of 0.9 and a learning rate of 0.005. The max number of epochs is 100 and final model is used for testing as we empirically notice that the training is converged after 100th epoch in our setting. The batch size is set to 128 and the weight decay is 1e-4. Our experiments are conducted with PyTorch 1.6. The hyper-parameters in focal loss are: ↵ = 1, = 2. The number of bins when computing the ECE is 10.   Table 3 Internal-validation results on NLST (%). TS represents temperature scaling and LAS represents label-aware smoothing.
Stage  Table 5 External validation results on UPMC. TS represents temperature scaling and LAS represents label-aware smoothing.

Experimental Results
The results on NLST, UCD, and UPMC are shown Table 3, Table 4, and Table 5, respectively. The Focal loss has improved discrimination over CEL, which has observed across all three datasets (e.g., 91.53% vs. 90.71% in NLST, 87.20% vs. 83.50% in UPMC). The temperature scaling generally improves the calibration performance across the losses of CEL and Focal. In most situations, the LAS does not improve the calibration performance but can improve the discrimination minorly.

Result Analyses
In general, Temperature scaling, as a simple post-processing, improved the calibration performance across our experiments. Interestingly, approaches that worked well for computer vision tasks failed to improve performance by the same margin in medical prediction. For example, LAS greatly improved both class discrimination and confidence calibration in CIFAR10-LT, while it resulted in a decrease in discrimination and calibration of lung cancer diagnosis. We believe the di erences between natural image classification and medical imaging detailed below may explain the failure of some calibration models in lung cancer diagnosis.
First, the lung cancer diagnosis is based on a multi-instance learning (MIL) pipeline. The training label is assigned on the whole CT image (i.e., the "sample" of MIL) level rather than individual pulmonary nodules (i.e., the "instance" of MIL) level. Developing models on less granular, image-level labels is known to be more challenging than doing so on fine-grain, object-level labels.
Second, most calibration models in computer vision are developed and evaluated under well-defined and standardized contexts. For example, "cat" and "bird" images from CIFAR10 can be easily distinguished if high-level features are extracted e ectively. While in lung cancer diagnosis, malignant and benign nodules can be hard to distinguished even by radiologists. These di culties yield poor discrimination and calibration in clinical contexts.
Third, the number of training samples in CIFAR10 experiments is larger than that of lung cancer diagnosis. As deep learning models are data hungry, small training data size can negatively impact discrimination and calibration.

Discussion
Confidence calibration is important for prediction models, especially in the context of clinical diagnosis. It is an essential consideration for the ethical discussions surrounding implementation of deep learning in clinical decision making. Accurate confidence calibration is an essential to informing physicians when a prediction is uncertain so that they may question the prediction in the context of their clinical judgment. Even though deep learning has been successful in class discrimination and calibration studies have been conducted on standard or simulated datasets, the calibration on the challenging clinical decision-making contexts with deep learning models is understudied.
In this paper, we conducted a comparative study on confidence calibration and class discrimination with four high-impact calibration approaches. Those four approaches include one-stage and two-stage methods, and we have 12 method combinations for comparison. We evaluated these approaches in three settings: balanced large-scale training set on natural image classification (i.e., CIFAR10-Ori), large-scale imbalanced training (i.e., CIFAR10-LT), and lung cancer diagnosis with cross-and external-validations using three cohorts.
Our work is the first to evaluate popular learning methods for domain-based calibration di erences between natural and medical images. We have found the following: • Balanced training with CEL leads to overconfident predictions large calibration error (as shown in Figure 3(a), the prediction prevalence near 1 is high). Both the mixup and focal loss improves the calibration performance by reducing the overconfidence of the predictions (as shown in Figure  3(b), 3(c)). However, combining focal loss and mixup resulted in under confident predictions which damaged the calibration (see Figure 3(d)).
• Stage-2 model achieves similar calibration performance (Table 1 and 2) and risk distribution ( Figure  3 and 5) across 4 stage-1 models (this is more obvious in temperature scaling).
• The calibration performance of both focal loss and label-aware smoothing was worse on medical images compared to natural images, suggesting di erent learning contexts (e.g., large-scale v.s. smaller-scale training sets, classical learning v.s. multi-instance learning, fundamental di erence in image structure) can lead to di erent performance of calibration methods. We have conducted discussions (in Section 5.4) to analyze the gaps between computer vision and medical imaging domains.
• The simple approach of temperature scaling improved the calibration performance in both domains, suggesting that simpler approaches may be more robust to domain di erences. Overall, in CIFAR10, LAS can improve discrimination and calibration in long-tail settings. However, LAS can decrease the model's discrimination and calibration.
• In agreement with previous work [13,47,32,42], temperature scaling maintained the discrimination and improved calibration in all scenarios, including balanced-vs. imbalanced, nature image vs. medical image et cetera; focal loss, mixup, and label-aware smoothing improve the calibration of natural image classification in both balanced and imbalanced setting; and label-aware smoothing further greatly improved the discrimination in CIFAR10-LT.
In summary, we address the confidence calibration challenge of prediction models with empirical validations and are the first to highlight the gaps of calibration between computer vision and clinical diagnosis. We believe our study has insights to better understand calibration models and knowledge transfer between domains. Our code is publicly available at: https://github.com/MASILab/ComparativeCalibration.


Eye-Tracking in Immersive Virtual Reality for Education: A Review of the Current Progress and Applications

The concept of using eye-tracking in virtual reality for education has been researched in various fields over the past years. With this review, we aim to discuss the recent advancements and applications in this area, explain the technological aspects, highlight the advantages of this approach and inspire interest in the field. Eye-tracking has already been used in science for many decades and now has been substantially reinforced by the addition of virtual and augmented reality technologies. The first part of the review is a general overview of eye-tracking concepts, technical parts, and their applications. In the second part, the focus shifted toward the application of eye-tracking in virtual reality. The third part, first the description of the recently emerged concept of eye-tracking in virtual reality is given, followed by the current applications to education and studying, which has not been thoroughly described before. We describe the main findings, technological aspects, and advantages of this approach.

INTRODUCTION
Eye-tracking is a technology that has for a long time been used in the humanitarian, language (Kaushanskaya and Marian, 2007;Suvorov, 2015;Aryadoust and Ang, 2019), medical research (Pernice et al., 2014;Rigby et al., 2020), and quite recently it has found its way into the field of user experience design. One of the most well-known and cited examples of the use of eye-tracking is given in a study from Nielsen (2006), where people were browsing the internet and looking at the search engine results, while their gaze was tracked. It has been shown that their attention is scattered on the page in the shape of a triangle or letter F. This research has then been followed up and proved later in other studies (Brunyé et al., 2019). Active use of eye-tracking technology over recent years has facilitated the delivery of information to the consumer through the media. Eyetracking has been also applied to biomedical research and showed good results in the rehabilitation measures after ischemic brain damage (Krupinski et al., 2006;Cameirão et al., 2016;Faria et al., 2016;Peshkovskaya et al., 2017;Maggio et al., 2019a,c,d).
The successful use of eye-tracking technology in medical treatment has influenced the research to turn toward the use of eye-tracking with healthy patients. Thus, when studying the decisionmaking process of an individual related to the need for social interactions in a virtual environment, it was demonstrated that eye-tracking technologies can be used as a forecasting tool. The frequency and duration of fixation, the peak velocity, and amplitude of saccades can be used to identify and eliminate transient states of uncertainty when making perceptual decisions (Maggio et al., 2019b), as well as to assess the complexity of a problem when it becomes too complicated due to many details (Brunyé and Gardony, 2017). In predicting decision making, gaze fixation, fixation duration, and the number and duration of visits are of great importance. Sustainable combinations of these parameters form various starring strategies (Spinks and Mortimer, 2016).
Moreover, with the help of eye-tracking, it is possible to identify higher nervous functions such as emotions, regret, and disappointment (Nakhaeizadeh et al., 2020), as well as memorization processes (Bault et al., 2016). It has also been shown that eye-tracking technologies can be used to control decision-making (Helbing et al., 2020) for example by increasing its value (Fridman et al., 2018). A recent study (Smith and Krajbich, 2019) showed the importance of the multiplicative model, which states that greater attention to options when making a choice, has higher influences on choices. Another important study has demonstrated (Pärnamets et al., 2015) that the number of eye movements between informationally important objects in the context of the current task can be used as an indicator, which has an inverse relationship to the quality of solving a cognitive task.
Research in the field of eye movements and eye trajectories analyses in the context of virtual three-dimensional space are fairly new. Some works are aimed at studying the emotional behavior of a person interacting with objects in a virtual space, in particular studies (Rosa et al., 2017;Reichenberger et al., 2020), implicit eye movements and their significance were analyzed with regards to the influence of attention on the emotional learning processes. Some researchers are studying the possibility of forming interactive interfaces for influencing the virtual environment by changing the trajectory of the gaze (Paletta et al., 2020). Research in the field of education is also beginning to appear, for example, in Piotrowski and Nowosielski (2020), the possibility of constructing a predictive system for students' skill level was studied by analyzing their trajectory of gaze in a virtual environment. Part of the research is also aimed at analyzing the usability of virtual reality (VR) interfaces by studying the gaze trajectory fixed in virtual three-dimensional space (Orlosky et al., 2019). Thus, over the past 5 years, sufficient experience has been gained in using eye-tracking as a technology for predicting, evaluating, and managing the solution of emotional and cognitive tasks.
The concept and applications of eye-tracking in education has already been described in several review papers. Rappa et al. (2019) have investigated different ways in which eye tracking can be implemented into the learning process by providing a detailed scoping review with measures and emerging patterns of eye tracking in learning environments. A recent review on applications of eye tracking into the digital learning environment (Gorbunovs, 2021) summarized the concept of eye tracking and current challenges in the field that have to be considered when applying eye tracking into the learning environment, such as the cultural background, gender, age, and literacy of the prospective users. Another new interesting work published in the special issue dedicated to the Research of Visual Perception in Educational Science (Kaakinen, 2021) indicates issues of the eye tracking research related to the operationalization of the theoretical constructs. More specifically, the paper explores the basic concepts of eye tracking, the measure of students' engagement as well as the measure of teachers' expertise. Moreover, the paper goes on to explore eye-movement as an indicator of studentteacher interaction. The second major point raised in the paper, is the analyses of eye tracking data.
Although there has been extensive research and many review papers on the topic of eye-tracking technology in various education areas, the concept of eye-tracking in virtual reality for educational purposes is a rather new direction that has only appeared in the last few years. Compared to previous review papers (Lai et al., 2013;Brunyé et al., 2014;Tien et al., 2014;Asan and Yang, 2015;Alemdag and Cagiltay, 2018;Weiss et al., 2018;Aryadoust and Ang, 2019;Rappa et al., 2019;Radianti et al., 2020), the current review combines both eye-tracking descriptions in various fields, focusing on education, and virtual reality, as well as their combination, which allows a novel perception of the existing techniques and technologies aimed at the improvement of the education process. With the existing knowledge, it is important to highlight the current progress in the area of eye-tracking in virtual reality for educational purposes. The study focuses on the existing applications in various fields of education as well as on proposing new ones. Over the past years, the most reported area of education with regards to eye tracking is medical education, many examples of which are mentioned in the current review. However, one of the aims of the article is to highlight the importance of research and implementation of technology in other areas of natural sciences education.
The article consists of three parts: a brief overview of eyetracking techniques and application fields with the highlights of major developments, the implementation of eye-tracking in virtual reality in different fields of science, and the description of ongoing new research in the field of eye-tracking in virtual reality and education.

Literature Review
For the current review, the Scopus Electronic Library had been used to conduct the literature analysis. Various search queries were used, which included combinations of keywords in the Title, Abstract, and Keywords fields. For searching the articles that are related to various aspects of eye-tracking technology these combinations of keywords have been used: "eye tracking, " "eye movement, " "eye tracking" AND "behavioral science, " "eye tracking" AND "education, " "eye tracking" AND "medicine, " "eye tracking" AND "design, " "eye tracking" AND "virtual reality, " "eye tracking" AND "virtual reality" AND "education, " "eye tracking" AND "virtual reality" AND "studying, " "eye tracking" AND "virtual reality" AND "teaching, " "eye tracking" AND "virtual reality" AND "learning." For the 4th section of the current review, the initial search query included "eye tracking" AND "virtual reality" and resulted in a total of 1,632 papers. Out of them, 644 were obtained by limiting the presence of the word "learning." There were 409-64% articles of Computer Science subject area, 204-32% of Engineering, 129-20% of Medicine, 105-16% of Mathematics, and 99-15% of Neuroscience. Further filtering of the 1,632 articles by the word "training" gave 527 documents, where the ratio of subject areas was: Computer Science -302 (55%) articles, Engineering -179 (32%), Medicine -138 (24%), Mathematics -82 (14%), and Neuroscience -62 (10%). The first search that included the word "teaching" resulted in 124 documents within the following subject areas: 62-48% articles of Computer Science, 38-29% of Medicine, 37-28% of Engineering, 26-19% of Social Sciences, and 17-12% of Mathematics. Also, 93 articles were discovered by including the word "studying" into the search query within these 1,632 documents: 64% of them were dedicated to Computer Science (61 articles), 21% to Engineering (21 articles), 18% to Medicine (19 articles), 15% to Neuroscience (16 articles), and 10% to Mathematics (11 articles). By including the word "education" into the search within initial 1,632 articles, 307 papers had been identified. The priority areas covered by these documents were: Computer Science (59% -188 articles), Engineering (30% -97 articles), Medicine (23% -78 articles), Mathematics (13% -47 articles), and Social Sciences (11% -40 articles). A more detailed overview of the papers used in the current review, organized by sections of the review, is given in Table 1.

EYE-TRACKING CONCEPTS AND APPLICATIONS
When it comes to human cognition, eye-tracking is a tool that has extensively been used for various research (Rappa et al., 2019). The first eye tracker device came about in the early 20th century, and it consisted of special contact lenses that were used with a pointer attached to them. This technique was then changed and optimized by using light beams and recording their reflection on film instead (Huey, 1968;Duchowski, 2003). In the 1960's modern eye-tracking approach had been developed, since then it has been further studied and improved (Buswell, 1937;Tatler et al., 2010). It is safe to say that the methods used for detecting eye movements have become significantly more accurate in recent years. At the moment, the most used methods involve video systems with computer vision techniques usage (Yarbus, 1967). Recent technological advances such as high-quality cameras have facilitated the use of eye-tracking devices of small size. These small devices can now be combined with portable "smart" glasses or a VR headset to provide accurate measures of eye movement and large data sets. Eyetracking methods have been described in a variety of different experimental works (Hansen and Ji, 2010). Due to technological advances, eye-tracking is now widely and efficiently used in human cognition research.

Technological Aspects
The modern eye tracking devices can track the gaze by using an array of infrared or near-infrared light sources and cameras (Deubel and Schneider, 1996). The concept behind the majority of currently used system is based on an array of light sources illuminating the eye, which then produces a corneal reflection.
The relationship between the produced reflection and the center of the eye pupil is then recorded, so that the vectors linking the eye position to the location in the perceived world can be calculated (Yarbus, 1967). The calculated viewpoint in space moves with the movement of eyes. Several hardware configurations of eye trackers are available on the market at the moment. These include systems that provide head stabilization, the so called «chin up» systems, as well as the remote ones that can only account for a limited amount of head movement. Moreover, there are also mobile eye-wear-based systems, which are the most modern of all. All of these hardware types have different advantages and disadvantages that are related to spatial (i.e., tracking) accuracy, tracking speed, mobility, portability, availability, and cost Funke et al., 2016).
Several indicators of visual behavior play an important role in understanding the interpretive processes behind gaze. Such indicators are, for example, position, motion, numeracy, and latency indicators (Deubel and Schneider, 1996). The fixations are the instantaneous pauses of the eyes that are measured in milliseconds. Saccades are the movements of the eye between successive fixations (Deubel and Schneider, 1996). Since the eye is constantly moving between fixation points, eye trackers are able to collect large amounts of data. Motion measurements provide quantitative analyses of the eye movement patterns in space during saccades. Examples of such measurements include the distance between successive saccades and saccade speed. Cartesian coordinate space is used to measure position of the gaze. The frequency of fixations and saccades is identified by the numerical measures of the eyes during scene perception. It is for example recorded, how these numerical values changes the position of the subject changes. The latency measures are then used to estimate the temporal dynamics of fixations and saccades, the example of these measures can include the duration of fixations and of saccades.
Once the fixations and sequential saccades have been recorded, the analysis is then carried out. The factors that are taken into consideration when analyzing are the internal and external states. The first includes states like frustration, uncertainty, anxiety, etc., while the latter is, for example, the organization of the stimulus. After explaining the technical side of the eye-tracking process, it is important to consider the areas of science and research where this technology has been most successfully applied in recent years.

Eye-Tracking in Behavioral Science
One of the fields where eye-tracking has been extensively used and studied over the last decade is without a doubt behavioral and cognitive science. There have been many studies that present eye-trackers for tracking changes in cognition (Lai et al., 2013;Holmqvist and Andersson, 2017). In one such study, it has been shown that the total viewing time and fixation time in areas corresponding to non-cooperative solutions are associated with the general level of participants' cooperation. The increase in the total viewing time and fixation time on the areas that correspond to non-cooperative solutions happens due to the preference for non-cooperation by participants and a decrease in the overall level of cooperation. Therefore, the viewing  (9) Psychology (6) Computer science (6) Neuroscience (6) Arts and humanities (4) Nursing (3) Agricultural and biological sciences (3) Biochemistry, genetics, and molecular biology (3) Engineering (3) Business, management and accounting (1) Decision sciences (1) Mathematics (1) Health professions (1) Augmented reality, Immersive virtual reality, and 3D interfaces  (5) Mathematics (1) Neuroscience (4) Psychology (3) Medicine (2) --Process-based Outcomes-based Frontiers in Education | www.frontiersin.org All three types of evaluation methods time clearly correlates with the decision-making process. The number of fixations on group attributes is associated with group identity, but it does not immediately lead to cooperative behavior (Maggio et al., 2019d). Eye-tracking technology has been a useful tool in research that is related to social attention. For example, it has been shown that results from certain experimental studies correlate with measures of social impairment and with autism symptom severity. It has been found that reduced attention to social stimuli or increased attention to non-social stimuli is correlated with behavioral measures of autism (Bird et al., 2011;Chawarska et al., 2012;Eckstein et al., 2017). Face processing, as well as language skills, are also significantly correlated with measures of social attention. A strong association between face processing skills and attention to faces has been reported for children (Shic et al., 2011), more studies suggested that attention to a speaker's mouth and eyes could be a predictive measure of how fast the words are recognized among children with ASD (Parish- Morris et al., 2013). Moreover, when the processing of such social information as eye and mouth movement is atypical, it is correlated with difficulties in language learning or social impairment. An increase in attention toward the mouth has been associated with increased social adaptation (Tenenbaum et al., 2014) and communicative competence (Klin et al., 2002). These results suggest that eye-tracking methods are promising for studying social attention in ASD and can be successfully used in studies with children.

Eye-Tracking in Education
A major application and focus of this review is the use of eyetracking in education (Norbury et al., 2009;Jarodzka et al., 2017Jarodzka et al., , 2021Strohmaier et al., 2020). Over the past decades, there has been substantial research on the fundamental concepts and applications of eye tracking in education. There are numerous great review papers on the topic some of which will be mentioned in this subsection.
In the work by Sun et al. (2018), it is argued that the application of the newest information technologies to traditional teaching methods can not only increase the value of technologies but also improve the progress in learning and effectively integrate various areas of education. A special educational software package has been developed based on eye-tracking technology. By analyzing data on the movement of students' eyes, teachers can improve the quality of teaching by adapting the teaching structure, while students can focus more on their interests and develop personalized education strategies. The study, Lai et al. (2013) aims to show how eye-tracking technology has been applied to learning research. A total of 81 articles, including 113 studies, were selected from the Social Sciences Citation Index database from 2000 to 2012. The tendencies of eye-tracking technology in the educational process are studied under several specific topics, e.g., patterns of information processing and patterns of decision-making. This study concludes that the eyetracking method provides a promising channel for researchers to link learning outcomes with cognitive processes.
A work by Alemdag and Cagiltay (2018) is a systematic review that is dedicated to the cognitive processes in multimedia learning studying with relevant variables through eye-tracking technology. In the review, 52 articles with 58 studies were analyzed. The results showed that there is a growing interest in the use of eye-tracking technology in multimedia learning research. Eye movement measurements allowed the two authors to conclude the cognitive processes of choice, organization, and integration. Also, one of the results was that the multimedia content itself, individual differences, and emotions were potential factors that could affect eye movement measurements. Eyetracking has proved itself to be an important tool that provides specific spatial and temporal measures to monitor, measure, analyze, and evaluate educational processes (McNamara and Jain, 2019). The topic of eye tracking in education is further developed with specific application to medicine as the next subsection.

Eye Tracking in Medicine
Eye-tracking has for a number of years been widely implemented in medical research, which has been reported and reviewed in many papers over the past decade (Krupinski, 2005;Krupinski et al., 2013;Bond et al., 2014;Tien et al., 2014;Asan and Yang, 2015;Rigby et al., 2020). For example, eye-trackers have been used alongside machine learning to improve the diagnostics, and also to predict diagnostics errors before they can even occur. With the use of eye trackers, automatic cueing or feedback can be provided to learners during image examination (Voisin et al., 2013). Such automatic feedback is realized by parsing medical images into diagnostically relevant and non-relevant regions (ROIs), while using expert annotation or automated machine vision techniques (Brunyé et al., 2014;Mercan et al., 2016;Nagarkar et al., 2016). Once these regions are known to the system that is used for eye tracking, fixations on the important regions are recorded. The recordings are then used to study the spatial distribution of attention over a digital image and the time of fixations. Once there is enough data available, it can be fed into the machine learning algorithm to provide automated diagnostics. Another example of the use of eye-tracking is demonstrated in the work by Tien et al. (2014), the review includes studies describing the use of eyetracking when performing, learning, or evaluating completion of a task or acquiring skills for surgeons. The reviewed literature demonstrates the ability of vision tracking to provide reliable quantitative data as an objective assessment tool with potential application in surgical training to improve performance. Vision tracking remains a promising area of research with the possibility of further implementation in the assessment of surgical skills.
Another significant area of medical research that is associated with the use of eye-tracking devices is related to rehabilitation. It has been shown that the combination of virtual reality technology and eye-tracking, improves rehabilitation of such brain functions as attention, memory, motor and visual-spatial abilities, and speech when compared to traditional therapy methods (Rigby et al., 2020). Moreover, it has been reported that virtual reality training was able to stimulate the patients' motivation. The universal ability, which is not tied to the nature and specifics of the disease, to increase motivation and demonstrate better results in the rehabilitation of lost cognitive functions has been reported. This universal property was demonstrated for brain injuries (Barlinn et al., 2016) and neurodegenerative diseases (Krupinski et al., 2006;Maggio et al., 2019a,c). This finding allowed to reduce the total hospitalization time of patients while increasing the duration of rehabilitation training (Maggio et al., 2019a). These few examples demonstrate the importance of this technology in the medical field and the applications to medical education will be discussed further.
Research in the field of medical education suggests that the pattern of eye movement of students changes with regard to diagnostics as they progress in their studies. More specifically, the eye movements become more rapid and move toward the importance for diagnostic regions (Jarodzka et al., 2010). Hence, the eye movement of students becomes more and more like that of an expert, this process can be time and analyzed. It has also been suggested that this process can be accelerated by showing students video material of the expert's eye movement, this method is called eye-movement modeling examples (EMMEs) (Frank and Danoff, 2007).
The technology and application of EMMEs has been rapidly developing and many interesting studies have already been produced. EMMEs include video of expert eye movement as well as audio description of the action by the expert (van Gog et al., 2009;Jarodzka et al., 2013). The scientific basis behind this method lies in neuroscience, where it has been shown that the brain can mirror actions, when another person's action is observed. Such a response is known as a "mirror system" and can be integrated into the learning process (Calvo-Merino et al., 2005, 2006. This technology provides students with the unique opportunity to learn from the experts in the field without them being physically available, this brings the home-or distant education to a new level of quality, which is particularly relevant during the pandemic outbreak (Jarodzka et al., 2010). EMMEs method had been used outside the medical education field, one of the studies reported that the use of EMMEs improved the ability of notice aircraft inspectors to detect aircraft faults (Sadasivan et al., 2005). The same has been observed for circuit board inspectors (Stein and Brennan, 2004;Nalanagula et al., 2006). Moreover, EMMEs accelerated the speed of debugging for the software engineers (Mason et al., 2015). Reports show that with the use of EMMEs students can become better readers and solve logical tasks such as puzzles faster (Velichkovsky, 1995).
When the gaze is analyzed with the use of eye-trackers, in the medical diagnostic field it is seen that a sequence of saccades and fixation over a medical image appears. In one study, radiographers viewed the eye movements of either fellow novices or experts during the learning process and then interpreted a chest X-ray (Litchfield et al., 2010). Interestingly, it has been shown that the ability to notice and locate pulmonary conditions has improved in comparison to "free-search" not only from observing the experts but also from observing novices.
In a recent study, medical students observed a video of child epilepsy cases. The video was played in three different settings, in the first one the expert was narrating the video with voice, this was a control video sample. In the second, the eye-movement of the expert was traced onto the video by a small circle on the images. In the third video, the eye movements of the expert were also presented, but the area of the image, which the expert didn't focus on, was blurred out. The results of this study showed an increase in the diagnostic of students after viewing the third video and no change for the first or the second. Therefore, specific viewing conditions can facilitate the use of a "mirror" system in the brain and enhance the learning process (Jarodzka et al., 2012).
Alongside applications in medical education and research, eye tracking and VR can also play a part in assessing the standards of teaching in the medical field. New frameworks for those studying and working in healthcare are being developed by the international accreditation establishment to ensure the highest level of professionals. An example of such a framework is Competency-based medical education (CBME), which is aimed at ensuring that healthcare workers have high expertise that isn't just certified on paper but also proven by practice (Aggarwal and Darzi, 2006). In recent years CBME is already being implemented by including new standards of teaching, assessment, and curriculum into medical education and practice (Simpson et al., 2002;Frank and Danoff, 2007;Swing, 2007;Nasca et al., 2012;Holmboe et al., 2016). Due to these changes, we need new technology to facilitate the assessment procedure and eye-tracking in VR can surely become one of the major tools in this process.

Eye Tracking in Design
Variety of studies reported the use of eye tracking in design, for example in graphic design and other computer-based visual evaluation methods, which are used to measure the distribution of visual attention (Poole and Ball, 2005;Hollander et al., 2019). In a study from 2016, eye-tracking was used to assess computer-based visual tools, which help decision-making to investigate ecosystem services (Klein et al., 2016). It has been suggested that the ability of users to understand ecosystem services can be improved through the use of such decision support systems. Other studies have analyzed the how people can perceive landscapes, using the data from eye trackers such as fixation time, saccade amplitude, etc. (De Lucio et al., 1996;Dupont et al., 2013;Potocka, 2013). The opportunities, as well as challenges for the use of eye-tracking in the fields of cartography, geographic information science, spatial cognitions, etc., have recently been reported in a broad review (Kiefer et al., 2017). Eye-tracking has also been investigated in the built environment, for example, a study of a contextual guidance model with a Bayesian framework was used to predict regions of gaze fixation of people while they search for objects in space (e.g., pedestrian paintings on the street) (Torralba et al., 2006). Similar work has been done by Ehinger et al. (2009) with an addition of the pedestrians being present or absent in the picture. Eye-tracking in the built environment has also been used to find out about visual preferences that people have with regard to the general objects in public spaces (Noland et al., 2017). The effectiveness of GIS software was studied with eye-tracking when the subjects performed orientation tasks with the help of GIS (Kudelka and Dobesova, 2015). This study provided a useful insight into the way people were orientated, and the subjects that were successful in the self-localization task were found to spend more visual attention on objects that provided helpful clues. Recently, realtime eye-tracking systems have been implemented to increase the efficiency and quality of interactive graphics applications as well as large-scale display systems (Cheng and Pulo, 2003;Bergeron et al., 2015;Saxena et al., 2016;Celine et al., 2018;Weiss et al., 2018;Merali et al., 2019;Pottle, 2019;Rutkowski et al., 2019;Ustun et al., 2020).

EYE-TRACKING IN VIRTUAL REALITY
The natural development of technology and research in the field of eye-tracking had led to the combination of eye tracking with a virtual reality tool (McNamara and Jain, 2019). VR is a powerful tool that can change the way we work and relax; it can also transform learning techniques in the near future. New applications of VR are invented every day, so it is safe to say that it could be present in our daily life in a relatively short time. The extent of VR research is rather large and recently many virtual reality systems with eye-tracking have emerged. A vast part of VR research is aimed at improving user experience and reducing usability issues, and it is believed that eye-tracking technology can be of assistance in this task (Clay et al., 2019).
Essentially, eye-tracking gives the ability to identify what the user's gaze is focusing on in the virtual reality environment (VRe). Moreover, VR can be used to change the focus of attention if it will increase the positive outcome of the task. There are techniques to draw attention to certain things in the VRe, which can be used if needed. The success of these techniques can be constantly checked since the gaze of the user is tracked in realtime. The use of eye-tracking in VR can be a helpful asset, not only improving the work of various applications but also identifying the disadvantages of some VRe. In VR, full-body motion tracking can be used, so the environment can react to the user's movement, action, and gaze.
In comparison to real-world eye-tracking, the one in VR has the advantage of the easier definition of the regions that the user had looked at. These regions of interest can also be identified in time and reconstructed. The experimental setup with eyetracking in virtual reality is much more flexible and promising for many fields of science since it can be thoroughly controlled. One can control the data collection, environmental settings, and make the stimuli more natural for the user, therefore enhancing the possibilities of research. It is especially useful in research that focuses on human cognition and behavior. It has been reported that such eye-movement signals as pupil diameter can be used for emotional recognition, and therefore there is a correlation between various emotions and pupil size (Hess and Polt, 1960). Research has provided a comparison between eye-tracking in VRe emotional recognition and the classic EEG approach. This research suggests that although the eye-tracking classification method was not as efficient as EEG, it still had statistically good enough results to be considered a useful tool for this task (Zheng et al., 2020).
Various implementations of VR and eye-tracking have been numerously reported in the field of computer science. Some research includes very detailed technical reports of VR and eyetracking implementations. For example, a review paper by Clay et al. (2019) gives a step-by-step explanation of the available hardware and techniques used to implement eye tracking into the VR set. Other studies have been dedicated to the development of VR devices and the improvement of existing hardware. There is a particularly interesting study, which assesses what technical requirements a VR set needs to have in order to generate a wellknown immersive effect of being "in reality, " or as researchers refer to it, in "presence" (Radianti et al., 2020).
A lot of recent VR research is being conducted in the area of computer gaming. The ability to use eye-gaze in the play has for a long time been a question of investigation and a desired feature for the gaming industry. This topic was given further rise with the launch of VR headsets with built-in eye-tracking, such as Vive, FOVE, and other devices available on the consumer market (Fove Inc, 2017; High Tech Computer Corporation HTC Vive Pro Eye, 2019). Apart from using gaze as an instrument of gameplay, eyetracking can also collect data on where the attention of the gamer is scattered in 3D VRe, therefore, an adaptive game mechanism can create reactive content in the gameplay. One of the papers has shown that the use of gaze to interact with remote objects in the VRe is much faster than the use of hands (Tanriverdi and Jacob, 2000), which could be used as a great advantage for shooting or racing games or any other applications where special attention is paid to speed and aiming quality.
Gaze can also be used as means of non-verbal communication, especially useful in collaborative VR environments. There has been research showing that the use of VR can regulate the interaction between people in this process (Gergle et al., 2013). The integration of eye movement in avatar interactions is actively researched with the use of eye-tracking (Hußmann and Oechsner, 2017).
In the area of medical research, VR technology has been studied and integrated into the educational process. An example of this would be the study of nurse education in a collaborative immersive system (Weiss et al., 2018), or another research on the medical training in a virtual hospital and medical professional training (Saxena et al., 2016;Pottle, 2019). VR technology has been reported and researched in dental medicine, such as simulated cavities removal exercise for dental students in VR (Celine et al., 2018) or a surgical education system where finger tracking is used to show the students the location of the fingers and the exact movements of the expert's fingers during surgery (Merali et al., 2019). Another use of VR in medicine is rehabilitation, where the technology is used on patients. Many papers regarding rehabilitation have been reported so far, an example is a VR-based therapy for vestibular problems (Bergeron et al., 2015), VR breathing exercises for people with Chronic Obstructive Pulmonary Disease (Rutkowski et al., 2019). VR has been used to visualize the body and actually move through the neural tissue; this feature has been used in medical research (Ustun et al., 2020). In recent years VR with eye-tracking has been appearing in the medical field. It has proven itself especially useful in the area of medical education and will be described below.
Here we have described the major areas of eye-tracking in VR research which has been rapidly developing in the last years. With the development of this scientific area, more and more research emerge on the advantages, and importance of VR in educational programs. This will be discussed in more detail in the next section.

EYE-TRACKING IN VIRTUAL REALITY FOR EDUCATION
A concept that has been the major interest of this review is eyetracking in VR, which can be used in education to enhance the learning process and to assess the knowledge of students. This idea has been reported in some literature over the past decade, especially with regards to medical education, however, the field is still evolving and the wide use in the other fields is yet to be seen (Aggarwal and Darzi, 2006). Here, we describe the findings of the aforementioned papers and suggest ways in which the technology can be applied to the assessment of knowledge in the classroom. An immersive education environment has the functionality to carry out assessment procedures with minimal distraction. The knowledge level of the taken course as well as individual aspects of the student, such as cognitive abilities, and achievements can be taken into account. Therefore the learning trajectories of the students can be personalized to enable the best possible result. A particular planned study with a focus on eye-tracking in a virtual learning environment is described in a paper by Schlechtinger (2020), where the author argues that the literature has provided enough initial evidence for the use of eye-tracking in the context of learning. This planned study will evaluate how well eye-tracking works when it comes to detecting objects that cause excessive cognitive load in virtual reality conditions. Some research has recently been conducted in the area of knowledge assessments, for example, students were asked to read a text or a fragment of text being in the virtual reality environment (Pilgrim and Pilgrim, 2016). It has been stated that their knowledge in a particular field can be judged by data collected from eye-trackers, more specifically, the concentration on specific words and expressions during the task (Mason et al., 2015).
While eye-tracking has already been used for years in medical education and training, it is argued that eye-tracking in VR will provide even more opportunities to enhance the learning process. One of the main findings in this field shows that there is a difference in the way experts and novices move their eyes (Jarodzka et al., 2010). It has been mentioned that the use of eyetracking in medical education can perhaps decrease the time it takes a novice to become an expert by accessing gaze patterns.
The available research suggests that vision tracking in desktop VR in LCD display can be a useful tool for developing diagnostic skills in medical students (Litchfield et al., 2010). It is well known that the process of medical education and training is a long multi-step process, therefore, using eye-tracking in VR alongside conventional teaching techniques might be able to increase the efficiency and accelerate the training.
With the successful and effective implementation of eyetracking in virtual reality into medical education and training, it makes sense to interpolate these findings into other fields of education. There is also the potential of implementing eyetracking technologies in virtual reality to engineering education . Kang et al. (2020) describe the so-called Deepwater Horizon operation, where VR allows the assessment of the awareness of trainees and operators in a non-dangerous environment. One of the unobtrusive and viable methods for assessing situation awareness can be the use of eye movements, in particular, time-ordered visual scanning trajectories. Based on the presented data, it can be argued that a field of science education, which usually involves a lot of lab work, would greatly benefit from the novelty of eye-tracking in virtual reality. The personalization of learning trajectories in chemistry education can be achieved in a variety of ways.
The implementation of virtual reality can create an environment where students learn each topic at their own pace. For example, a recent technology report has shown how chemistry education can be enhanced with the use of VR (Maksimenko et al., 2021). Furthermore, eye-tracking could be used to collect data regarding the progress of a student on a chemical topic (Muna and Bahit, 2020;Vandenplas et al., 2021). It can be anticipated that the better knowledge students have, the less time they will spend looking at incorrect answers and the quicker they will concentrate their gaze on the correct one in tests. The same assessment can be done with regards to chemical exercises. Naturally, some students will learn faster than others, and eye-tracking could be a good measure of this process. The educational process can then be personalized, more on this will be specified in the outlook. The attention of the students can be measured as described by Yoshimura et al. (2019) where the study shows the development of attention-restoring visual signals for displaying when gaze tracking detects that students' attention is shifting from important objects since in educational virtual reality it is important to solve the problems of students' inattention to the presented content . Experiments to compare different signals and their parameters to assess the effectiveness and trade-offs, as well as to assess the impact of eye tracking are proposed. Eye-tracking is used both to detect inattention and to control the appearance and location of signals.

OUTLOOK FOR EYE-TRACKING IN VIRTUAL REALITY FOR EDUCATION
As eye trackers are becoming cheaper, more portable, easier to use, and generally more available to the consumers, research on principled methods for using eye-tracking for competency assessment is expected to increase (Al-Moteri et al., 2017). However, the implementation of eye-tracking in VR into the educational setting is a new concept of educational science, therefore, there are many directions for future research that we would like to point out.
First of all, more experiments in the educational environment are to be conducted with the use of eye-tracking in VR, not only in the aforementioned areas of medical education, but also other natural sciences. A major advantage of VR in the educational field is the ability to combine classic laboratory experience while eliminating dangers that are associated with it for school and university students. Therefore, the experimental work is necessary in order to establish the extent to which the learning process is inhibited in such an environment.
The wider use of the technology will provide more data sets that are crucial for understanding the effectiveness of such educational approaches. Moreover, large data sets can be analyzed to optimize the experience. They are also needed in order to establish a stronger correlation between cognition and education, and therefore ensure qualitative assessment for students when using eye-tracking. Thus, analyzing the data from the eyetrackers is another vital direction of the research.
However, the methods to be used when analyzing the data are not established. Thus, a major research direction is the development of techniques to evaluate the effectiveness of eye-tracking in education. This should be done by using both quantitative and qualitative research methods and assessing the knowledge of students, increased level of skill, overall learning experience as well as the downsides of the approach. For example, a possible method for assessment of engagement can be done by the implementation of a machine-based algorithm, as suggested for eye-tracking in education (Goldberg et al., 2021).
Eye-tracking can facilitate the transition to personalized education, where eye-trackers in VR education applications are to measure cognitive load, fatigue, tiredness, and concentration level of individual students. The development of methods that will be used in order to effectively convert data from eye-tracking devices into the quantitative assessment of students' involvement is to be a major part of the research. Further research can potentially show how personalized education will effect cheating in tests and exams while in the classroom, as well as anxiety and peer pressure that motivate students to avoid extra questions on the topic.
It is important to investigate the drawbacks and limitations of the approach with respect to education but also the technical drawbacks. For example, one drawback can be that eye-trackers need to be individually calibrated, which is tedious and timeconsuming in an experimental lab environment. Another is the movement of the headset which would lead to inaccurate data produced by the eye-tracker. Therefore, the position of the headset needs to be fixed before the activity, sharp and fast movements need to be avoided. Some of the problems related to physical discomfort can be solved by making a series of short experiments and having periods of rest or providing students with an active and engaging task to distract from physical discomfort. However, further research needs to be conducted with respect to the aforementioned limitations, motion sickness, and others that arise in the process of technology implementation.
Another fascinating research direction is the use of a multimethod approach, where eye-tracking can be combined with others by which eye-tracking is combined with other physical and cognitive measures in order to provide a more detailed insight into the learning process. An example of such a measure can be a head pose (Ballenghein et al., 2020;Goldberg et al., 2021). It has already been described that eye-tracking in VR can be used to also test the expertise of teachers (Seidel et al., 2021) or increase it, as well as to assess student-teacher engagement (Haataja et al., 2021). We suggest that more detailed research should be done in these directions.
Together with many exciting ideas and advances, eye-tracking in VR also has a fair number of drawbacks. However, we suggest that exploring these technologies alongside each other, will facilitate and perhaps accelerate the overcoming of these problems. And inevitably it will bring these technological advances to many classrooms across the globe.

CONCLUSION
In conclusion, the current review has provided firstly an overview of the areas where eye-tracking technology has been researched and implemented over the past decades. A specific focus has been dedicated to eye-tracking in education and the areas of application. Secondly, it shows that VR technology has been widely applied in many fields of scientific research over the past years and the interest in it continues to grow. It can be observed that both these technologies are sophisticated and well researched, moreover, we argue that now they can be used together and more extensively applied to the area of education.
Implementation of eye-tracking in VR provides new interesting approaches for studying the attention and motivation of students, possibly accelerating and making the education more efficient as well as a tool for assessment. The ability to use VR with different environments, model and control every aspect of the process, makes it an indispensable educational tool. In this review, we provide a possible trajectory for the development and application of these technologies in the classroom.
It is worth noting that the development of new methods for using eye-tracking in VR for education is especially relevant in the current situation, with many countries worldwide switching to distant teaching and the educational system trying to adapt to the changes. The methods of eye-tracking in VR implementation with regards to hardware and software have been numerously reported, therefore this technology can soon become available in day-to-day life.
However, there are certain disadvantages to this technology, a major one of which being motion sickness and visual discomfort which appears while using VR for a longer time. It can prevent students from using the technology to its full potential or even using it at all. Nonetheless, these obstacles will be surpassed with the help of further research in the area.
Overall, we believe that eye-tracking in combination with VR presents a powerful tool that can change the way we perceive education and greatly expand the potential of modern educational programs. This combination has already been successfully applied in the field of medical education, therefore we suggest that more people from different fields should pay attention to this technology and consider the possibility of its implementation into the educational process.

AUTHOR CONTRIBUTIONS
MK, NM, and MM conducted conceptualization and contributed to writing-review and editing. MK contributed to project administration and validation. MM contributed to writingoriginal draft. MM and NM contributed to later drafts of the manuscript.

FUNDING
This work was supported by the Russian Science Foundation, grant no. 20-78-10135.