Eye Movement Feature Classification for Soccer Expertise Identification in Virtual Reality

Latest research in expertise assessment of soccer players pronounced the importance of perceptual skills. Former research focused either on high experimental control or natural presentation mode. To assess perceptual skills of athletes, in an optimized manner, we captured omnidirectional in-field scenes, showed to 12 expert, 9 intermediate and 13 novice goalkeepers from soccer on virtual reality glasses. All scenes where shown from the same natural goalkeeper perspective and ended after the return pass to the goalkeeper. Based on their responses and gaze behavior we classified their expertise with common machine learning techniques. This pilot study shows promising results for objective classification of goalkeepers expertise based on their gaze behaviour.


Introduction
Several sports related studies on perceptual-cognitive skills have shown the potential of perceptional skills of athletes regarding their contribution to superior performance in sports [1][2][3][4][5][6][7]. The method of choice in research of perceptual-cognitive skills are video based. Observation of perceptual-cognitive skills with video based methods allows to isolate different characteristics to develop a knowledge base that explains certain perception based advantages of athletes.
Research on perceptual-skills has taken advantage of innovations in computer science, i.e. new presentation devices, interaction interfaces or biometric feature recording devices such like eye trackers. In fact, one of the main challenges in sport related research on perceptual-skills, remains the trade-off between experimental control and a natural valid presentation mode, Kredel et. al [8] postulateed in a meta review of over 60 studies from over 40 years of research on natural gaze behaviour.
Larkin et al. [9] concluded based on a review of 25 studies that video based training can enhance perceptual-cognitive performances. One fundamental aspect is a highly natural presentation mode, which leads to pronounced expertise effects in gaze behaviour and decision-making. Mann et al. [10] found moderator effects of the stimulus presentation mode, postulating a relationship between an increased natural presentation mode and increased expertise effects.
For all research on perceptual-cognitive skills, the need of a optimized trade-off between natural presentation mode and experimental control -for comparable results -is of high importance. Ignoring a natural presentation mode prevents the athletes to apply their natural gaze behaviour. Disregarding high experimental control prevents comparable and precise results. Both, Vater et al. [11] and Mann et al. [10] suggest that sports-related perceptual-cognitive skills should be examined by taking care about both sides of the trade-off. A natural environment that mimics the complexity of the task, while -from a scientific perspective -paying particular attention to the level of experimental control.
So far eye tracking studies focused on one side. Either in-field setups with natural presentation mode (field camera) or laboratory setups with high experimental control [12][13][14][15][16][17] were conducted. For optimal research conditions both sides need to be improved. As a new upcoming technology, virtual reality (VR) devices are used more often as stimulus presentation mode and interaction device. Research focused either on photorealistic stereoscopic views of sports environments combined with interaction techniques for natural movements in a virtual reality [18] or on modeling athletes' behaviour to create expertise based adaptive interfaces or training systems. VR has the power to optimize the trade-off and even create synergetical effects. VR can show realistic and immersive environments and by using a built-in eye tracker infer a close to natural gaze behaviour of the users. VR can even replace CAVE systems [19][20][21][22]. There are several other advantages of VR. Bideau et al. [23] summarized these advantages. Their main contribution is to show that interactive and immersive virtual realities can elicit experts responses similar to real-world responses.
Another trend in computer science can help to improve the experimental control and the analysis of the results. With more frequent usage of eye trackers, more accurate, faster and ubiquitous devices, huge amounts of precise data can be generated. Machine learning provides the power to deal with huge amounts of data. In fact, machine learning algorithms typically improve with more data and allow fast, precise and objective reproducible ways for data analysis. Machine learning methods are used in different kinds of eye tracking studies. Especially expertise classification problems can be solved, as shown by Castner et al. [24,25] in dentistry education or expertise identification in microsurgery [26][27][28][29]. Machine learning techniques are the current state-of-the-art for expertise identification and classification. Both, supervised learning algorithms [25,26] and unsupervised methods or deep neural networks [24] have shown their power for this kind of problem solving.
Expertise identification and classification leads to adaptive and personalized designs of systems, i.e. virtual cognitive training systems. The choice of difficulty can be adapted based on the expertise of the user. For higher skilled users, the difficulty of a level can be raised by pointing out less cues. With enough data it is also possible to adapt a training level based on personal deficiencies that were found during expertise identification.
Our focus in this work is in particular to objectively identify and classify expertise based on perceptual-cognitive skills that are represented by eye movements. Further, we are interested in obtaining explainable features, that could explain differences between expertise groups and might not be obviously but found by a feature selection approach. In this work we present a system that is based on photorealistic 360°videos, viewed on VR glasses and a machine learning approach for data analysis. We show techniques to find explainable differences between three groups of expertise in goalkeepers gaze behaviour. This work is meant to be a fundamental work for a machine learning based perceptual-cognitive diagnostic system in virtual reality.

Project description
The HTC Vive is a consumer-grade virtual reality (VR) headset. Gaze can be recorded, through integration of the SMI high speed eye tracker, at 250 Hz. The SteamVR framework is an open-source software that allows to interface common real-time game engines with the VR glasses to display custom virtual environments. We projected omnidirectional 4k footage on the inside of a sphere that envelopes the field of view of the user, which leads to a high immersion and presence into a realistic scene.

Stimulus material
We captured the 360°footage by placing a Insta Pro 360 (360°camera) on the soccer field on the position of the goalkeeper. Members of a german first leagues elite youth academy were playing a 6 (5 field player plus goalkeeper) versus 5 match scenes. Each scene was developed with a training staff team of the german football association (DFB). We took only scenes that have binary decisions.

Participants
We captured data of 12 experts during a DFB youth elite goalkeeper camp. The data comes from german youth elite soccer goalkeepers (U-15 to U-21). The data of 8 intermediates was captured in our laboratory and come from regional league soccer players (semi-professional). Data of 13 novices was either from players of lower leagues or people with less or no experience in soccer.

Procedure
The study was confirmed by the ethics committee of the faculty of economics and social sciences of the university of tuebingen. After signing a consent form to allow the usage of their data we familiarized the participants with the footage. 5 different screenshots and stimuli were played and explained to allow the participant to acclimate to the setup. To learn the decision options we also showed a schematic overview. By doing this, we reduced the number of possible answers (see figure 1, plus "kick out" option). The general procedure is as follows: One of the 26 stimuli is played in the VR glasses. Directly after receiving the last pass (to the goalkeeper), the video stops and a black screen is presented. The participant now has 1,5 seconds time to tell the decision option one wants to make and the color of the ball, which was printed on the last return pass (to force all participants to recognize the last return pass realistically). The second block contains the same 26 stimuli but in a different order. Each decision made on the continuation of a video has a binary rating, as only one decision is counted as 1 (correct). The remaining options are rated as 0 (incorrect). A correct answer is always the the one teammate that stands free.

Method
The raw data of the SMI Eye tracker can be exported from the proprietary BeGaze software as csv files. BeGaze already provides the calculation of different eye movement features based on the raw gaze points. The following section describes the steps that are necessary to train a model based on eye movement features.
Each participant viewed 26 stimuli twice, resulting thus in 52 trials per subject.

Data cleaning
First, we viewed the samples of these 52 trials and checked the confidence measures of the eye tracking device. We removed all trials with less than 75% tracking ratio, as gaze data below this threshold are not reliable. Due to errors in the eye tracking device, not all participants data is available for all trials. Hence, we only used trials, that we consider as valid. The number of trials was still 52, except for 3 participants, that only had 41 valid trials. We checked these remaining trials for data quality of saccades. This data preparation is necessary to remove erroneous and low quality data that come from poor detections of the eye tracking device and do not reflect the correct gaze. Therefore, we investigated invalid samples and removed (1)  (1) Invalid starting position: 0.22% saccades had a start at coordinates (0,0). This is an encoding for an error of the eye tracking device. As amplitude, acceleration, deceleration and velocity are calculated based on the distance from start-to endpoint these calculations result in physiological impossible values, e.g., over 360°saccade amplitudes.
(2) Invalid intra-saccade values: Another error of the eye tracking device is based on the way the saccade amplitude is calculated through the average velocity (equation 1) which is based on the distance of the mean of start and endpoints on a sample-to-sample basis (see equation 2). 3.6% of the saccades had at least one invalid gaze sample and were removed (example see figure 3).

V elocity * EventDuration
(1) On samples 7, 8, 14-16, 18-20 both, the x-and y-signal show zero values and thereby indicate a tracking loss. As the saccade amplitude is based on the average velocity which is calculated on a sample-to-sample formula (2), the velocity from samples 6 to 7, 8 to 9, 13 to 14, 16 to 17,17 to 18, and 20 to 21 extremely increase the average velocity as the distances are high (on average over 2400 px for x-signal and over 1000px for y-signal, which corresponds to a turn of 225°on x-axis and 187,5°on y-axis in the time of 4 ms between two consecutive samples).
There are two interpretations for saccadic amplitude. The first refers to the shortest distance from start to end point of a saccadic movement (i.e., a straight line) and the second describes the total distance traveled along the (potentially curved [30], p.311) trajectory of the saccade. The SMI implementation follows the second definition. We could potentially have interpolated invalid intra-saccade samples instead of completely removing the complete saccade from analysis, however this leads to uncertainties that can affect the amplitude depending on the amount of invalid samples and also does not necessarily represent the true curvature of the saccade. (3) As the velocity increases as a function of the saccade amplitude [31], 4.8% of the saccades were ignored on ground of the restriction of velocities greater than 1000°/s. Similar to extreme velocities, we removed all saccade samples that exceeded the maximum theoretical acceleration and deceleration thresholds. Saccades with longer amplitudes have higher velocity, acceleration and deceleration, but can not exceed the physiological boundaries of 100.000°/s 2 [30]. 3.0% and 4.0% respectively, of all saccades exceeded this limit. As most of the invalid samples had more than one error source, we only removed 5.5 % of the saccades (3.5% of all samples) in total.
After cleaning the data we use the remaining samples to calculate the average, maximum, minimum and standard deviation of the features. This results in 36 individual features. We use those for classifying expertise in the following.

Training
In the following, we refer to expert samples as trials completed by an elite youth player of the DFB goalkeeper camp, intermediate samples as those of regional league players and novice samples as those of amateur players. We built a support vector machine model (SVM) and validated our model in two steps: cross-validation and leave-one-out validation. We trained and evaluated our model in 1000 runs, with both validations. For each run, we trained a model (and validated with cross-validation) with samples of 8 experts, 8 intermediates, and 8 novices samples, and used the samples of the remaining participants to predict their classes (leave-out validation). The experts as well as the intermediates and the novice samples in the validation set were picked randomly for each run.

Sample assignment
We found that the way the samples of the data set are split into training and evaluation set is very important and a participant-wise manner should be applied. By randomly  figure 4). This leads to an unexpected learning behavior which does not necessary classify expertise directly but rather the origin of a sample to a specific participant and thereby indirectly the membership to the participant's expertise level. Which means a model would work perfectly for known participants but is unlikely to work for unseen data. Multiple studies showed that the gaze behavior of humans follows idiosyncratic patterns. Holmqvist et al. [30] show that a large amount of eye tracking measures underlay the participants idiosyncrasy, which also means that the inter-participant differences are much higher than intra-participant differences. A classifier learns a biometric, person-specific measure instead of an expertise representation.

Model building
To find a model which is robust to high data variations, we applied a cross-validation during training. The final model is based on the average of k=50 models, with k = number of folds in the cross-validation. For each model m i , with i ∈ {1, . . . , k}, we use all out-of fold data of the i-th fold to train and evaluate m i with the in-fold data of the i-th fold. The final model is evaluated with a leave-out validation. The cross-validation step during training is independent from the leave-out validation with totally new data (never seen by the model), as information of the cross-validation is used during building and optimizing the model and leave-out validation is just an information provider about the prediction accuracy of the model when using completely new data.

Prediction
With a total of 810 valid samples, equally distributed on expert, intermediate and novice samples, we built a subset of 552 samples for training the model and a subset of 258 samples for evaluation. As each sample represents one trial, our approach here is to

Significant features
Secondly, we had a look at the features themselves and check wether there are differences between the single features according to their class and check for significance level of differences of the features of over 5%. We build a model based on the features that have a significance level of over 5% (table 1, 2 and 3 all white cells, gray cells mean there is no significant difference between the groups).

Most frequent features
In a third approach we reduced the amount of features by running the prediction on all 46 features 1000 times. By taking the most frequent features of the model, we search for a subset of features which prevents the model from overfitting and allows interpretable results that represent the differences between the expertise classes with a minimum  We consider samples as belonging to a smooth pursuit, when the dispersion of the samples is greater than 100 px. As the size of the players in the stimulus varies around 90 pixel + a buffer.

Intra-expert classification
To strengthen the implicit assumption of this paper, that it is possible to distinguish between novices, intermediates and experts based on their gaze behavior, we evaluated our expert data separately by flipping a subset of experts with intermediates. After 100 iterations in which half of the experts where randomly labeled as intermediates, the average classification accuracy was below chance-level, which means the model can not differentiate between experts properly. This strengthens our assumption that the differences between experts are smaller than the differences between experts, intermediates and novice.

Results
We first report the results of the classifiablity test then provide a deeper analysis on the model trained with all features and two models based on certain features obtained through 1) their significance level and 2) their frequency in the all feature model. The classifiability test shows promising results. The binary model is able to distinguish between experts and intermediates with an accuracy of 88.8%. The model has a false negative rate of 1.6% and a false positive rate of 18.6%. This means the binary model predicted two out of 260 samples falsely as class one and 29 samples that are class zero as class one. As the false negative rate is pretty low, the resulting miss rate is only 11.9%. The confusion matrix ( figure 6) shows the overall metrics. The binary model is better in predicting class zero samples than class one samples. The overall accuracy of 88.1% is sufficient to investigate on ternary classification. In the following we show

Accuracy
The differences between the three approaches are barely visible when looking at the median (ALL: 75.08%, MFF: 78.20%, SF: 73.95%), but even greater when comparing the 75th percentile (ALL: 80.99%, MFF: 85.44%, SF: 79.25%). All models show a wider range of accuracy values which means these models might overfit more on some runs and underfit on others. The lower adjacent of all models is higher than chance level (ALL: 53.46%, MFF: 52.93% and SF: 52.41%), which means all models perform better as guessing. As the accuracy is a rough performance metric which only tells about the number of correct predictions (true positives and true negatives), we have a more detailed look into the performance of the methods by comparing the miss rates of the single approaches.

Miss rate
The miss rate is a metric that tells about the rate of wrongly classified samples that belong to class x, but predicted to belong to class y. The ternary models models are better in predicting the membership of samples to class one and class two than to class zero. This results in miss rates that are only little lower than chance level when looking at the median miss rates (All: 28.12%, MFF: 23.81% and SF: 26.80%). The upper adjacent shows a high range of miss rates reaching even values of over 43.19% for the SF-model. The MFF-model has the lowest median miss rate of all three methods with a miss rate of 41.96%.

Recall
Recall tells about the rate of samples being predicted as belonging to class x in relation to the number of samples that really belong to class x. All three models have a median recall of over 70%. In the ternary case, chance level is at 33.33% which means all models have a recall of over two times higher than chance level as the lower adjacent of all three models is higher than 33.33%. The MFF-model median is the highest at 76.18% followed by the SF-model at 73.19% and the ALL-model at 71.87%. Again the MFF-model has the best performance values of all three methods.

Feature explanation
The most frequent features in 100 runs are summarized in table 4. Only the minimum of the saccade duration has p > 0.05. Which means the differences are not statistically significant. All other features show significant differences, which means a Mann-Whitney-U-test discards the null hypothesis that there are no differences with p < 0.05 for each of the features. Looking more closely at the most frequent features and their significant values, it becomes clear that 1) experts (SD = 35.54 ms) as well as novices (SD = 38.86 ms) have a homogeneous gaze behaviour compared to intermediates (SD = 59.72 ms). The lengths of the saccades differ less. However, a fallacy would be to attribute the same viewing behavior to novices and experts -due to the standard deviation and minimum duration of the saccades, which is quite similar for all three (novice: 26 ms, intermediate: 25 ms, expert: 25 ms) -since, for example, the average dispersion of smooth pursuits for novices (622.80 pixels) is 1/3 higher than for experts (399.93 pixels). This means that both groups have similarly long saccades among themselves, but the novices have similarly long saccades and the experts similarly short saccades. Conversely, this means that the experts have longer fixations than the novices and intermediates. In a study, Mann et al. [10] show that experts are overrepresented in fewer but longer fixations, because they have more time to process and absorb information.
Further differences between the groups can be found in velocity of the saccades. On the one hand there is a continuous increase in the maximum speed of the saccades from novices (4166.262°/s 2 ) to intermediates (4770.063°/s 2 ) to experts (5492.287°/s 2 ), which is consistent with the findings of Zwierko et al. [32]. The authors say that the deceleration behaviour can be inferred from different expertise classes. This allows, besides the differences in the distribution of the maximum speed of the saccades (Novizes: 129.294°/s, Intermediates: 136.529°/s, Experts: 157. 74°/s), to conclude that one set of experts have faster saccades, but on the other hand also show a more targeted, strucurated but fast gaze behavior. They are more likely to adapt to the situation. Novices perceive a scene as a random situation and try to look in all directions equally in order to keep the overview.
Further differences between the groups can be found in the velocity of the saccades. On the one hand there is a continuous increase in the maximum deceleration speed of the novices' saccades (4166.26°/s 2 ) to intermediates (4770.06°/s 2 ) to experts (5492.28°/s 2 ), which is in line with the findings of Zwierko et al. [32] who say that the deceleration behaviour can be inferred from different expertise classes. Besides the differences in distribution of the maximum velocity of the saccades (novices: 129.29°/s, intermediates: 136.52°/s, experts: 157.74°/s), this suggests that experts on the one hand have faster saccades, but on the other hand also show a more targeted structured and fast gaze behavior. They adapt themselves more to the situation. Novices perceive a scene as if it were an ordinary situation and try to look in all directions equally in order to maintain an overview.
One observation during the study was that novices often follow the ball with their gaze for a long time. This behavior is less evident among experts. They tend to only look at the ball when it has just been passed or when they themselves are not in play. At these times, the ball can not change its path. This observation is supported by the values of the smooth pursuit dispersion. With 505.031 pixel maximum and 336 pixel minimum, experts have a very narrow window of smooth pursuit lengths. Basically, the maximum smooth pursuit of the experts is less than half as long as the novices (1085.90 pixel) and the minimum smooth pursuits (expert: 399 pixel, intermediate 425 pixel, novices 622 pixel) is still 1/3 shorter than the novices. The intermediates are placed in the middle between the two groups. Again the values are continuously decreasing.

Discussion
Such a setup opens the door for dynamic and online analysis of gaze features based on natural gaze behavior. We are however aware that the small sample size, restricts the conclusions that can be drawn and might lead to debatable results. Another limitation of this work is the restriction to head movement unrelated eye movement features and the absence of a detailed smooth pursuit detection algorithm, which might be important. Therefore in our future work we will implement a event calculation method i.e. based on the work of Agtzidis et al. [33].
This work is meant to be a preliminary work for expertise prediction leading to objective perceptual skill assessment in virtual reality. We show that the study setup with an omnidirectional video source, high speed eye tracker and non-restrictive and realistic virtual environment are promising techniques for optimizing the gap between natural presentation mode and experimental control, and therefore allowing the participants to apply their natural gaze behavior on a realistically mimicked environment. We are aware that the small sample count restricts the meaningfulness of the classification results and to shape a robust model, more samples are needed. But this work strengthens the assumption that there are differences between the gaze behavior of experts, intermediates and novices, and that these differences can be obtain through the mentioned methods. Especially when looking at the values of the most frequent features of the model in detail, the differences are noticeable and in line with latest research. These differences lead to the conclusion, that experts scan their environment in a more structured and faster way than intermediates and novices.

Conclusion and Implications
In this work we present a diagnostic model for eye movement feature classification into expert, intermediate and novice. The model presents a first step in the direction of automatic and dynamic design of levels of a training system in a virtual environment based on personalized user gaze behavior. We show that this kind of data is classifiable with high accuracy and that the mentioned methods are suitable to obtain explainable features of the gaze behaviour of the user. After the binary and ternary classification of expertise, the following step should be a finer grained gradation, which allows, by mapping expertise on a bigger amount of classes, the dynamic manipulation of the difficulty level of an exercise of a training system or game level in virtual environments. Next to a training system for athletes and other professional groups, the difficulty level in a VR game can be dynamically adjusted based on the gaze behavior of the user. In our further work, we plan to expand our data set to more subjects, add more classes, add a physical response mode and focus on research of person-specific, gaze-based expertise weakness detection. Another point is to integrate the model into an online diagnostic system. To use the model online, the gaze signal can be directly drawn online at 250 Hz from the eye tracker by using the provided API of the vendor. Using a multi-threaded system, the data preparation and feature calculation can be done directly online in parallel to data collection. Only the higher level features (e.g. SD) need to be computed when the trial ends and fed as feature vector to the already trained model, to estimate the class of the current trial. As predicting is done by solving a function, the prediction result is supposed to be available few moments after the trial ended. Which is necessary as the prediction is the input for the adaption of the training. This work will be implemented in an online system for realtime gaze based expertise detection in virtual reality systems with an automatic input for the presentation device for dynamic manipulation of the difficulty of the scene.