## Classification of Cognitive Load and Expertise for Adaptive Simulation using Deep Multitask Learning

Pritam Sarkar , Kyle Ross , Aaron J. Ruberto 1 1 2 , Dirk Rodenburg , Paul Hungler , Ali Etemad 3 4 1 1 Department of Electrical and Computer Engineering 2 Departments of Emergency Medicine and Critical Care Medicine 3 Faculty of Engineering and Applied Science 4 Department of Chemical Engineering Queen's University, Kingston, Canada { pritam.sarkar, 12kjr1, 13ajr3, djr08, paul.hungler, ali.etemad } @queensu.ca

Abstract -Simulations are a pedagogical means of enabling a risk-free way for healthcare practitioners to learn, maintain, or enhance their knowledge and skills. Such simulations should provide an optimum amount of cognitive load to the learner and be tailored to their levels of expertise. However, most current simulations are a one-type-fits-all tool used to train different learners regardless of their existing skills, expertise, and ability to handle cognitive load. To address this problem, we propose an end-to-end framework for a trauma simulation that actively classifies a participant's level of cognitive load and expertise for the development of a dynamically adaptive simulation. To facilitate this solution, trauma simulations were developed for the collection of electrocardiogram (ECG) signals of both novice and expert practitioners. A multitask deep neural network was developed to utilize this data and classify high and low cognitive load, as well as expert and novice participants. A leave-one-subject-out (LOSO) validation was used to evaluate the effectiveness of our model, achieving an accuracy of 89 4% . and 96 6% . for classification of cognitive load and expertise, respectively.

Index Terms -Multitask Learning, Deep Neural Network, Cognitive Load, Classification of Expertise, ECG, Wearable Device.

## I. INTRODUCTION

Simulation has been shown to be a highly effective pedagogical strategy and has been widely adopted within the healthcare industry [1], [2]. Not only can simulation be used to aid novice healthcare professionals in becoming experts it can also help in the continual development of fundamental skills [3], [4]. For trauma medicine, a fundamental skill required for responders is the management of their cognitive load.

Exceeding cognitive capacity has been shown to significantly degrade medical performance [5]-[7]. Cognitive load has been found to be inversely correlated with level of expertise [8]. To become expert trauma responders, novice trainees need a way to learn to deal with intense cognitive load without any risks to the patient.

Cognitive load is comprised of mental load and mental effort. Mental load is imposed based on the amount of information given whereas mental effort is the mental capacity that must be allocated to the information [9]. Experts are able

This work was supported by the Innovation for Defence Excellence and Security (IDEaS) program, Canada.

to ignore extraneous details and information decreasing both their mental load and mental capacity. This, in turn, reduces the cognitive load of experts compared to novices [8].

It has previously been shown that learning is impaired when a task exceeds the learner's cognitive load or when the learner's cognitive load is exceedingly low [10], [11]. If simulations do not match the expertise and cognitive load of the participant, the simulations cannot achieve the desired learning outcomes [12]. Consequently, to increase efficacy, simulation environments should match the level of expertise and cognitive load of each learner.

Biometric measures have been shown to indicate cognitive load [13]. Variations in cognitive load can be captured with metrics quantified from ECG measurements [14]. Therefore, since the link between cognitive load and expertise [15] has previously been established, we hypothesize that ECG can be utilized for accurate classification of expertise. To the best of our knowledge, no work has been done on attempting to classify the level of expertise of healthcare practitioners utilizing ECG for training simulations.

In this paper, we create a novel framework for an adaptive simulation for training trauma responders, capable of dynamically adapting to the cognitive load and the level of expertise of the learner (see Figure 1). In our proposed framework, augmented reality (AR) enhancements will be utilized to modulate certain aspects of the simulation based on the cognitive and mental state of the user. To support this framework, we designed and performed two sets of trauma simulations for the acquisition of cognitive load and expertise data from trauma responders. The ECG data were collected using a wearable device, along with quantitative self-reported values of cognitive load. Expertise levels were designated based on past training and education history. Next, collected ECG data was pre-processed and time and frequency domain features were extracted. We then developed a deep multitask neural network capable of accurately classifying high vs. low cognitive load as well as expert vs. novice participants simultaneously. A leave-one-subject-out (LOSO) validation was used to evaluate the effectiveness of our model on new participants. Our proposed architecture showed great performance when compared to other works in the field, achieving an accuracy

of 89 4% . and 96 6% . for classification of cognitive load and expertise, respectively.

The rest of this paper is organized as follows. In Section II, we discuss the related work on classification of cognitive load as well as other forms of affect using bio-signals. Next, in Section III, we describe the experiment and simulation setup, data collection protocol, feature extraction techniques, and our proposed deep neural network used for the classification of cognitive load and expertise. In Section IV, we present the results of our method and compare the performance to other works in the field. In this Section, we also analyze the results of the two classified attributes with respect to one another, followed by a discussion on the limitations and potential future work. Lastly, Section V concludes the paper.

## II. BACKGROUND

Affective computing using physiological signals has received considerable attention in recent years [16]-[19]. While this field has mostly focused on applications such as driver monitoring [17], [20], mental health [18], [21], and multimedia quality of experience [22], [23], adaptive and dynamic training and simulation is another area where affective computing has great potential. For example, in [24], a probabilistic model for monitoring the user's level of engagement during a computerbased educational game was proposed. By analyzing ECG, electromyogram (EMG), and galvanic skin response (GSR) through a dynamic decision network, it was illustrated that bio-signals can be used to generate interactions tailored to both the user's learning and emotional state.

The types of affective states often studied in the field range from emotions [16], [18], [21] such as stress and happiness/sadness to cognitive load [17], [20] and level of expertise [25]-[27]. While analysis of emotions conform to the classical arousal vs. valence model, cognitive load and expertise analysis are not necessarily placed on the circumplex of emotions [28]. Instead, these states are often correlated with how people experience and handle arousal and valence, and therefore, the same type of approaches used for emotional states can be used for such studies.

Analysis of emotions using bio-signals for stress detection during driving was performed in [17]. Physiological signals were collected from 24 participants who went through 50 -minute to 1 5 . -hour driving tasks. ECG, EMG, skin conductivity, and GSR along with gaze data were collected during the study. Features were then extracted from segments of 5 minute intervals, and 3 different stress levels were detected using linear discriminant analysis, achieving an accuracy of 97 3% . . Additionally, through analysis of continuous features calculated at 1 -second intervals, it was shown that skin conductivity and heart rate metrics are highly correlated with each other. In another work in this area, high and low emotional valence and arousal were investigated based on electroencephalogram (EEG), EMG, and GSR [29]. The study was performed on 19 subjects where 8 videos were shown to participants in 4 trials. Each trial consisted of a 30 -second video clip as a baseline, followed by 2 minutes of music videos corresponding to low/high, arousal/valence states. Finally, classification was performed using a support vector machine (SVM) classifier with a polynomial kernel. In the case of arousal detection, a combination of EEG and GSR data showed the best performance, achieving an F1-score of 0 638 . , whereas, in the case of valence, the best performance was achieved using all the sensor data (EEG, GSR, EMG), reporting an F1-score of 0 585 . . To provide a gaming experience tailored to individual players, an affect-based dynamic difficulty adjustment (DDA) mechanism was developed in [18]. In the proposed setup, a player's physiological signals, including ECG, EMG, skin conductivity, and body temperature, were analyzed to infer anxiety levels. The difficulty of the game was adjusted in real time according to the player's affective state. Furthermore, a comparative study of regression tree (RT), k -nearest neighbour ( k NN), and Bayesian network (BN) was performed, and showed that the SVM outperformed the other approaches, achieving an accuracy of 88 9% . .

Toward automated analysis and classification of cognitive load, a number of studies have been previously conducted. In [30], cognitive load was examined in the context of human computer interactions. This dataset was collected from 40 participants who went through a series of mental tasks, with 6 varying difficulty levels, to test their cognitive reaction with respect to difficulty. ECG, EMG, GSR, and body temperature were collected in order to perform the classification task. Performance comparison of 3 different machine learning classifiers, k NN, naive Bayes (NB), and random forest (RF), was presented using LOSO and 10 -fold validation techniques. It was found that RF outperformed the other two classifiers with a reported accuracy of 57 84% . . In another study, classification of drivers' cognitive load was performed [20] using an EEG dataset, where data were collected from 33 participants in a high-fidelity moving-base driving simulator. The tasks were performed in three different simulated driving scenarios: hidden exit, crossing, and side wind. A case-based reasoning (CBR) classifier was designed based on extracted time domain and frequency domain features. The highest accuracy achieved was 70% while performing hidden exit scenario.

Lastly, toward classification of expertise, the use of biosignals have been quite rare, to the best of our knowledge. Instead, analysis of expertise based on handwriting has been explored [26], [27]. In [27], high-school students went through a task of solving mathematical problems, while a digital pen was used for recording data. Features were then extracted through analysis of pen strokes on the solution sheet. Then, three different machine learning classifiers, namely SVM, RF, and NB, were applied, reporting an accuracy of 66 7% . , 70 8% 66 7% . , . , respectively. In [25], a method was proposed to assess the performance of trainee surgeons during fundamental laparoscopic surgery tasks such as peg transfer, precision cutting, intracorp knot, and ligating loop. The data were captured using a 3 -axis accelerometer and a 3 -axis gyroscope. An unsupervised machine learning algorithm, k -means clustering, was then performed on the motion data to analyze the correlation between hand movements and level of expertise.

Fig. 1. The proposed architecture is presented. Simulations were designed for trauma responders tending to a critically injured patient (mannequin). A Microsoft HoloLens was used to place augmented reality objects in the simulation room and to capture first-person video during simulations. The ECG data were captured from participants. Pre-processing was performed, followed by QRS detection, RR interval calculation, and feature extraction. A deep multitask neural network was then developed to classify the extracted features into expert and novice, as well as high vs. low cognitive load. The outcome can then be utilized (in future work) to modulate the simulation based on the attributes of the trainee.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-50481f49-1f02-4250-b05a-2bfceb412cc3.jpg)

## III. METHOD

In this section, the method including simulation design, data collection protocol, pre-processing, feature extraction, and machine learning techniques used for classification of expertise and cognitive load are presented.

## A. Experiments and Data

Two separate trauma simulations were developed for the collection of ECG data. In one simulation the patient had suffered a gunshot wound to the abdomen. This was referred to as the Penetrating Trauma Simulation . In the other simulation, the patient had been involved in an automobile roll over, referred to as the Blunt Force Trauma Simulation . Two versions of each simulation were created, basic and enhanced. The enhanced version utilized a Microsoft HoloLens, worn by the participants, to place augmented reality objects in the simulation room to add visual complexity and distractors to the scenario [31], whereas the basic version of the simulations did not use any visual enhancement. Figure 2 shows the simulation room setup with patient monitor (distractor) displaying superfluous information to participants as part of the enhanced version of the simulation. First-person video was recorded from the HoloLens' front facing camera for use in debrief sessions. The goal of the simulation was for participants to monitor and tend to the patient throughout the duration of the simulation.

A Shimmer3 ECG wearable sensor [32] was used in this study for collecting 3 -channel ECG data. Five electrodes were attached to the chest, while the electronic module was attached to the waist for the purpose of mobility. The ECG data were captured at a sampling rate of 500 Hz and streamed to Matlab for the purpose of live visualization and recording.

Ethics approval was secured from the Research Ethics Board of Queen's University, Canada. There were a total of 9 participants, where 5 were physicians specially trained in emergency medicine (considered expert participants). This group had an average age of 34 8 . and standard deviation of 2 31 . . 4 of the participants were Queen's University medical students at the end of their 4 th year of medical studies (considered novice participants) with an average age of 28 5 . and standard

Fig. 2. The simulation room and setup is presented, including the mannequin. The AR distractor shown was introduced as part of the enhanced version of our trauma simulation.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-c4ff31bd-57fc-456d-a0a8-b20f63fce08c.jpg)

deviation of 3 77 . . Nonetheless, the students had some exposure to trauma medicine as they had been rotated through multiple medical specialties (i.e. internal medicine, surgery, emergency medicine, etc.). The reason for selecting students with some background in trauma medicine as novices was so that they could complete the aforementioned simulation.

The novice and expert participants were randomly divided into two groups. The first group performed the basic blunt force trauma simulation followed by the enhanced version of the penetrating trauma simulation. The second group performed the basic penetrating trauma simulation then the enhanced blunt force trauma simulation. Each simulation was designed for 10 minutes in duration. Two minutes of ECG data was captured prior to the simulation to be used as baseline data , for normalization purposes. At the end of each simulation, participants went through a debrief session where high and low cognitive load was first explained to them. Next, their first-person video was shown, and a measure of cognitive load was recorded for critical events throughout the simulation on a 1 to 9 scale.

## B. ECG Pre-processing

1) Electrocardiography: ECG signals were recorded using a 5 -electrode ( 4 leads and 1 ground) wearable ECG System.

ECG signals capture the variations within cardiac electrical potentials over time, showing quasi-periodic behaviour comprised of a sequence of heart beats. Each beat is composed of three waves, a P wave, the QRS complex, and a T wave, as shown in Figure 3. The P wave represents atrial depolarization, the QRS complex represents ventricular depolarization, and the T wave represents ventricular re-polarization [33]. QRS detection acts as the starting point for ECG feature extraction. Through detecting consecutive R-peaks (the positive peaks of QRS complexes), RR intervals can be extracted. RR intervals themselves can then be used for analysis of heart rate variability (HRV), which provides significant information regarding cardiovascular behaviour, which can be influenced by factors such as health and affective/mental states of the subjects.

2) Pan-Tompkins algorithm: The raw ECG signals were processed using the Pan-Tompkins (PT) algorithm [34], [35]. A Butterworth band-pass filter, with a passband frequency of 5 -15 Hz, was used to remove muscle noise, powerline noise, baseline wander, and T-wave interference. After noise removal, the output was differentiated to determine the slope information of the QRS complex. The absolute value of the signal was taken and a moving average filter was used to obtain wave form features in addition to the R-peaks. Filtered signals were segmented in 10 -second windows with 5 seconds of overlap. A thresholding technique was then applied to detect R-peaks, which were further used to extract time-domain and frequency-domain features.

## C. Features

Features were extracted from the identified RR intervals. All features were extracted from both the baseline data and simulation data. The features from the simulation data were normalized with respect to the baseline data. The following sections describe the two categories of features extracted successive to normalization.

1) Time domain features: First, statistical features were extracted from the RR intervals. 11 time domain features [17], [36] were extracted as follows: minimum (RRmin), maximum (RRmax), difference between RRmax and RRmin (RRdiff), mean (RRmean), standard deviation (RRSD), and coefficient of variation (RRCV). In addition, the RMS value of the squared differences of RR intervals (RMSSD) and standard deviation of successive differences (SDSD) were calculated. To understand short-term heart rate variability changes, the number of RR intervals greater than 50 ms (NN 50 ) was calculated, along with the percentage of NN 50 (PNN 50 ). The average heart rate (HR) of each time window was also calculated.

2) Frequency domain feature: In addition to time domain features, 9 frequency domain features were extracted [17], [36]. A Lomb periodogram [37] technique was used for power spectral density (PSD) analysis. PSD estimation provides the basic information of how the power of the signal is distributed as a function of frequency. The features were extracted using the band power of different ranges of frequencies, such as ultra low frequency (ULF) band ( &lt; . 0 003 Hz , very low frequency ) (VLF) band (0 003 . Hz -0 04 . Hz , low frequency (LF) band )

Fig. 3. A sample ECG segment (left) and the wearable ECG monitoring device used in this study (right) are presented.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-f696246f-cbae-4fdb-80df-55706402f32b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-b5ce44f7-d331-4d91-a889-b78959480095.jpg)

(0 04 . Hz -0 15 . Hz , high frequency (HF) band ) (0 15 . Hz -0 4 . Hz , and total power ) (0 Hz -0 4 . Hz ) of the power spectrum were calculated. Furthermore, 4 relative frequency domain features were calculated, namely, normalized LF, normalized HF, ratio of low to high frequency power (LF/HF), and the sum of low and mid-frequency (MF) range normalized by the high-frequency range ((LF+MF)/HF), where MF is defined as (0 08 . Hz -0 15 . Hz . The features calculated in the ) frequency domain are highly indicative of the sympathetic and parasympathetic systems. For example, LF is often associated with the sympathetic nervous system, while HF is associated with the parasympathetic system, and LF/HF corresponds to the overall balance between sympathetic and parasympathetic systems [36].

## D. Classification

A deep learning approach [38] was used for classification of expertise and cognitive load. Our proposed model was based on a feed-forward neural network. Generally, when attempting to classify different attributes using neural networks, two approaches can be employed: ( ) single-task learning, where i each attribute is learned individually using a separate model. This is often used as the default method for classification; and ( ii ) multitask learning , where the multiple attributes to be classified are learned simultaneously using the same model [39]-[41], sharing several hidden layers prior to calculating separate loss functions and performing the classification task for each attribute. Through this approach and by forcing the classifier to learn the internal structures of the related tasks, multitask learning performs better in cases where the learnable tasks possess some interconnections. As a result, given the probable relationship between expertise and the way in which experts and novices handle and manage cognitive load, we utilized a multitask learning approach in this work.

Our proposed deep neural network (as shown in Figure 4) was created with 7 hidden layers, the first 3 layers consisting of 64 neurons, followed by another 3 layers containing 128 neurons each. The final hidden layer consisted of 256 neurons. After each dense hidden layer, a leaky ReLu activation function was used along with dropout to overcome overfitting during training. The last dense layer was connected to two parallel sigmoid output layers in order to classify expertise and cognitive load simultaneously. The network parameters (hidden layers, hidden neurons, dropout, and activation functions)

}˜{]√]√˜(⊕}∐̂

∮{√√√(⊕∐glyph[arrowbt]˜√

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-e33764e7-aa4b-4125-8b8d-a7bb571ca5bb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-1db69bd3-1468-4248-ad10-5d28d59b3c74.jpg)

⊗√√√√√(⊕∐glyph[arrowbt]˜√

∮]̂̂˜{(⊕∐glyph[arrowbt]˜√√

Fig. 4. Our proposed deep multitask neural network architecture is presented.

were all set empirically. Slightly different parameter values, for example 5 or 6 hidden layers or slightly smaller number of hidden neurons, also resulted in good a performance. However, the selected parameters resulted in a marginally better outcome. To determine the estimator function, crossentropy loss was used:

$$L = - \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \left [ y _ { i } \log P ( y _ { i } ) + ( 1 - y _ { i } ) \log ( 1 - P ( y _ { i } ) ) \right ], \ ( 1 ) \quad \text{These $p$} \quad \text{$two$ perfo$} \\ \quad \text{to$ perfo$}.$$

where y is the label and P y ( ) is the predicted probability of novice/expert or high/low cognitive load for all N points. For training the model, a loss function that combines the two individual losses and minimizes both at the same time was used. Our approach considers equal weights of both individual losses:

$$L _ { t o t a l } = L _ { e x p e r t i s e } + L _ { c o g n i t i v e \ l o a d } \quad \ \ ( 2 ) \quad \text{ass}$$

The Adam Optimizer [42] (a stochastic optimization method) was used with a learning rate of 0 001 . , to train the neural network. During training process, neurons sometimes become mutually dependent in fully-connected layers, resulting in the network overfitting to the training data. To prevent the occurrence of this phenomena, a dropout rate of 50% was used after each fully-connected layer so that during training, certain sets of units are not considered during a particular set of forward and backward propagation steps. Additionally, a L 2 regularizer was introduced to overcome overfitting during training.

## IV. RESULTS AND DISCUSSIONS

Implementation: Successive to extraction of the features, our proposed classification architecture was implemented using Tensorflow [43] on a NVIDIA 1070 Ti GPU. Our multitask deep neural network described above was trained with a batch size of 64 , for 400 epochs with a learning rate of 0 001 . . To evaluate the performance of our model on new subjects, we used the LOSO validation scheme.

Performance: The results of our proposed architecture are presented in Table I. Our model was able to achieve an average accuracy of 96 48% . and 88 74% . for classification of level of expertise and cognitive load, respectively. To further measure the performance of our method, precision, recall, negative predictive value (NPV), and F 1 -score were also calculated.

Fig. 5. The training losses for expertise and cognitive load, as well as the total loss vs. the number of epochs are presented.

TABLE I

THE RESULTS OF OUR METHOD USING THE LOSO SCHEME.

|           |   Accuracy |   Precision |   Recall |   NPV |   F1-score |
|-----------|------------|-------------|----------|-------|------------|
| Expertise |       96.6 |        97.7 |     95.4 |  95.6 |      0.965 |
| Cog. Load |       89.4 |        95.7 |     81.8 |  85   |      0.882 |

These parameters showed the high classification performance of our proposed deep multitask neural network and its ability to perform well on new participant data.

In Figure 5 we demonstrate the training losses of expertise and cognitive load against the number of epochs by averaging all the experiments of the LOSO validation scheme. Additionally, the total loss calculated as the sum of losses of expertise and cognitive load is plotted. The figure shows that the cognitive load classification loss is slightly higher than expertise classification loss, which can also be confirmed by the higher accuracy of expertise classification versus cognitive load. However, both the losses closely follow each other, showing stable training through the multitask approach. Training was performed for 400 epochs, as it remained steady after a few hundred epochs. To further investigate and experiment with the training process of our proposed model, we removed the dropout layers and the L 2 regularizer. Overfitting was observed in the absence of dropout and regularization, resulting in low test accuracy.

Comparison: In Table II, we compare our results to prior studies on classification of cognitive load while we could not find any other works attempting to automatically classify expertise. Though there are some literature on classification of cognitive load based on different biological signals like EEG, ECG, and GSR, in our study, we only used ECG, given our goal of achieving a simple and low-cost wearable solution. Accordingly, we decided not to compare our results to studies that had utilized EEG due to its high price and difficulty in integration with AR systems. Nonetheless, other bio-signals such as EMG, GSR, and body temperature are good candidates for future integration into our developed system and provide a good baseline for evaluating the accuracy of our model.

The table illustrates that our proposed model performs robustly when compared to several other studies in the field. In [30], cognitive load classification was performed based on ECG, EMG, GSR, and body temperature captured during a question answering task. Classification was performed using

TABLE II COMPARISON OF OUR PROPOSED DEEP MULTITASK NEURAL NETWORK (DMNN) WITH PREVIOUS APPROACHES.

| Ref.   | Task                | Attribute           | Signals             | Method     | Acc.                    |
|--------|---------------------|---------------------|---------------------|------------|-------------------------|
| [30]   | Mental Task         | Cog. Load           | ECG, EMG, GSR, Temp | k NN NB RF | 50 . 4% 56 . 3% 57 . 8% |
| [18]   | Computer Game       | Anxiety             | ECG, GSR, Temp      | k NN BN RT | 80 . 4% 80 . 6% 80 . 4% |
| [17]   | Driving task        | Stress              | ECG, EMG, GSR       | LDA        | 97 . 3%                 |
| [44]   | Arithmetic Task     | Stress              | GSR                 | SVM LDA    | 81 . 3% 82 . 8%         |
| Ours   | Training Simulation | Expertise Cog. Load | ECG                 | DMNN       | 96 . 6% 89 . 4%         |

three machine learning classifier namely k NN with k = 5 , an RF with 11 trees, and an NB with 11 kernels with a minimum bandwidth of 0 1 . . A LOSO validation accuracy of 50 4% . , 56 3% . , and 57 8% . were achieved for k NN, NB, and RF respectively. It is evident that our model outperforms this study by a considerable margin. However, it should be noted that three classes of cognitive load (low, medium, and high) were used in [30]. Next, to further evaluate our model, in the absence of prominent literature on automatic classification of cognitive load and expertise, we have compared our work to well-cited literature in another field of affective computing, anxiety/stress. The reason that we have selected this attribute for comparison is the fact that there has been a clear relationship established between stress and cognitive load [44]. It can be seen in Table II that our work performs well compared to the mentioned studies.

Cognitive Load vs. Expertise: Further investigations showed an interesting relationship between participants' level of expertise and their cognitive load. In Figure 6, the output of our model for each of the two attributes given every input feature-vector is presented. The probabilities of these attribute are then used as x and y coordinates to derive Figure 6. As shown in the figure, the outputs form two clusters around [0 , 1] and [1 , 0] . This distribution points to the fact that our model has learned a relatively inverse relationship between cognitive load and expertise, inferring that throughout our simulation, experts generally are expected to experience less cognitive load (or handle cognitive load better than novices), while novice participants experience higher amounts of cognitive load. The figure does show some output data along the expertise axis (e.g. around = 0 5 . ) for high predicted cognitive load, indicating that in some cases, higher values of cognitive load can also be experienced by expert subjects.

Limitations and Future Work: One of the limitations of our study is the number of subjects ( 9 ) making up the dataset. While the amount of data recorded for these participants was sufficient to successfully train and test our proposed multitask neural network, we believe further data will aid in a more accurate and robust classifier. Furthermore, electrodermal ac-

Fig. 6. The output probabilities for cognitive load vs. expertise using our model for each input feature-vector is presented. Brighter areas indicate larger number of produced outputs. A relatively inverse relationship is observed. The colorbar indicates the concentration of the data points.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00385v1-figure-90c9e133-b07d-4d99-8a57-c963f14704a2.jpg)

tivity is known to provide important information for measuring cognitive load and other forms of affect [17], [44]. In our future work and during the next round of data collection, GSR will be recorded along with ECG, which will allow for additional affect-related features to be extracted and learned by classifiers. Lastly, as part of our future work, adaptive components will be added to the simulation through the AR headset. The algorithms developed in this study will be used to estimate the expertise and cognitive load of participants, which will, in turn, modify certain components of the training simulation. The impact of such dynamic and adaptive changes will then be monitored and studied for gaining insights into the impact of fine-tuned simulation for trauma training purposes.

## V. CONCLUSIONS

We proposed and developed a pipeline for facilitating an increase in educational efficacy through the development of adaptive simulation, dynamically tailored to the learner's level of expertise and cognitive load. We tackled this problem through classification of cognitive load and expertise. ECG signals were recorded from expert and novice trauma responders during two trauma simulations. Feature extraction was performed in both time and frequency domains for use in a deep learning classifier. A deep multitask neural network was developed and a LOSO validation scheme was used to evaluate the accuracy of our model. Average accuracies of 96 6% . and 89 4% . , and F 1 scores of 0 965 . and 0 882 . were achieved for classification of level of expertise and cognitive load, respectively, showing great performance when compared to other works in the field. The results show the feasibility of accurate classification of expertise and cognitive load using wearable devices. Our proposed framework can enable an increase in educational efficacy through the development of adaptive simulations, dynamically tailored to the learner's level of expertise and cognitive load.

## REFERENCES

- [1] I. Motola, L. A. Devine, H. S. Chung, J. E. Sullivan, and S. B. Issenberg, 'Simulation in healthcare education: A best evidence practical guide. amee guide no. 82,' Medical Teacher , vol. 35, no. 10, 2013.
- [2] M. L. Good, 'Patient simulation for training basic and advanced clinical skills,' Medical Education , vol. 37, no. s1, p. 1421, 2003.
- [3] B. Mehler, B. Reimer, and J. F. Coughlin, 'Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task,' Human Factors: The Journal of the Human Factors and Ergonomics Society , vol. 54, no. 3, p. 396412, 2012.
- [4] R. Aggarwal, O. T. Mytton, M. Derbrew, D. Hananel, M. Heydenburg, B. Issenberg, C. MacAulay, M. E. Mancini, T. Morimoto, N. Soper, A. Ziv, and R. Reznick, 'Training and simulation for patient safety,' BMJ Quality &amp; Safety , vol. 19, no. Suppl 2, pp. i34-i43, 2010.
- [5] D. Rodenburg, P. Hungler, S. A. Etemad, D. Howes, A. Szulewski, and J. Mclellan, 'Dynamically adaptive simulation based on expertise and cognitive load,' in IEEE Games, Entertainment, Media Conference (GEM) , pp. 1-6, Aug 2018.
- [6] K. L. Fraser, P. Ayres, and J. Sweller, 'Cognitive load theory for the design of medical simulations.,' Simulation in healthcare : journal of the Society for Simulation in Healthcare , vol. 10 5, pp. 295-307, 2015.
- [7] J. J. G. Van Merrinboer and J. Sweller, 'Cognitive load theory in health professional education: design principles and strategies,' Medical Education , vol. 44, no. 1, pp. 85-93, 2010.
- [8] S. Kalyuga, P. Ayres, P. Chandler, and J. Sweller, 'The expertise reversal effect,' Educational Psychologist , vol. 38, no. 1, pp. 23-31, 2003.
- [9] G. Paas, 'Training strategies for attaining transfer of problem solving skills in statistics : a cognitive-load approach,' Journal of educational psychology , vol. 84, no. 4, pp. 429-434, 1992.
- [10] R. Br¨ unken, T. Seufert, and F. Paas, 'Measuring cognitive load,' 2010.
- [11] K. H. Teigen, 'Yerkes-dodson: A law for all seasons,' Theory &amp; Psychology , vol. 4, no. 4, pp. 525-547, 1994.
- [12] J. Sweller, 'Cognitive load theory, learning difficulty, and instructional design,' Learning and instruction , vol. 4, no. 4, pp. 295-312, 1994.
- [13] F. G. Paas, J. J. Van Merri¨ enboer, and J. J. Adam, 'Measurement of cognitive load in instructional research,' Perceptual and motor skills , vol. 79, no. 1, pp. 419-430, 1994.
- [14] M. N. Sahadat, S. Consul-Pacareu, and B. I. Morshed, 'Wireless ambulatory ecg signal capture for hrv and cognitive load study using the neuromonitor platform,' in 6th International IEEE/EMBS Conference on Neural Engineering (NER) , pp. 497-500, Nov 2013.
- [15] F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. M. V. Gerven, 'Cognitive load measurement as a means to advance cognitive load theory,' Educational Psychologist , vol. 38, no. 1, pp. 63-71, 2003.
- [16] R. W. Picard, E. Vyzas, and J. Healey, 'Toward machine emotional intelligence: Analysis of affective physiological state,' IEEE Transactions on Pattern Analysis &amp; Machine Intelligence , no. 10, pp. 1175-1191, 2001.
- [17] J. Healey and R. W. Picard, 'Detecting stress during real-world driving tasks using physiological sensors,' IEEE Transactions on Intelligent Transportation Systems , vol. 6, no. 2, pp. 156-166, 2005.
- [18] C. Liu, P. Agrawal, N. Sarkar, and S. Chen, 'Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback,' International Journal of Human-Computer Interaction , vol. 25, no. 6, pp. 506-529, 2009.
- [19] J. Zhai and A. Barreto, 'Stress detection in computer users based on digital signal processing of noninvasive physiological variables,' in International Conference of the IEEE engineering in medicine and biology society , pp. 1355-1358, IEEE, 2006.
- [20] S. Barua, M. U. Ahmed, and S. Begum, 'Classifying drivers' cognitive load using eeg signals.,' in pHealth , pp. 99-106, 2017.
- [21] S. Jerritta, M. Murugappan, R. Nagarajan, and K. Wan, 'Physiological signals based human emotion recognition: a review,' in IEEE 7th International Colloquium on Signal Processing and its Applications , pp. 410-415, IEEE, 2011.
- [22] D. Gabana, L. Tokarchuk, E. Hannon, and H. Gunes, 'Effects of valence and arousal on working memory performance in virtual reality gaming,' in 7th International Conference on Affective Computing and Intelligent Interaction (ACII) , pp. 36-41, Oct 2017.
- [23] R. Navarathna, P. Carr, P. Lucey, and I. Matthews, 'Estimating audience engagement to predict movie ratings,' IEEE Transactions on Affective Computing , 2017.
- [24] C. Conati, 'Probabilistic assessment of user's emotions in educational games,' Applied Artificial Intelligence , vol. 16, no. 7-8, pp. 555-575, 2002.
- [25] G. S. Kirby, R. M. Kwasnicki, C. Hargrove, J. L. Rees, M. H. Sodergren, G.-Z. Yang, and B. P. Lo, 'Wireless body sensor for objective assessment of surgical performance on a standardised fls task,' in Proceedings of the 9th international conference on body area networks , pp. 147-153, ICST (Institute for Computer Sciences, Social-Informatics and , 2014.
- [26] S. Oviatt, K. Hang, J. Zhou, K. Yu, and F. Chen, 'Dynamic handwriting signal features predict domain expertise,' ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 8, no. 3, p. 18, 2018.
- [27] J. Zhou, K. Hang, S. Oviatt, K. Yu, and F. Chen, 'Combining empirical and machine learning techniques to predict math expertise using pen signal features,' in Proceedings of the ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge , pp. 29-36, ACM, 2014.
- [28] R. E. Plutchik and H. R. Conte, Circumplex models of personality and emotions. American Psychological Association, 1997.
- [29] D. Girardi, F. Lanubile, and N. Novielli, 'Emotion detection using noninvasive low cost sensors,' in 7th International Conference on Affective Computing and Intelligent Interaction (ACII) , pp. 125-130, IEEE, 2017.
- [30] A. Oschlies-Strobel, S. Gruss, L. Jerg-Bretzke, S. Walter, and D. HazerRau, 'Preliminary classification of cognitive load states in a human machine interaction scenario,' in International Conference on Companion Technology (ICCT) , pp. 1-5, IEEE, 2017.
- [31] Microsoft, 'Microsoft hololens - mixed reality technology for business.' [Online]. Available: https://www.microsoft.com/en-us/hololens, 2019. [Accessed: 2018-04-04].
- [32] Consensys, 'Consensys ecg.' [Online]. Available: http: //www.shimmersensing.com/products/ecg-development-kit. [Accessed: 2018-04-04].
- [33] E. A. Ashley and J. Niebauer, Cardiology explained . Remedica, 2004. [34] J. Pan and W. J. Tompkins, 'A real-time qrs detection algorithm,' IEEE Transactions on Biomedical Engineering , vol. BME-32, pp. 230-236, March 1985.
- [35] A. F. Hussein, A. k. N, M. Burbano-Fernandez, G. Ramrez-Gonzlez, E. Abdulhay, and V. H. C. De Albuquerque, 'An automated remote cloud-based heart rate variability monitoring system,' IEEE Access , vol. 6, pp. 77055-77064, 2018.
- [36] A. J. Camm, M. Malik, J. T. Bigger, G. Breithardt, S. Cerutti, R. J. Cohen, P. Coumel, E. L. Fallen, H. L. Kennedy, R. Kleiger, and F. Lombardi, 'Heart rate variability: standards of measurement, physiological interpretation and clinical use. task force of the european society of cardiology and the north american society of pacing and electrophysiology,' 1996.
- [37] N. R. Lomb, 'Least-squares frequency analysis of unequally spaced data,' Astrophysics and space science , vol. 39, no. 2, pp. 447-462, 1976.
- [38] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, p. 436, 2015.
- [39] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, 'Facial landmark detection by deep multi-task learning,' in European Conference on Computer Vision , pp. 94-108, Springer, 2014.
- [40] X. Li, L. Zhao, L. Wei, M.-H. Yang, F. Wu, Y. Zhuang, H. Ling, and J. Wang, 'Deepsaliency: Multi-task deep neural network model for salient object detection,' IEEE Transactions on Image Processing , vol. 25, no. 8, pp. 3919-3930, 2016.
- [41] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, 'Joint face detection and alignment using multitask cascaded convolutional networks,' IEEE Signal Processing Letters , vol. 23, no. 10, pp. 1499-1503, 2016.
- [42] D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014.
- [43] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man´ e, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vi´ egas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, 'TensorFlow: Large-scale machine learning on heterogeneous systems,' 2015.
- [44] C. Setz, B. Arnrich, J. Schumm, R. La Marca, G. Tr¨ oster, and U. Ehlert, 'Discriminating stress from cognitive load using a wearable eda device,' IEEE Transactions on Information Technology in Biomedicine , vol. 14, no. 2, pp. 410-417, 2010.
Pritam Sarkar Kyle Ross Aaron J. Ruberto Dirk Rodenburg Paul Hungler Ali Etemad
Classification of Cognitive Load and Expertise for Adaptive Simulation using Deep Multitask Learning
Simulations are a pedagogical means of enabling a risk-free way for
healthcare practitioners to learn, maintain, or enhance their knowledge and
skills. Such simulations should provide an optimum amount of cognitive load to
the learner and be tailored to their levels of expertise. However, most current
simulations are a one-type-fits-all tool used to train different learners
regardless of their existing skills, expertise, and ability to handle cognitive
load. To address this problem, we propose an end-to-end framework for a trauma
simulation that actively classifies a participant's level of cognitive load and
expertise for the development of a dynamically adaptive simulation. To
facilitate this solution, trauma simulations were developed for the collection
of electrocardiogram (ECG) signals of both novice and expert practitioners. A
multitask deep neural network was developed to utilize this data and classify
high and low cognitive load, as well as expert and novice participants. A
leave-one-subject-out (LOSO) validation was used to evaluate the effectiveness
of our model, achieving an accuracy of 89.4% and 96.6% for classification of
cognitive load and expertise, respectively.