The Impact of a Rater Training Program on the TGMD-3 Scoring Accuracy of Pre-Service Adapted Physical Education Teachers

This study aimed to examine the impact of a TGMD-3 rater training program on pre-service adapted physical education (APE) teachers’ ability to score fundamental motor skills for children with developmental disabilities (DD). The training program consisted of a description of the TGMD-3 instrument and DD, as well as content focused on skill performance and correct scoring through systematic analysis of 13 FMS via the instrument. Three experts established the standard score through the TGMD-3 performance evaluation of a child with DD. Thirty-nine pre-service APE teachers in the experimental group and 30 pre-service teachers in the control group completed the pre- and post-test process in this study. There were significant impacts on the pre-service teachers’ ability to score the locomotor, ball skill subtests, and total skill scores (p < 0.05) compared to the control group. Specifically, the results of the three locomotor skills (running, horizontal jump, & slide) and three ball skills (two-hand strike, dribble, & kick) significantly improved in scoring accuracy. The results of this study can evidence that a TGMD-3 rater training program for pre-service APE teachers enhances the scoring accuracy of FMS among children with DD.

Introduction
In the U.S., about 1 in 6 (17%) children 3-17 years of age were diagnosed with developmental disabilities (DD) based on the 2009-2017 National Health Interview Survey (NHIS; Zablotsky et al., 2019) [1]. This trend is increasing; Zablotsky and colleagues (2019) [1] found a significant increase when they compared the two time periods of 2009-2011 (16.2%) and 2015-2017 (17.8%) in the same age group. More specifically, the percentage of children with autism spectrum disorder (ASD) increased from 1.1% to 2.5%, and the percentage of children with intellectual disabilities (ID) increased from 0.9% to 1.2% in the same age group. Considering the results of this study, the Center for Disease Control and Prevention (CDC) and the Health Resources and Services Administration (HRSA) suggested that public health and healthcare service providers plan for medical, educational, and social services to support children and their families.
Fundamental movement skills (FMS) are required to perform daily life and sports activities and have an important influence on emotional and social development in infancy and childhood (Clark and Metcalfe, 2002;Payne and Isaacs, 2020) [2,3]. Many studies regarding the development of FMS in early childhood have shown that significant improvements in the FMS of young children may stem from educational environments as opposed to free playtime (Logan et [4][5][6][7]. The level of FMS performance or the development of FMS in children has been assessed by educators and practitioners using quantitative and/or qualitative approaches (Liu et al., 2017) [8]. There are various kinds of movement assessment instruments to identify the developmental condition of FMS, such as the Test of Gross Motor Development (TGMD; Ulrich, 1985Ulrich, , 2000Ulrich, , 2019 [9][10][11], the Bruinks-Oseretsky Test of Motor Proficiency-2 (BOT-2; Bruininks, 2005) [12], and the Movement Assessment Battery for Children-2 (MABC-2; Henderson et al., 2007) [13]. The TGMD was developed to assess gross motor skill development in children with and without disabilities between 3 and 10 years of age. In 2019, the TGMD-3 was re-released with changed items and transformed performance criteria within both the locomotor and ball skill subtests   [14]. A total of 13 FMS skills falls under six locomotor and seven ball skills on the TGMD-3. Previous studies assessed FMS performance in children with (Brian et [19][20][21][22][23], using the TGMD assessment instrument in educational settings to measure gross motor skills via locomotor and object control subtests. This assessment instrument is an important indicator for monitoring the levels and progress of FMS in children with developmental disabilities (Kim et al., 2012;Kirk and Rhodes, 2011;Macdonald et al., 2017) [24][25][26].
Regarding the research on adapted physical education (APE), efforts to participate in regular physical activity and to develop motor skills for children in APE programs were proven to have positive impacts in many studies (Esentürk and [27][28][29][30][31]. Those studies were implemented based on treatments in health settings, organizational research, and various professional fields and supported by scientific and objective evidence, which is referred to as evidence-based practice (EBP; American Psychological Association, 2005; Williams and Glisson, 2014) [32,33]. The importance of EBP has been emphasized in the APE area (Hutzler, 2011;Bouffard and Reid, 2012;Reid et al., 2012) [34][35][36].
Children with low levels of FMS experience several barriers to participation in physical activities (Obrusnikova and Dillon, 2011;Ohrberg, 2013;Pan et al., 2011) [37][38][39], and this has a negative effect on self-confidence and motivation (Goodwin and Thurmeier, 2006) [40]. There is no doubt that providing adequate services to enhance the acquisition and development of FMS for children with DD is important (Stodden et al., 2008; Strong et al., 2005) [41,42]. However, although the TGMD has been recognized for its convenience and is a standardized test that has been used in many studies and education services ( [24,[45][46][47]. For example, because this instrument measures subjective judgments, results between raters may not be consistent, especially when scoring FMS among children with disabilities (Kim et al., 2012) [24]. Kim and colleagues (2012) [24,46] found that statistically inconsistent results occurred among raters in the scoring results of TGMD-2 in children with ID. Furthermore, Palmer and Brian (2017) [46] scored the 12 FMS on the TGMD-2 and presented statistically significant differences between novice and experienced raters in 10 of the skills, except for the kick a stationary ball and gallop skills. On the other hand, Maeng et al. (2017) [48] and Rintala et al. (2017) [49] each examined the intra-and inter-rater reliability of the TGMD-3 when used by experienced experts, and reported good to excellent agreement in all skill items on the assessment instrument.
Based on the importance and common performance traits of FMS among children with DD, rater training is necessary to evaluate FMS accurately in this population, especially for pre-service or novice teachers. According to Lounsbery and Coker (2008) [50], the proficiency of general physical education (GPE) teachers in skill analysis was not significantly accurate. They presented that all PE educators should learn to improve their ability to score more accurately. In doing so, the curriculum in physical education teacher education should be reformed to provide comprehensive learning experiences in skill analysis. A systematically structured rater training program and its implementation can be utilized to maximize the use of the movement instrument and tools when PE teachers score and evaluate FMS among children, especially those with DD. The purpose of this study was to investigate the impact of a TGMD-3 rater training program on pre-service APE teachers' ability to score fundamental motor skills for children with DD. By establishing standards, we intend to develop a training program to evaluate the FMS of children with DD more accurately in the field of physical education, through analyzing the sub-technology and performance criteria of the TGMD-3. In addition, the results of this study provided evidence not only of the effectiveness of rater training, but also the development of practical rater training protocols. It will also contribute to accurately evaluating the FMS of children with disabilities, enabling the provision of effective teaching.

Participants
Participants were college students majoring in adapted physical education in South Korea. The inclusion criteria were as follows: (a) majoring in the APE program, (b) had not taken the motor development class in the program, and (c) did not score fundamental motor skills on the TGMD-3. The author recruited participants in two APE classes to ask for the cooperation of instructors. After explaining the experimental purpose, procedures, risks, and benefits of this study, the author obtained written informed consent from the participants. Two groups in the present study, the experimental and control groups, were randomly selected between the two classes. A priori sample size was calculated using G*Power (version 3.1.9.7; Franz Faul, Kiel, Germany) with a medium effect size (0.5) to support detected significance in the results. The sample size of this study required a minimum sample size of 38 participants with a power of 95% and an alpha of 0.05. In the experimental group, a total of 39 pre-service adapted physical education teachers (12 females, mean age ± SD = 21.73 ± 1.68) participated in the TGMD-3 training program of this study. The experimental group completed the training program and scored FMS on the TGMD for a child with developmental disabilities. For the control group, a total of 30 pre-service APE teachers (13 females, mean age ± SD = 21.15 ± 1.24) participated in the study. The control group was given general information about the TGMD-3 and the scoring methods of FMS on the assessment instrument.

Training Program of the TGMD-3
To develop the training program, the present study established the information of the TGMD-3, including 13 FMS with performance criteria for pre-service teachers (i.e., novices) to score FMS on the assessment instrument. This training program was also intended to train pre-service teachers on how to accurately score when considering the components of each performance criterion, as well as to understand the behavioral and movement characteristics of children with DD, in order to evaluate their performance on the TGMD-3. Regarding the components of the performance criteria, for instance, the training program clarified what "Rotate" and "Derotate" mean, as well as the difference to the two-hand striking skill on the TGMD-3. Additionally, if a child with DD shows challenging behaviors in the performance trial, such as stereotyped patterns of behavior or unexpected movements, how to correctly score according to each performance criterion was covered. First, a review of reference studies related to the TGMD-2 and 3 was conducted to arrange findings and suggestions for the psychometric properties of those instruments. Six experts in motor development and adapted physical education (APE) areas (motor development: 3; APE: 3) were engaged in deriving limitations and improvement methods when using the TGMD-3 for children with DD. This study used the administration video by Dr. Ulrich (2014) [51], who is the TGMD developer, to describe the general information of the TGMD-3, and TGMD-3 skill performance videos of children with DD in order to provide sample performances and scoring in the training program for the pre-service teachers. The rater training program consisted of an introduction to the TGMD-3, the scoring method, performance characteristics of children with DD, and the guidelines for correct scoring according to each performance criterion (see Table 1). The training program provided not only content on children with DD and their behavior and movement characteristics, but also scoring of FMS for children with DD as a practical experience. During the scoring practice, discussion and feedback were given to provide explicit evidence and reasoning for the correct score of either 1 or 0 on each performance criterion on the TGMD-3.
The fidelity of this training program was assessed using the instrument criteria based on fidelity and measuring internal validity (McKay et al., 2018; Kaderavek and Justice, 2010) [52,53]. The training program was recorded on video for 120 min, with the participants' consent. Three experts scored the training program plan and program implementation on the fidelity check sheet. There were three eligibility criteria to be considered an expert rater: (a) a graduate degree in motor development or adapted physical education/activity, (b) experience assessing skills on the TGMD-3, and (c) a minimum of 5 years' experience teaching FMS curricular content to children with DD in physical education, adapted physical education, or physical activity programs. The intervention program fidelity was 93.4%, with a high inter-observer agreement of 92% on the training program plan and recorded video.

Study Design & Procedure
This study featured an experimental design that included a pre-and post-test, an experiment, and a control condition (see Figure 1). The dependent variables in this study were the three FMS categories (locomotor, ball skill subtest, and total skill score), and a total of 13 FMS on the TGMD-3.
The primary researcher received approval from the Institutional Review Board (IRB) for all procedures before training and data collection. First, the performance videos of a child with DD were shared with three expert raters who scored the FMS of the children with DD. The standard scores of FMS on the TGMD-3 were established by the three experts to compare with the scores of the participants. The experts independently scored the skill performance of a child with DD on the TGMD-3. Then, they had a meeting to reach 100% agreement on scores on the TGMD-3 by comparing and discussing a total of 13 skill performances with all performance criteria on the assessment instrument. The primary researcher received approval from the Institutional Review Board (IRB) for all procedures before training and data collection. First, the performance videos of a child with DD were shared with three expert raters who scored the FMS of the children with DD. The standard scores of FMS on the TGMD-3 were established by the three experts to compare with the scores of the participants. The experts independently scored the skill performance of a child with DD on the TGMD-3. Then, they had a meeting to reach 100% agreement on scores on the TGMD-3 by comparing and discussing a total of 13 skill performances with all performance criteria on the assessment instrument.
Participants in the experiment group not only engaged in the TGMD-3 training program, but also scored 13 skills on the TGMD-3 of a child with DD two times, as a pre-and post-test. After learning general information about the TGMD-3 in the classroom setting, they watched the 13 TGMD-3 skill performances of a child with DD through a confidential web storage link and scored the skill performance on the TGMD-3 scoring form. Then, behavior and movement characteristics of a child with DD and correct scoring methods in the training program were provided to the participants before the post-test scoring. Lastly, they scored the same skill performance on the TGMD-3 again as the post-test in this study. The primary researcher collected the pre-and post-test scores of the TGMD-3 from the participants.
Participants in the control group received 20 min of introduction to the TGMD-3 in the classroom to score 13 skills on the TMGD-3 on the pre-test. After the introduction time, they were asked to score the same FMS of a child with DD as in the training program. The pre-test score sheet was submitted to the primary researcher. Considering the learning effect, the control group was asked to submit the score sheet of the same skill performance on a confidential web storage link within the next 5 days for the post-test.

Data Analysis
The mean and standard deviation of the skill scores in both groups were calculated for the locomotor (i.e., the sum of 6 skills) and ball skill (i.e., the sum of 7 skills) subtests, Participants in the experiment group not only engaged in the TGMD-3 training program, but also scored 13 skills on the TGMD-3 of a child with DD two times, as a pre-and post-test. After learning general information about the TGMD-3 in the classroom setting, they watched the 13 TGMD-3 skill performances of a child with DD through a confidential web storage link and scored the skill performance on the TGMD-3 scoring form. Then, behavior and movement characteristics of a child with DD and correct scoring methods in the training program were provided to the participants before the post-test scoring. Lastly, they scored the same skill performance on the TGMD-3 again as the post-test in this study. The primary researcher collected the pre-and post-test scores of the TGMD-3 from the participants.
Participants in the control group received 20 min of introduction to the TGMD-3 in the classroom to score 13 skills on the TMGD-3 on the pre-test. After the introduction time, they were asked to score the same FMS of a child with DD as in the training program. The pre-test score sheet was submitted to the primary researcher. Considering the learning effect, the control group was asked to submit the score sheet of the same skill performance on a confidential web storage link within the next 5 days for the post-test.

Data Analysis
The mean and standard deviation of the skill scores in both groups were calculated for the locomotor (i.e., the sum of 6 skills) and ball skill (i.e., the sum of 7 skills) subtests, total skill scores (i.e., the sum of 13 skills), and each skill on the TGMD-3, respectively. The scoring differences between both groups (i.e., experiment and control groups) from pre-to post-scoring were also analyzed using a two-way repeated-measures analysis of variance (Two-way RM-ANOVA) to examine the improvement in scoring accuracy of preservice teachers compared to experts. Bonferroni's post hoc analysis was used to examine differences in the group. All analyses were performed with IBM SPSS Statistics, version 28.0 (IBM Corp, Armonk, NY, USA). The significance level of the p-value was below 0.05.

Results
The comparison results of locomotor, ball skill, and total gross motor skill scores between the pre-service teachers and experts are shown in Table 2. The results, based on RM-ANOVA, were for the locomotor subtest, including six skills (F (1, 67) = 42.52, p < 0.001), the ball skill subtest, including seven skills (F (1, 67 = 31.41, p < 0.001), and the total gross motor scores, consisting of 13 skills (F (1, 67) = 47.59, p < 0.001), on the TGMD-3. The scores of the locomotor subtest changed from a mean and standard deviation (M ± SD) of 32.92 ± 4.31 to 36.44 ± 2.27. The ball skill subtest scores changed from 38.56 ± 4.62 to 41.00 ± 3.66. In the total skill scores, the mean score on the pre-test (M = 71.49, SD = 7.24) was also changed on the post-test (M = 77.47, SD = 4.14). The locomotor, ball skill subtest, and total skill scores from the expert were 39.00, 46.00, and 85.00, respectively. All scores for the pre-service teachers in the experiment group were closer to the expert score compared to the control group. This indicated that the scoring of the experiment group significantly reduced in difference from the pre-to the post-test. Figure 2 shows the change of mean score differences in the locomotor, ball skill subtest, and total skill scores of the experiment and control groups between the pre-and post-test.  Each skill in the locomotor subtest on the TGMD-3 showed different results, as can be seen in Table 3. Three locomotor skills, the run, horizontal jump, and slide, were significantly improved, and were closer to the expert scores on the post-test compared to the control group. These three locomotor skills had significant p-values in the comparison be-  Table 3. Descriptive and statistical results of the TGMD-3 skill score comparison between pre-service APE teachers and expert.

Category
Group Pre-Service APE Teacher a Expert  F (1,67) Post-Test Each skill in the locomotor subtest on the TGMD-3 showed different results, as can be seen in Table 3. Three locomotor skills, the run, horizontal jump, and slide, were significantly improved, and were closer to the expert scores on the post-test compared to the control group. These three locomotor skills had significant p-values in the comparison between pre-and post-test scoring (run:  Table 3. Descriptive and statistical results of the TGMD-3 skill score comparison between pre-service APE teachers and expert.

Discussion
The purpose of this study was to investigate the effect of a rater training program for pre-service APE teachers on the scoring accuracy of fundamental motor skills (FMS) on the TGMD-3 in children with DD. Establishing standards of scoring FMS on the TGMD-3, the training program of this instrument, when used to rate children with DD, might enhance scoring accuracy on the performance criteria of the TGMD-3. The findings in this study provide evidence to satisfy the suggestions in previous studies (Kim et al., 2012;Palmer and Brian, 2017) [24,46] that emphasized the necessity for a TGMD rater training protocol in physical education teacher education programs, in order to improve scoring accuracy of FMS on the assessment instrument.
The TGMD-3 training program which was developed for the present study could be used as a basic resource for training physical education educators and pre-service teachers of children with disabilities, especially developmental disabilities. This training program could also have an initiating role in enhancing the quality of physical education and physical therapy services for children with disabilities. Though there were scoring differences on the TGMD-3 in the comparison between expert and pre-service teachers, the pre-service teachers showed several improvements in scoring on the assessment instrument following the training program. This finding is evidence for the reduction of rater effects by rater training (Weigle, 1998) [54], especially regarding the TGMD assessment instrument for children with disabilities (Kim et al., 2012) [24]. Overall scoring accuracy of the locomotor, ball skill subtests, and total gross motor skill scores on the TGMD-3 were significantly improved. According to Palmer and Brian (2017) [46], there were significant differences in both locomotor skill and object skill (i.e., the ball skill in the TGMD-3) subtests on the TGMD-2 between novice and expert raters. In the locomotor skill subtest, all skills were significantly different, with the exception of the gallop skill (p = 0.09). In the comparison with the present study, the scoring differences between novice (i.e., pre-service teachers) and expert raters were improved in the run, horizontal, and slide skills on the TGMD-3 compared to the control group. The other three locomotor skills (i.e., gallop, hop, skip) on the TGMD-3 did not show a significant improvement in the within-subject comparison (time × group), though several effects of the intervention in the experiment group were significant in the gallop and skip skills (p < 0.05). This implies that the gallop and skip skills on the TGMD-3 should be investigated to verify either the impact of the rater training program or the learning effect, according to the comprehensiveness of the performance criteria of those skills, on scoring accuracy for novices. At the same time, the content approach of the locomotor skills in the TGMD-3 rater training protocol should be improved based on differing perspectives for each performance criterion. Kim and colleagues (2012) [24] investigated rater effects in scoring FMS items on the TGMD-2 among children with intellectual disabilities, and found that the run (17.89%) and horizontal jump (12.84%) skills had relatively large error variance by rater effects compared to other locomotor skills on the instrument. The results of the present study found similar results. The mean scoring differences between the expert and novice raters in the run (experiment M = −1.92; control M = −2.90) and horizontal jump (experiment M = 2.36; control M = −2.53) skills on the pre-test were larger than in the other skills. The run, horizontal jump, and slide skills in the locomotor subtest were significantly improved according to the effect of rater training in the present study. The TGMD-3 rater training program in this study had a positive impact on reducing rater effects on the run and horizontal jump skills, as pointed out by Kim and colleagues (2012) [24], as well as on the slide skill. In addition, this implied that scoring the gallop, kick, and skip skills on the rater training program should be dealt with in more detail for each performance criterion of the skill.
For the subtest of ball skills, there was a significant improvement in the scoring accuracy for the two-hand strike, dribble, and kick skills due to the rater training intervention. The kick skill on the TGMD-2 had the lowest difference between the expert and novice raters for object control (i.e., ball skills on the TGMD-3) skills (Palmer and Brian, 2017) [46]. Additionally, Maeng et al. (2017) [48] concluded that the kick skill on the TGMD-3 had the lowest inter-rater reliability among the ball skills (ICC: 0.51). Moreover, Kim and colleagues (2012) [24] asserted that this skill had a large error variance (18.00%) by rater effects and a low agreement (ICC: 0.50). However, the present study showed that the one-hand strike skill (experiment M = −1.76; control M = −2.30) had the highest-scoring difference compared to the experts. Moreover, the two-hand strike skill showed the second highest-scoring difference (experiment M = −1.70; control M = −2.01). It could be inferred that the performance criteria of the skills on the TGMD-3 among children with DD should be examined to identify the error variances by raters according to their characteristics in future research. In other words, reliability and validity examinations of the TGMD-3 among children with DD should be considered in future studies.
In the present study, the catch skill had the lowest scoring difference compared to the experts (both experiment and control M = −0.07). This implied that the catch skill has comprehensive performance criteria regardless of the expertise of raters. Interestingly, scoring the catch skill of a child with DD in the experiment group had a larger mean difference after the rater training program. The scoring difference of the control group was closer to the expert than the experimental group on the post-test. Moreover, the scoring difference of the control group compared to the expert changed from negative to positive. However, the p-value (p = 0.062) of the catch skill was close to 0.05, though it was not significant. This unusual impact of the training program may indicate that the content for correct scoring on the catch skill should aim to minimize misunderstanding of the performance criteria.
The change in the mean differences between the two groups (i.e., experiment and control) in the overhand throw skill compared to the expert was not significant (p > 0.05) because the change in the scoring means from pre-to post-test was similar (experiment: 0.53; control: 0.50). The same trend of change appeared for the underhand strike skill (experiment: 0.13; control: 0.10). Regarding the one-hand strike skill, the change in the mean differences in the experiment group was not significant, although the scoring accuracy of the control group was decreased. We found that the rater training program did not provide effective training content to score the overhand throw, underhand throw, and one-hand strike skills on the TGMD-3 based on the results. These findings suggested that the TGMD-3 rater training program should describe correct scoring methods based on the proper interpretation of each performance criterion on the skills in the future rater training protocol in order to improve the scoring accuracy of novices.
Palmer and Brian (2017) [46] asserted that most skills on the TGMD-2 showed a significant scoring difference between experts and novice raters, except for the gallop in the locomotor and the kick in the object control (i.e., ball skill) subtests. Thus, they suggested training protocols and preparation to correctly score FMS on the TGMD assessment instrument. This study applied a TGMD-3 training program to examine its impact on scoring accuracy when evaluating FMS on the instrument among children with DD. There were significant impacts on the scoring accuracy of the skills on the TGMD-3, although changes in some skills did not reach statistical significance. This implies that the training program for pre-service teachers can improve scoring accuracy by reducing rater effects when evaluating FMS on the TGMD-3 among children with DD. The finding indicates that the TGMD-3 rater training program can help general and adapted physical education (GPE/APE) teachers to improve their scoring competency in skill analysis (Lounsbery and Cocker, 2008) [50], to adequately make placement decisions (Akuffo and Hodge, 2008;Columna et al., 2010) [55,56] and to prepare proper APE programs for children with disabilities (Lytle et al., 2010) [57].
The limitations in this study include that only one subject with DD skill performance on the TGMD-3 was scored to investigate the effect of the TGMD-3 rater training intervention on scoring accuracy. Due to varying behavior and movement characteristics of children with DD, the scoring accuracy of novices may be influenced when evaluating motor skill performance of those populations. Therefore, the results may differ in studies including children with DD who have unique types of behavior and movement characteristics. Regarding the participants, this study recruited only pre-service APE teachers who had not taken the motor development class in the program. However, the APE program curriculums are different from university to university. Thus, there is a limit to generalizing the result of this study to every pre-service APE teacher. In addition, though there was a lack of scoring accuracy among pre-service APE teachers before the training, its range was wide according to personnel experiences or their college year. Future research is recommended applying to more specific eligibility (e.g., dealing with individuals with disabilities, siblings with disabilities) for recruitment to have equivalent conditions for participants. Lastly, a TGMD-3 rater training program was executed for 120-min. There were limitations to providing information on scoring correctly not only diverse challenging behaviors of children with DD but also their various levels of skill performance on the TGMD-3. A TGMD-3 rater training program for novices should be conducted to cover more arguments in scoring FMS among children with DD in future research.

Conclusions
This study aimed to examine the impact of a TGMD-3 rater training intervention on scoring accuracy for fundamental motor skills (FMS) among children with developmental disabilities (DD). There were significantly positive effects of the TGMD-3 rater training intervention on correctly scoring skill items on the TGMD-3 in a child with DD by preservice adapted physical education teachers. The positive results included the following: (a) three locomotor skills (i.e., run, horizontal jump, slide); (b) three ball skills (i.e., two-hand strike, dribble, kick); (c) the locomotor subtest; (d) the ball skill subtest; and (e) total skill scores on the assessment instrument. This TGMD-3 rater training protocol can improve performance scoring by reducing the rater effects of different characteristics and experiences. It will be helpful when evaluating FMS on those measures among children with DD or a lower level of skill proficiency. However, future research involving children with DD with diverse challenging behaviors and different levels of movement performance and investigation of specific skill items and their performance criteria on the TGMD-3 should be conducted to address the limitations discussed above.  Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.


Challenges of Nursing Students in Relation to Academic Performance at the Faculty of Technical Medical Sciences

Nursing students face various challenges during their years of study, which may hinder their academic performance. The success of nursing students is particularly important given that they must provide quality care to patients, family members and the community. Academic performance in nursing programs includes advances in theoretical subjects and the acquisition of practical skills in the clinic. A series of studies have shown that during the years of study, nursing students face many difficulties that affect their learning skills and consequently their academic performance. This study aims to identify and describe the difficulties experienced by nursing students of the Faculty of Technical Medical Sciences of Tirana in relation to academic performance. This goal was achieved by using a qualitative research method, such as focus group discussion. Data were collected through discussion with 51 Bachelor students, 17 of whom in the first year, 15 in the second year and 19 participants in the third year. For each year of the study 2 semi-structured discussions were held and the data were analyzed through thematic analysis. The findings of this study found that nursing students face these difficulties: Low level of students engagement in clinical practice; Low number of nursing courses; Passive teaching methods; Unfair and non-transparent evaluation; Large number of exams per season; Insufficient time to prepare for the season; Employment and finances; Physical environment in the faculty and public transport. Heads of Departments and Faculty can use the findings of this study to provide the necessary support that nursing students seek in reducing the difficulties they face in order to enhance their academic performance.

Introduction
Nursing students face various challenges during their years of study, which may hinder their academic performance. The success of nursing students is particularly important given that they must provide quality care to patients, family members and the community (Ratanasiripong et al. 2021). Academic performance in nursing programs includes advances in theoretical subjects and the acquisition of practical skills in the clinic. A series of studies have shown that during the years of study nursing students face many difficulties that affect their learning skills and consequently, their academic performance. Dante et al. (2013) in a systematic review of the literature regarding academic performance indicators confirmed the complex nature of academic performance for nursing students, arguing that the influencing factors for one student may be very different for another. While Mthimunye & Daniels concluded that nursing students face difficulties such as: part-time employment, off-campus stay, curriculum structure, insufficient academic and clinical support, and unfavorable physical faculty environment (Mthimunye & Daniels 2020). Previous studies have shown that during the years of study, nursing students face many difficulties that affect their learning skills and consequently, their academic performance. By identifying and recognizing the difficulties encountered by students during their years of study, measures can be taken to reduce barriers and improve academic performance (Fooladi et al. 2022).
This study was based on the conceptual model of reducing the gap towards student success, by Perna & Thomas (2006). These challenges can be the result of the interaction of personal, family, school as well as social, economic, and political context (Perna & Thomas, 2006).
This study aims to explore and describe the challenges experienced by nursing students of the Faculty of Technical Medical Sciences (FTMS) of Tirana in relation to their academic performance.

Methodology
This study used a qualitative research approach, such as focus group discussion, to identify difficulties encountered by students in relation to their academic performance. The study was conducted at the Faculty of Technical Medical Sciences (University of Medicine of Tirana, Albania), which prepares nurses in accordance with the Bologna system, Bachelor 3 years and Professional Master one year and Master of Science two years. Qualitative research seeks to provide a detailed explanation of human attitudes, feelings, and perceptions (Tong et al., 2012). This method was useful in this study to explore the difficulties faced by nursing students.
The sample consisted of 51 Bachelor students, who were randomly selected using the name lists of each year of study. Participants were informed that the principle of confidentiality and anonymity would be respected and their involvement in the study was voluntary. During the selection of the sample not all students agreed to participate. 17 participants from the first year, 15 participants from the second year and 19 participants from the third year took part in the research.
Prior to data collection, a pre-test was performed with 4 students, to improve validity in qualitative data collection procedures and interpretation of findings (Brown, Lindenberger, & Bryant, 2008).
Two focus group discussions took place with each study year in the period January-March 2022, and each discussion lasted about 50 minutes. All six focus group discussions took place in a FTMS meeting room.
Focus group discussion is a technique where the researcher discusses with a group of individuals a specific issue, aiming to extract from the complex personal experiences, beliefs, perceptions, and attitudes of the participants through a moderate interaction (Hayward, Simpson, & Wood, 2004).
Data were collected with the help of study assistants. The lead researcher together with colleagues supervised the process and the quality of the collected data. The following questions were asked to obtain the required data on the difficulties faced by nursing students regarding academic performance: -Describe the difficulties that have affected your academic performance encountered in the faculty settings.
-Describe the difficulties that have affected your academic performance encountered in clinical settings.
-What efforts have you made to ensure satisfactory academic performance and achievement?
-What do you think needs to be improved regarding overcoming difficulties that affect your academic performance?
Other investigative questions were used during the discussions to further delve into the problems students encountered during their years of study. The discussion took place in Albanian language, and it was recorded with the permission of the participants by using a recorder. Study assistants kept brief notes about each participant's comments that answered the semi-structured and open-ended questions. Qualitative research and especially focus group interviews generate large amounts of data. Yin (1989) states that data analysis consists of a series of stages such as examination, categorization, and tabulation or recombination of evidence, to address the initial purpose of the study. Data analysis was based on transcribing focus groups, identifying key topics, organizing data, interpreting topics and ideas, and reporting.

Findings
Data analysis generated 194 citations, which were categorized into 3 topics and 11 categories. The categorization of the data and the clarification of the topics was based on the model adapted by Perna and Thomas (2006).

Category 1 Low level of student engagement in clinical practice
Bachelor's nursing students admitted that they were not properly supported by the clinical supervisors employed by the faculty to oversee the students at the clinical premises. In the discussions that took place, most of the second-and third-year students stated that the difficulties they encountered in the clinical environment negatively affected their academic performance. The students admitted that the practice hours are insufficient for them, which affects their practical training. The students stated that: "We are large groups, and we cannot even learn the basics." (FGD 5, BN 3)

Category 2 Curriculum structure
This discussion highlights the challenges directly related to the curriculum experienced by students. They say they have a large number of subjects and exams, and some subjects (in their judgment) are unnecessary for the nursing branch. They declare: "I think we have many subjects and exams, different from other faculties; some of these subjects seem unnecessary to me." (FGD 3, BN 2) "Some subjects should not be included in curriculum, because they are not related to the nursing branch, and unimportant subjects should have less credits." (FGD 6, BN 3) "The curriculum has a big volume and a lot of subjects, there are also subjects with one credit." (FGD 5, BN 3) "The literature and program need to be clearer and more concrete at to what we need as nurses." (FGD 6, BN 3)

Category 3 Educational workload
The transition from high school to university confronts first-year students with the burden and volume of lectures that make it difficult for them to understand and learn.
First year students suggest that it is more effective for them to develop laboratory practice in smaller groups and that the large number of subjects is a barrier to learning. Some of the students' sayings are: "I had difficulties at the beginning of the academic year with the adjustment regarding the volume of material received to find the right method of study and to manage time." (FGD 5, BN 3) "The lectures seem very long to me compared to high school and it is difficult to learn." (FGD 2, BN 1) "Anatomy is very difficult, we should go in smaller groups to the laboratory to learn." (FGD 1, BN 1)

Category 4 Organizing the exam season and scheduling
Students state that the exam season is short and insufficient to prepare for a large number of subjects. They say: "We have a very large number of exams and the time between exams is very short, which we find impossible to cope with and very stressful." (FGD 3, BN 2)

Category 5 Teaching methods
Students say that some of the professors do not use interactive methods during the lessons, which leaves the student passive only by listening and does not help them in acquiring knowledge, resulting in loss of interest. Some of their statements are: "I lose interest when certain professors start reading the material on the laptop without giving concrete examples to make it more attractive, this is a matter of concern not only to me, but also to my friends." (FGD 1, BN 1) "Lecturers should develop the lesson more as a conversation… we listen to the lecturer for 2 or 3 hours. I think they should concretize the subject with concrete examples from real life. " (FGD 2, BN 1) "Some lecturers only read the slides and do not explain to concretize what they say." (FGD 5, BN 3)

Category 6 Student Assessment
Regarding the assessment of knowledge by lecturers, students say that they face many difficulties and lack of transparency, which has reduced their motivation. Some of them say that the evaluation is not based on the merit of everyone.
"When I complained about the grade because I rightly deserved more, the professor, pressuring me, said that he could lower my grade even more." (FGD 5, BN 3) "What has influenced me is the fact that you work, study, get tired and in the end, you are evaluated less than you deserve… because there is a prejudice that not all students in a class could get grade 9 or 10." (FGD 3, BN 2) "… Our answers in seminars and course assignments are not considered in the final evaluation." (FGD 4, BN 2 "We are evaluated unfairly in exams, and we are not given the opportunity to see the exam thesis." (FGD 6, BN 3)

Category 7 Physical facilities of the faculty
Another concern mentioned by the students were the conditions at the faculty. They say that they did not have the minimum conditions such as. heating, elevator etc.
"Conditions are not good in the faculty, during the winter we did not have heating… only the boiler was not enough, the toilets do not even have the minimum hygiene." (FGD 4, BN 2) "We are on the fifth floor, and we lack the elevator, heating, reel, etc." (FGD 6, BN 3)

Category 1 Economic situation
Some students said that they face financial difficulties that force them to work during their studies. They show that the school fee and the prices of rented houses are very expensive. According to students: "School fees are expensive compared to salaries in Albania." (FGD 4, BN 2) "Living is very expensive and I have to work… after school I have to go to work and come back in the evening… and I prepare only 1 or 2 days before the exam, which is not enough, but I am obliged." (FGD 2, BN 1)

Category 2 Transportation
Despite of reduced transportation fares for students, urban transportation remains a concern for them. According to students: "One of the most negative experiences for me has been transport, as I live in the area of Astir and it is far; I cannot reach the faculty to attend the first hour due to traffic." (FGD 3, BN 2) "For me, the distance from the family, the transport that costs and takes a long time due to traffic are obstacles for the results." (FGD 1, BN 1)

Category 1 Student Efforts
One of the basic factors of academic performance is the efforts of the students themselves. During the study period the priority of students is learning to successfully complete university. Students have given due importance to some of the factors such as time planning, systematic study, utilization of other resources to meet the needs of learning such as the Internet and health institutions. They claim that: "I have prepared regularly by learning every lesson despite the fact that it was not worth, because I got lower grades than the other students, who cheated at the exam".

Category 2 Student suggestions for improving the situation
Students think that curricula and educational load should be improved; more attention should be paid to clinical practice, students should be motivated, teaching should be improved. They identify and suggest: "The exams we take in the season (at the end of the semester) should be reduced because they are too many." (FGD 3, BN 2) "Lecturers need to evaluate fairly, and improve teaching." (FGD 4, BN 2) "Lecturers should be more transparent with the student and create opportunities to develop more regular practices." FGD 6, BN 3 "Reduce the number of not very important subjects, pay attention to practice, the explanation should be more complete." (FGD 5, BN 3)

Discussion
This study found that the difficulties faced by nursing students at FTMS of Tirana in relation to academic performance are as follows: low level of student engagement in clinical practice, large number of courses and exams, insufficient time to prepare for the season, low number of nursing courses, passive teaching methods, unfair and nontransparent assessment, employment and finances, faculty physical facilities and public transport.
Nursing students do not receive adequate clinical support at the facilities of University of Medicine from internship mentors. According to (Baraz et. al. 2015), the clinical learning experience is at the heart of nursing education and also in a study by (Papastavrou et al. 2016), the quality of nursing education depends largely on the quality of clinical experience. Students suggest that more needs to be done in regulating internships performed in hospitals, they seek a more supportive and hospitable learning environment. During clinical practice, students often face challenges, which make them feel vulnerable (Chan 2001); this requires the serious commitment of the faculty to facilitate the practice and create a supportive environment for students. Other studies have argued that instructor incompetence, negative attitudes, and poor student support can cause adverse effects on student academic performance. (Anthony 2011;Sharif 2005).
In the discussions that took place, the second-and third-year students stated that the difficulties they encountered in the clinical environment negatively affected their academic performance. Another learning challenge from the perspective of most of nursing students, who participated in this study, was the lack of reliability of clinical evaluation. This is because these assessments were not able to properly assess students' scientific knowledge, practical skills, and professional conduct. This result is consistent with the results of the study of Bourbonnais et al. (2008). The relationship between faculty and student is an important part of student success during clinical practice (Foreman et al. 2016). The environment where students study and practice include important aspects such as social climate, care activities, communication, and interpersonal relationships as well as reflection (Kamphinda & Chilemba, 2019).
According to the students, the lecturer-student communication and cooperation is not at the right level and that this leads to an increase in their distrust regarding the veracity of the professional evaluation and judgment. A considerable number of the students express their concern about the assessment, the transparency of the lecturer regarding the assessments and it often seems that the grade is a "strong weapon" in the hands of the lecturer.
In a study in Iran, Cheraghi came to the conclusion that during the learning process regarding the acquisition of knowledge, there was not much attention to students and aspects of learning (Cheragi et al. 2007).
Assessment methods can have their share of negative impact on quality student performance as they often result in a large amount of stress (Wang and Yeh, 2005).
Both Boulton and O'Connell (2017) noted that the amount and difficulty of learning material in the nursing program is a common stress for nursing students.
Jeffreys (2012) discussed study skills as an important factor influencing academic performance. He noted that reading skills, note-taking should also be accompanied by attitudes of accepting responsibility for study, activities such as self-direction, willingness to adapt and detailed plans.
The internal context of the students is related to the approaches, the student's behavior in relation to the chosen vocational education. They claim that they seek higher results, try to learn more by using all possible resources (internet, additional literature) to supplement their knowledge in the field of nursing. They have organized the day; they have chosen to attend classes regularly and some even admit that they learn in groups to improve academic performance.
Employment and finance have a major impact on students' academic experience, according to the UCE Center for Quality Research (CRQ). This study found that students are under pressure to afford living, the cost of renting housing, paying for public transportation, educational materials, and are forced to work part-time by not devoting enough time to lessons. In the National Student Engagement Survey (NSSE) in 2012, students were asked how finances affected their academic performance. The results showed that 60% of students said this interfered with their performance negatively. This proves that work and financial worries lead to fatigue and lack of concentration which endanger the performance of students in universities (Al-Muslimawi & Hamid 2019).

Conclusions and recommendations
The purpose of this study was to explore and describe the challenges experienced by nursing students of FTMS of Tirana in relation to academic performance. Low level of students engagement in clinical practice, large number of exams, insufficient time to prepare for the season, low number of nursing courses, passive teaching methods, unfair and non-transparent evaluation, employment and finances, physical facilities of the faculty and public transport are some of the difficulties encountered by FTMS nursing students in relation to academic performance. These findings indicate that measures should be taken to help students cope with difficulties that hinder their academic performance. Heads of Departments and Faculty can use the findings of this study to provide the necessary support that nursing students seek in reducing the difficulties they face to enhance their academic performance.


s2fieldsofstudy: Education Psychology

text:
Validation of the Students ’ Cognitive Engagement Measure : Evidence from University Students of Karachi

The goal of the study is to determine the validation of the Cognitive Engagement measure of students in the context of Pakistan. The rationale for such an investigation is that while enrollments in Pakistani higher education are increasing, passing out numbers are very low; this problem necessitates an investigation of student cognitive engagement as well as other factors. Because the Construct of Engagement arose in response to the issue of lower student turnout. The Cognitive Engagement Subscales of Attitudes towards Mathematics (ATM) were used to collect data for this study, and the convenience sampling technique was used. The sample included 528 university students from ten different universities in Karachi. The Structural Equation Modeling (SEM) approach was used for data analysis. The results revealed 21 items that were compatible with the Pakistani context, with Shallow Strategy Use (SSU) being the most endorsed subscale with a mean of 3.7. This study provided a validated instrument for future studies to determine cognitive engagement levels. Future research should look into other disciplines and the relationship with other variables.

Introduction
Student engagement is a concept that represents the degree to which a student actively engages in learning activities (Fredricks et al., 2011). The concept of student involvement has received a lot of attention in education research, policy, and practice in recent years. This might be because of its documented connections with desirable scholastic and non-scholastic outcomes such as academic success (Reyes et al., 2012), school completion (Archambault et al., 2013), and physical and psychological well-being (Steele & Fullagar, 2009). Natriello was one of the first researchers to establish a formal explanation of the concept of student involvement (Mosher & MacGowan, 1985). Engagement, according to Natriello (1987), "exists when pupils participate in the activities given as part of the school curriculum.
However, the problem has mostly been explored via the lens of disengagement, which appears as absenteeism (i.e., unexplained absence), indifference (i.e., a low level of effort), and criminality (e.g., cheating, stealing). Several factors such as students' origin, learning environment, and school policy can influence involvement, which in turn affects students' academic achievement and social behaviors (e.g., disrupting classroom activities).
Although Natriello (1987) conceptualization of student involvement as a purely behavioral variable comprised of school participation and behavior was limited, it did open the door for future conversations in the education field. Researchers have sorted out or derived models from motivational theories and models, therefore it is not a novel construct. The notion of Engagement, like motivational theories, has three dimensions: emotional, behavioral, and cognitive.
Researchers have sorted out or derived models from motivational theories and models, therefore it is not a novel construct. The notion of Engagement, like motivational theories, has three dimensions: emotional, behavioral, and cognitive. Interest, assimilation, attentiveness, and active participation in learning activities are all examples of cognitive engagement, as is the desire to learn new things more than academic requirements. Because engaging or disengaging students cognitively affects their behavior and academic engagement (Fredrick's & Paris, 2004). Cognitive engagement plays a dual role. In the last ten years, researchers have begun to focus on the concept of student engagement.
University enrolment in Pakistan started increasing since 2001-2002 in the subsequent years; during 2010-2015 Pakistani higher education witnessed a 78 percent increase in the number of universities and degree-awarding institutions, consequently, student enrollment increased profoundly by 174 percent (HEC, 2015). The main reason was a change in government policies towards higher education. The private sector was provided incentives to enter (Economic Survey of Pakistan, 2015) and mushroom growth of private universities started. That increase in the number of universities led to an increase in enrollment to great extent; nevertheless, the enrollment rate is still lower than in other South Asian countries (HEC, 2015). See Table 1 for detailed enrollment information of respondents. However, it evident from statistics available in different HEC publications is that student turnout has been decreasing (Sultana, 2017;Muslim et al., 2017;Zaheer et al., 2016). This could be due to higher dropout rates (Khanam et al., 2016) or low student achievement. Moreover, since proper planning and long-term perspective was not kept into consideration, this increase in human capital has not significantly contributed to economic development (Asghar & Zahra, 2012). Such a problem "exerts a harmful effect on the investment in education by creating a non-productive and non-innovative environment" (Khanam et al., 2016).
This issue of lower student turns out necessitates an investigation of the level of student cognitive engagement as well as other factors that contribute to the low passing rate. Few studies in Pakistan, have included cognitive engagement aspects in part, such as Ullah et al. (2014), who investigated three cognitive engagement variables: in-depth, surface learning, and monitoring in their study. In Items 21, 22, and 23, Iqbal et al. (2009) included cognitive engagement factors. Similarly, some researchers, for instance (Nomaan et al., 2016;Amir & Kamal, 2011;Nausheen, 2016;Munir & Rehman, 2016), have used the Motivated Strategies for Learning Questionnaire (MSLQ) in their studies, that investigated subscales of cognitive engagement, self-regulation, elaboration, and cognitive strategies, among other variables (Fredricks et al., 2012;Pintrich, 2004). It is worth noting that all Pakistani researchers did not use the term "cognitive engagement." The primary purpose of this study was to confirm the appropriate factor structure of the Cognitive engagement scale in the context of Pakistan, in order to meet the validity and reliability (Schimmenti et al., 2020) by employing structural equation modeling.

Background of Study
Higher education in Pakistan is in a precarious state. It is not because people are inherently inferior in talent or moral values in comparison to any other nation in the world, but because of long colonial rule and imitation of other nations' systems, as well as political imbalance, that have spoiled some of the virtues and brought a bad name to the people's intellectual capacities. In Pakistan's history, university education got off to a shaky start. When Pakistan was founded in 1947, it had only one university: the University of Punjab in Lahore. Karachi University was established in 1950. Universities grew in the years that followed. The Lahore Institution of Management Sciences (LUMS) was the first private university in Pakistan, founded in 1984, followed by the Agha Khan University in 1985. The first significant growth in the number of public universities occurred under the Zulfiqar Ali Bhutto era, from 1971 to 1977 (Hoodbhoy, 2009).

Research Questions
Q.1 Is the Cognitive Engagement scale valid and reliable for Pakistani university students? Q.2 What is the widely used cognitive engagement strategy among Pakistani university students?

SAL and SRL Approach
Although the domain of motivation and learning for higher education research comprises several models and viewpoints "A key distinction in the field has been the contrast between two general perspectives".

• Student Approaches to Learning (SAL) • Information Processing
Most European and Australian researchers opt for the SAL approach whereas the IP approach is more common among North American researchers. It is noteworthy that although various models have been derived from the IP approach in the recent era Self-Regulated approach (SRL) has replaced the IP approach. The rationale behind this replacement is that SRL is more comprehensive as opposed to the IP approach that only includes cognitive factors. Whereas the SRL approach includes motivation and environmental factors as well (Pintrich, 2004).
It appears that different research traditions have become closer in recent years. Researchers from many traditions are now accepting current themes that emphasize active, constructivist, situational, and collaborative components of learning (Lonka et al ., 2004). There is the emergence of another approach that is student engagement, it is not new rather separates the models from both SRL and SAL perspectives. Student engagement has the same three dimensions as motivational theories say, affective, behavioral and cognitive. A clear definition of (CE) is not clearly defined in available literature (Fredricks et al., 2016). Although, researchers following the SAL perspective describe CE in terms of deep and shallow cognitive strategies.

Cognitive Engagement
Because "cognitive and emotional engagement is potentially mediators of academic and behavioral engagement" (Reschly & Christenson, 2012), engaging or disengaging students intellectually and effectively precedes their behavior and academic involvement.
Although there is little study on cognitive engagement, it focuses on broad concepts like thinking and the desire to put in the work required to achieve and learn complicated abilities and ideas. Following are the key components of student cognitive engagement

Self-regulation
Certain key assumptions of self-regulated learning were outlined by Zimmerman (2000). Although there are many different Self-regulation models, Pintrich claims that this assumption is the same in all of them. The first assumption is that learners actively engage in knowledge acquisition, making attempts to use both accessible resources and past knowledge. The next assumption is that a student has some influence over their learning processes and occasionally external circumstances as well, or that learners can self-regulate their learning process to some level (Pintrich, 2004).
The next premise is that a learner has a purpose to achieve and to do so, they manage their cognition, motivation, and behaviour" to govern their self-regulatory, motivational, and behavioral techniques and activities (Vermetten et al., 1999). The last premise is that a learner does not utilize her self-regulated methods just because of her personal history and priorities, nor alone because of the learning environment. To achieve her aim, a student uses self-regulated learning to relate her goals or objectives to classroom surroundings.
H1: The facture structure of the self-regulation sub-scale is sufficiently valid and reliable with the current sample.

Deep Strategy Use
Deep processing is a term used to describe a learner's efforts to foster the formation of progressively sophisticated structures of knowledge. For instance, if a student is using deeper thought while accomplishing any academic task, the students display elevated levels of (CE). This way of thinking has been related to higher academic accomplishment and a better understanding of course material (Entwhisle & Ramsden, 2015;Nolen, 1988;Miller et al., 1996;Garcia & Pintrich,1991;Schunk,1985). Deep processing techniques include rephrasing or summarizing material, usage of drawings or diagrams to help in problem-solving, evaluation of understanding, comparison, and contrast , Hofer et al., 1998.
H2: Facture structure of deep strategy use is sufficiently valid and reliable with the current sample.

Shallow Strategy Use
On the other hand, Shallow engagement, explains a learner's surface-level cognitive participation focused on replicating rather than elaborating on learning content. These tactics are intended at internalizing knowledge as is, rather than adopting metacognitive processes to deepen an individual's grasp of course content. Repetitive repetition and rote memorizing of material, underlining or highlighting text while reading, and reviewing notes are all examples of shallow processing procedures (Zimmerman & Pons, 1986;Miller et al., 1996;Weinstein & Mayer, 1986;Hofer et al., 1998;Meece et al., 1988;Nolen, 1988;Ravindran, et al., 2005).
H3: The facture structure of shallow strategy use is sufficiently valid and reliable with the current sample.

Persistence
Persistence refers determination and hard work of a student towards academic activities in the face of difficulties. High levels of persistence significantly contribute towards the accomplishment of the tasks and/or course completion (Sakurai et al., 2012). Some researchers categorize persistence under behavioral engagement, whereas others as cognitive engagement (Fredricks et al., 2011), such as Miller et al. (1996) referred to persistence as a construct of cognitive engagement. Whereas Skinner and Pitzer (2012) categorized persistence as behavioral engagement.
H4: Facture structure persistence use is sufficiently valid and reliable with the current sample.

Measurement of Cognitive Engagement
Measurement of CE varies in accordance with operational definitions of the construct Cognitive engagement has been defined as (1) perceptions of the importance or value of schooling, learning goals, and prospects; (2) cognitive strategy use (how thoroughly students study material); (3) self-regulatory or metacognitive strategies (how students manage knowledge acquisition such as organizing and seeking information); and (4) doing extra work and going above and beyond schoolwork. Such cognitive engagement measures consider motivation, self-regulated learning, and strategy use. When we go for notions of deep and shallow for CE, we find its roots in the work of Marton (1986) and his fellow researchers in 1970. Lonka et al. (2004) in 1976 Marton and Saljo (1976) coined the terms of deep and shallow learning strategies. To explore the constructs of deep and shallow learning they used qualitative methodology. Marton (1986) developed a qualitative method named phonomyography that uses the lens of students' perspective developing through the interaction of students experience and the learning environment (Lonka et al., 2004). In terms of the value of self-report surveys, self-report data has considerably added to our understanding of motivation and cognitive engagement.
The first handbook on student engagement was published in 2012 wherein Fredricks and McColskey (2012) discussed in detail the surveys instruments that include subscales measuring cognitive engagement along with theoretical origins. In 2015, Green reviewed self-report surveys being used over 20 years, she also reviewed the studies that do not mention the CE but the work they have done falls under the umbrella term CE. Although Greene et al.(2004) follows the SAL school of thought in her review, she brought under discussion the self-report surveys from both SAL and SRL approaches. Such as details about the Motivated Strategies for Learning Questionnaire (MSLQ) developed by Pintrich (1991) belong to the SRL approach.

Research Design
A descriptive method has been chosen for the current study. Descriptive research is defined as a research method that describes the characteristics of the population or phenomenon studied. This methodology focuses more on the "what" of the research subject than the "why" of the research subject (Siedlecki, 2020). However, the study's research topics include the description of university students' fundamental ideas and cognitive activities. Descriptive research is undertaken in many educational settings to "describe, compare, contrast, classify, analyze, and interpret the entities and occurrences" (Cohen et al., 2011) concerning people, practices, or content. A common descriptive method is survey design.

Measure
This study used Cognitive Engagement subscales of Attitudes towards Mathematics (ATM) an extensively validated measure in different cultures and subject domains (Fredricks & McColskey, 2012). As far as the utility of self-report surveys is concerned "self-report data have made significant and important contributions to the understanding of motivation and cognitive engagement" (Greene et al., 2004). This measure consisted of four subscales: self-regulation (9 items), deep strategy use (7 items), shallow strategy use (4 items), and persistence (8 items). Comprises 5-point Likert scale ranging from strongly disagree to strongly agree.
The Cognitive Engagement subscales of the Attitudes towards Mathematics (ATM) were used in this study, which is a widely validated measure across cultures and subject domains (Fredricks & McColskey, 2012). In terms of utility, "self-report data have made significant and important contributions to the understanding of motivation and cognitive engagement".
This scale had four subscales: self-regulation (9 items), deep strategy use (7 items), shallow strategy use (4 items), and persistence (8 items). A 5-point Likert scale ranging from strongly disagree to strongly agree is used.

Sample
The population of this study comprises all university students of Pakistan. For the current study, the convenience sampling technique has been employed, although it is a non-probability sampling technique still extensively used in social sciences research. The results of the convenience sample analysis can only be applied to the research participant group. It is critical to note that connections and effects discovered in a convenience sample cannot be applied to a target population. Convenience sampling, on the other hand, is less expensive, faster, and easier than other types of samples. Convenience sampling can be used to create hypotheses and objectives for use in more rigorous research projects when no other sample method is practical (Stratton, 2021).
Moreover, a list of all enrolled students is very difficult to obtain for a trio of researchers with limited time and resources, therefore random sampling was not a feasible option. Data were collected from 6 public and 4 private universities in Karachi city, 600 forms from 650 were returned, 528 were complete and free from common errors. The response rate remained 81%.
In order to confirm the validity of the cognitive engagement scale; exploratory factor analysis was conducted with SPSS software. AMOS is a statistical software programme created by IBM. The Amos programme is specially developed to aid in the testing of hypotheses about the relationship between variables. We may use this programme to determine the strength of the association between variables, including latent and manifest variables. How substantial is the link between variables, and how well does the hypothetical model suit the real-world data? The benefit of Amos is that we don't need a sophisticated syntax or computer language to use it (Purwanto et al., 2021).

Results and Data analysis
Before initiating factor analysis in order to see whether missing data can be imputed or not for the reason being that AMOS requires complete data set for analysis, Missing Completely at Random (MCAR) test was performed, which yielded insignificant for all four variables i.e., self-regulation, deep strategy use, shallow strategy use, and persistence. Insignificant MCAR test refers that missing data pattern is random and can be imputed (Little, 1988). Then expectation-maximization procedure was employed to impute missing data.
Research question two is about the mean level of endorsement of the constructs. It is shown from table 3 that shallow strategy use (SSU) has the highest mean value (3.74) as compared to other constructs of higher-order thinking; self-regulation (3.67), deep strategy use (3.57), and persistence has the lowest value (3.11). It implies that students included in the sample are employing shallow cognitive strategies and have lower rates of persistence. Table 3:

Descriptive Statistics
Initially, exploratory factor analysis was conducted Orthogonal Varimax rotation was used as was in Miller et al. (1996) to scrutinize the dimensionality of factors. The value of Kaiser-Meyer-Olkin (KMO) 0.86 was sufficient to conduct analysis (Leech et al., 2005;Barkus et al., 2006). The Bartlet test of Sphericity was found significant at 0.05, sufficient to reject the null hypothesis. Items having lower loadings (k < 0.30) were deleted (Bandalos & Finney, 2018). A total of 21 out of 28-factor loadings were identified to be sufficient to moderate.
"Internal consistency is the amount to which individuals who answer in one manner to things tend to respond in the same way to other items meant to test the same construct," says the author (Fredricks & McColskey, 2012). If Cronbach's alpha is 0.07 or greater, reliability analysis is acceptable (Leary, 2004). In this study, the values were self-regulation (0.80), DSU (0.73), SSU (0.77), and persistence (0.77). (0.72). The CR values in the table are all more than 0.7, indicating composite measure dependability (Raykov, 1997).

Construct Validity
Confirmatory Factor Analysis was used in the second phase to validate the factor structure since it "is a good method for evaluating construct validity" (Kline, 2005). Different measures were employed to assess model fit, the Comparative Fit Index (CFI), Root Mean Squared Error of Approximation (RAMSEA), Goodness of Fit Index (GFI), and Adjusted Goodness of Fit (AGFI) (Fan & Sivo, 2005), RMSEA (.058) and RMR (0.062) supported model fit (Hu & Bentler,1999), while GFI (.912) and AGFI (.887 >0.8) also indicate good model fit (Kline, 2005;Anderson, 1995), and TLI (.878) and CFI (.89) around 0.90 show satisfactory fit (Kline, 2005;Anderson et al., 1995). Table 4 shows the results. Validated convergent and discriminant validity (Kline, 2005) results appear in table 3. Inter-factor correlations ( Table 3) and figure 1 were determined to be weak to moderate. The results show that SSU is the most approved subscale, with a mean of 3.7.

Discussions and Conclusion
In this study negative relationship was found between persistence and shallow strategy use. This relationship is consistent with literature although not direct instead it is needed to understand the relationships of another construct. There are certain predictors/and or associated variables of shallow or surface-level strategies such as performance goals i.e., learning for the sake of good grades to get appreciation from family, and peers/teachers instead of the zest of learning . In this regard, it is argued students who are inclined towards performance goals, when confront with difficulties they show low persistence and their performance falls, and they are more at risk of dropping out. On the other hand, students interested in learning goals tend towards deep strategy use, self-regulation and even in the face of difficulties show higher levels of persistence, for the reason being that they feel pleasure in learning new things and remain open for challenges as well (Miller et al., 1993).
As far as mean levels of endorsement are concerned shallow cognitive engagement was found most endorsed. This result is very much consistent with what researchers have been indicating in various studies (Iqbal et al., 2010) another study employing a larger sample of 1850 university students found a lack of self-regulation, loss of confidence, and anxiety problems, although judgment cannot be made mere on student's self-reports, without thorough examination this result at least show students' perspective (Saleem & Mahmood, 2013). When students are facing such problems their risk of dropout increases, and mental discomfort decreases their learning and achievement (Greene et al., 2004;Fredricks et al., 2011).
To some extent existence of shallow engagement is of point of concern because students need to memorize a few things, as well as follow certain patterns as for research writing students must follow certain styles. The problem may occur when shallow engagement becomes problematic when it is a predominant construct as compared to other cognitive engagement constructs fall under the deep engagement phenomenon. It is evident from various research studies that deeper level processing i.e., self-regulation, deep strategy use, and persistence is associated with a higher level of performance. Whereas shallow engagement is associated with lack of interest, low achievement, and higher dropout rates (Garcia & Pintrich, 1991;Greene et al., 2004;.

Academic Implications
Cognitive involvement is an essential notion that has received little investigation in Pakistan and is still developing in the international arena. This research will contribute to both Pakistani and international research. Despite the study's shortcomings, it will equip other Pakistani researchers with well-validated tools. In the future, it is suggested that this instrument be improved with a larger sample size to ensure validity, as well as alternative approaches such as observation and teacher-administered surveys.
In this study shallow engagement was found most endorsed construct, to enhance students' deep cognitive engagement it is needed to collaborate students' expectations from learning activities with classroom instructions. Such as introducing topics/lessons in a way that is in line with students' future, to increase student achievement (Greene et al., 2004).

Research Limitations and Future Recommendations
Because this is a survey study, it is impossible to determine individual or small group concerns in educational settings. In the future, researchers may conduct qualitative investigations to provide a more in-depth study of the underlying condition.
It is impossible to guarantee that replies will be supplied honestly or unbiasedly, as with any self-report survey. Furthermore, because this study used convenience sampling, the results cannot be extrapolated to the target group of university students. Random sampling and longitudinal data studies can contribute a more robust piece of knowledge to the cognitive engagement construct.


SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which consists of multiple teacher models with multi-level capacities, to transfer knowledge into student model in an one-to-one manner. Sampling distribution plays an important role in SKD. We heuristically present three types of sampling distributions to assign appropriate probabilities for multi-level teacher models. SKD has two advantages: 1) it can preserve the diversities of multi-level teacher models via stochastically sampling single teacher model in each iteration, and 2) it can also improve the efficacy of knowledge distillation via multi-level teacher models when large capacity gap exists between the teacher model and the student model. Experimental results on GLUE benchmark show that SKDBERT reduces the size of a BERT$_{\rm BASE}$ model by 40% while retaining 99.5% performances of language understanding and being 100% faster.

A Multi-teacher Knowledge Distillation
For BERT-style language model compression, we verify the performance of MTBERT (Wu, Wu, and Huang 2021) whose objection function can be expressed as: where, N indicates the number of teacher models, CE(·, ·) is the cross-entropy loss, T denotes the temperature, y represents the ground-truth label, y i and y s refer to the outputs of i-th teacher model and the student model, respectively. We employ the student model used in Tiny-BERT (Jiao et al. 2020) which can be downloaded from https://huggingface.co/huawei-noah/TinyB ERT General 6L 768D. Furthermore, we choose five BERT-style models whose architecture information is shown in Table 1, as the teacher models from the BERT official website 1 . Furthermore, we fine tune the above teacher models on six downstream tasks of GLUE benchmark, i.e., MRPC, RTE, SST-2, QQP, QNLI and MNLI whose detailed introduction can be found in Section D. Moreover, the experimental settings of teacher models fine-tuning are shown as follows: • Learning Rate: For T 3 and T 4 , {6e-6, 7e-6, 8e-6, 9e-6} on MRPC and RTE tasks, and {2e-5, 3e-5, 4e-5, 5e-5} on other tasks. For other teacher models, {2e-5, 3e-5, 4e-5, 5e-5} on all tasks.
• Batch Size: {32} for teacher model fine-tuning on each task.
Other settings are following TinyBERT (Jiao et al. 2020). Subsequently, we employ the ensemble of the fine-tuned teacher models to compress the student model via (1) . Particularly, we only use the weighted multi-teacher distillation loss without the multi-teacher hidden loss and the taskspecific loss as in MT-BERT (Wu, Wu, and Huang 2021).

B SKD for Image Classification
To verify the effectiveness of general SKD on computer vision, we conduct a list of experiments on CIFAR-100 (Krizhevsky, Hinton et al. 2009) image classification dataset. Moreover, we employ ResNet (He et al. 2016) and Wide ResNet (WRN) (Zagoruyko and Komodakis 2016) with various architectures as the candidates of teacher team to distill a student model. Following the experimental settings in CRD (Tian, Krishnan, and Isola 2020), we train the student model and teacher model for 240 epochs, and employ SGD as the optimizer with a batch size of 64, a learning rate of 0.05 which is decayed by a factor of 0.1 when arriving at 150-th, 180-th, 210-th epoch and a weight decay of 5e-4, and show the results in Table 2. Similar to Tian, Krishnan, and Isola (2020), we show the test accuracy of the last epoch in Table 3 for a fair comparison. The proposed SKD also achieves novel performance on CIFAR-100. Compared to state-of-the-art CRD, our approach achieves the best performance in five out of six groups of distillation experiments.

C Why Does SKD Work?
In this section, taking uniform distribution based SKD as an example, we discuss why does it work.

C.1 Preliminaries: Knowledge Distillation
On classification task with C categories, vanilla KD employs the output of final layer, logits z z z = [z 1 , z 2 , . . . , z C ] T , to yield soft targets p p p = [p 1 , p 2 , . . . , p C ] T for knowledge transfer as where, T represents the temperature factor which is used to control the importance of p i . The total loss L with respect to input x x x and weights of student model W W W , consists of distilled loss L d and student loss L s as and L s (y y y, p(z z z s , T )) = C i=1 −y i ln(p(z si , T )) treat soft targets p p p(z z z t , T ) and ground truth y y y as their labels, respectively. Moreover, z z z t and z z z s represent the logits of teacher and student models, respectively. Particularly, an ensemble of multiple teacher models is used to provide

C.2 The Convergence of SKD
In order to optimize W W W , we should evaluate the crossentropy gradient of total loss with regard to z si , Here, the second term of the right side of (2) of SKD is identical with vanilla KD, so that we only discuss the first term. The cross-entropy gradient of distilled loss with regard to z si can be computed as For ∂ln(p(zsj ,T )) ∂zsi , there are two situations (i.e., i = j and Subsequently, we replace ∂ln(p(zsj ,T )) ∂zsi in (3) with (4), and it can be rewritten as On the one hand, in vanilla multi-teacher KD framework with an ensemble of N teacher models, z ti is the average of ensemble as so that its soft target can be computed as .
On the other hand, SKD randomly employs a single teacher from N teacher models to obtain the soft target at n-th iteration as where, the sampled probability of z z z (n) t subjects to uniform distribution U (1, N ).
We assume that the number of iterations is L. In the total training process of multi-teacher KD, the amount of gradient update on z i dubbed G (i) Avg can be represented as Beyond that, the amount of gradient update of SKD on z i dubbed G (i) SKD can be obtained by Obviously, the main difference between G  (5) as According to (7) and the term b i in (6), we can find that their relationship is similar to the relationship between Batch Gradient Descent (BGD) and Stochastic Gradient Descent (SGD). Moreover, BGD employs total data to obtain the gradient in each iteration, but SGD uses single one. Based on stochastic approximation (e.g. ROBBINS-MONRO's theory (Robbins and Monro 1951)) and convex optimization, the convergence of SGD has been proven (Turinici 2021). MRPC belongs to a sentence similarity task where system aims to identify the paraphrase/semantic equivalence relationship between two sentences.

D Details of GLUE Benchmark
RTE belongs to a natural language inference task where system aims to recognize the entailment relationship of given two text fragments.
CoLA belongs to a single-sentence task where system aims to predict the grammatical correctness of an English sentence.
STS-B belongs to a sentence similarity task where system aims to evaluate the similarity of two pieces of texts by a score from 1 to 5.
SST-2 belongs to a single-sentence task where system aims to predict the sentiment of movie reviews.
QQP belongs to a sentence similarity task where system aims to identify the semantical equivalence of two questions from the website Quora.
QNLI belongs to a natural language inference task where system aims to recognize that for a given pair <question, context>, the answer to the question whether contains in the context.
MNLI belongs to a natural language inference task where system aims to predict the possible relationships (i.e., entailment, contradiction and neutral) of hypothesis with regard to premise for a given pair <premise, hypothesis>,.
WNLI belongs to a natural language inference task where system aims to determine the referent of a sentence's pronoun from a list of choices.

E Evaluation Metrics and Hyper-parameters E.1 Evaluation Metrics
In order to choose the best model, we use appropriate metrics on GLUE-dev as shown in Table 4.

E.2 Fine-tuning Hyper-parameters
In this paper, we need to perform teacher model fine-tuning, student model fine-tuning and student model distillation. For the above three cases, the used hyper-parameters are shown in Table 5.

F Student Model and Teacher Team Details
The architecture information of each student model and teacher model is shown in Table 6. On the one hand, we direct treating the pre-trained model of TinyBERT 4 2 and 2 https://huggingface.co/huawei-noah/TinyBE RT General 4L 312D  TinyBERT 6 3 as the student models of SKDBERT. On the other hand, we choose 14 BERT models with various capabilities as the candidates for teacher team. Moreover, each pre-trained teacher model can be downloaded from official implementation of BERT 4 . Furthermore, the results of the student model and the teacher model on GLUE-dev are shown in Table 7 which can be used to obtain the teacherrank sampling distribution.

G Distillation Performance of SKDBERT with Various Teacher Models
In order to determine the student-rank sampling distribution for SKD, we employ each teacher model to distill  SKDBERT 4 and SKDBERT 6 on GLUE-dev, and show the results in Table 8.

H The Used Sampling Distributions for SKDBERT with Various Teacher Teams on The GLUE Benchmark
We implement extensive ablation experiments for SKD-BERT using various teacher teams: {T 04 -T 06 , T 07 -T 09 , T 10 -T 12 , T 12 -T 14 } for SKDBERT 4 and {T 04 -T 06 , T 07 -T 09 , T 10 -T 12 , T 04 -T 09 , T 04 -T 12 , T 01 -T 14 , T 04 -T 14 , T 07 -T 14 , T 09 -T 14 , T 12 -T 14 , T 13 -T 14 } for SKDBERT 6 . We show the used dis-tributions for the above teacher teams on various tasks in Figure 1 to 15. Particular, 0 indicates that the teacher models are not employed for stochastic knowledge distillation.