Effect of head-mounted displays on students’ acquisition of surgical suturing techniques compared to an e-learning and tutor-led course: a randomized controlled trial
Learning to perform surgical sutures is one of the most important basic manual skills for prospective surgeons, as well as for physicians and dentists who perform surgical procedures , . Unfortunately, many medical school graduates do not acquire adequate surgical skills during their studies , , . Furthermore, not all medical schools integrate suturing into their curriculum . Lack of skills can lead to both anxiety and incompetence and thus potential deficiencies in surgical suturing quality and subsequent patient care , . One of the main reasons for unsatisfactory student evaluation seems to be the discrepancy between the number of students and the availability of tutors . To ensure optimal teaching, a ratio of four students to one tutor is recommended . However, this may pose major challenges to the capacity of teaching staff , who have been further burdened by possible restrictions due to the coronavirus disease 2019 (COVID-19) pandemic , . Although there exist e-learning/computer-based programs, which could help to teach surgical suturing techniques in a time-independent and self-directed manner. Such programs still rely on an instructor to provide external feedback to students, thereby also encroaching on the capacity of the teaching staff and making the programs not truly self-directed , – . Aside from manual dexterity , , the learning experience and visual-spatial ability appear to play crucial roles in the acquisition of surgical skills , . The learning experience results from the totality of influences that affect the trainee during the learning process, such as individualization and the quality of the application or the stimulation of memorization . Additionally, visual-spatial ability, which describes the capability of an individual to perceive two-dimensional objects in three dimensions and to classify their position in space, may play a role . In this regard, previous studies have shown that individuals with low visual-spatial ability perform worse in acquiring surgical skills than individuals with high visual-spatial ability , . The use of virtual reality (VR) in medical education has shown great promise . Different systems are summarized under the term ‘VR’, such as simple (hand-based or screen-based) or enhanced systems (partially or fully immersive). Partially immersive systems include wall projectors, ImmersaDesks, and monocular head-based systems, while fully immersive systems include space-based (vehicle simulation or cave automatic virtual environments) and binocular head-based systems . In recent years, binocular head-mounted display (HMD) VR systems have become popular and widely used in the consumer entertainment market. They could also be used to deliver high-quality learning experiences and to advance the visual-spatial skills of a wide audience by showing immersive, stereoscopic content from a first-person perspective , . Immersive 360° videos displayed using VR-HMDs without stereoscopy have already shown an advantage in acquiring knot-tying skills . In laparoscopic surgery, stereoscopic video systems, compared to non-stereoscopic applications, have been found to enable higher-precision performance and an increase in the speed of execution . Furthermore, stereoscopic learning applications have positive effects in terms of the localization and identification of different structures, such as abdominal anatomical structures, using 3D presentation . This highlights the potential of intensifying learning through HMDs and the impact of visual-spatial ability in teaching surgical skills and knowledge , , . In contrast to VR-HMDs, applications enabled by augmented reality (AR) HMDs have already been demonstrated. AR must be distinguished from VR, which extends reality through holography. One randomized controlled trial (RCT) ( n =14) was able to show that the additional use of monoscopic AR-HMDs (Google Glass) led to more enjoyment in tutor-led teaching of surgical suture techniques . Recently, another RCT ( n =38) for self-directed training with an AR-HMD (HoloLens 2) showed its comparability to a video course . However, among the computer-based approaches, to our knowledge, there has been no investigation into the impact of VR-HMDs in teaching surgical suturing and its learning effects using immersive and stereoscopic learning content recorded from a first-person perspective. Therefore, we compared an HMD-based course (no feedback, stereoscopic) with e-learning (no feedback, mesoscopic) and tutor-led (feedback, stereoscopic) courses, respectively, in a multi-arm RCT. The primary endpoint was an improvement in suture quality (learning effect) among novices. The secondary endpoints were student self-assessed improvement, the correlation between improvement in suture quality and visual-spatial ability, and the number of sutures performed during training. 

In total, 160 medical and dental students enrolled in this RCT’s courses. Three students did not attend the course despite enrollment. Prior training in a surgical suturing course was denied by all attendees; however, 7 students had to be excluded because they were not novices. The RCT was conducted from March to November 2022, including recruitment and follow-up at a single center (University Hospital RWTH Aachen). Ultimately, the final study cohort included 150 novices with no adverse events. No other groups were added or dropped. The study was authorized by the local ethics committee of the University Hospital RWTH Aachen (approval code EK 352/21 and EK 22-329) and was registered, including the study protocol, in the German Clinical Trials Register (DRKS00029307). The reporting of the study followed the extension of the CONSORT 2010 statement for reporting multi-arm parallel-group randomized trials (Fig. ). Course and evaluation The cohort was stratified by study program (medicine or dentistry) and randomly divided into three trial arms by block randomization (block sizes of 3, 6, and 9). The random assignment sequence was generated by a statistician. Recruitment was done by P.P., and assignment to the intervention groups was done by B.P. There were no changes in the allocation of the groups. After course assignment and obtaining consent to participate, the visual-spatial ability of the subjects was measured with a Mental Rotation Test (MRT) . MRT was used to analyze the relationship described in the literature for other surgical tasks that involve a relationship between visual-spatial ability and surgical learning. Afterward, the students were given brief written instructions on how to perform the single-button suture and were asked to perform as many single-button sutures as possible within 5 min according to the written instructions. The procedure was recorded and documented with a camera (GoPro Hero 5, GoPro Inc., San Mateo, California, USA). A pre-study questionnaire was administered to collect general and demographic data, such as age, sex, handedness, visual ability, and previous experience with HMDs (Fig. ). Subsequently, a 40-min theoretical session was conducted. To ensure that the learning content was identical in all three trial arms, a standardized presentation was produced for the tutor-led course with important information on wound healing, suture material, various instruments, the correct execution of the single-button suture, 2D videos on the surgical instruments, suture materials, presentation of the main suturing methods, and how to perform the single-button suture. The 2D instructional video showed a bird’s eye view of the suturing procedure. An online teaching application identical to the standardized presentation made during the tutor-led course for the e-learning course was derived and presented via a learning platform ( https://emedia-medizin.rwth-aachen.de ). The web application (a digital lecture) included texts, pictures and videos, and subjects were provided with a computer to visualize the e-learning program to learn independently and in a self-directed manner (Fig. A). Students on the HMD-based course received a video version of the standardized lecture presentation, which was displayed on a screen. In addition, after a 2-min adaption period, each participant learned independently through an immersive stereoscopic instructional video using the Oculus Quest 2 VR-HMD (Meta Platforms Inc., United States) and the preloaded video player (Meta Quest TV). For this purpose, a 180° stereoscopic video from a first-person perspective that conveyed the correct way to perform the surgical suture with explanations and instructions from the tutor was recorded (Insta360 EVO, Arashi Vision Inc., China) (Fig. C). After that, the participants underwent 60 min of training. Participants of the e-learning course and the HMD-based course were allowed to continue using the respective platforms for as long as necessary during the learning session (Fig. B, D). Only students in the tutor-led course received direct feedback, including instruction, from a tutor (P.P.) (Fig. F). The tutor was an oral surgery resident (dental degree), who gained extensive suturing experience while working as a surgical technician for 5 years. For the suturing procedure, an ethylene vinyl acetate (EVA) patch 4 mm in thickness (Infimedix GmbH, Osteorode, Germany) was provided as a low-fidelity bench model . Ethilon 5-0 (Johnson & Johnson, New Brunswick, New Jersey, US) was used as the suture material. A Hegar needle holder, surgical forceps, and scissors were provided, and disposable gloves were worn by all participants throughout the course. Finally, all participants were asked to perform the demonstrated single-button suture technique again for 5 min under video recording. To maintain the subsequent anonymous evaluation, both the pretest and posttest video recordings included only the work environment so that only the gloves were worn and parts of the participants’ forearms were visible. Finally, all students were surveyed about their attended courses using a questionnaire. The quality of surgical sutures as recorded in the videos was evaluated in a blinded fashion by experienced residents with a dual degree in medicine and dentistry (1× second-last and 2× final year of training) from our department (B.P., M.O., and M.U.) using the Global Rating Score (GRS). The GRS is the summary of eight items (respect for tissue, time and motion, instrument handling, suture technique, procedure of surgery and advance planning, knowledge of specific procedure, quality of final product, and overall performance; see Table ) of the Objective Structured Assessment of Technical Skills (OSATS) and has been used since the late 1990s , . The primary endpoint (ΔGRS) was calculated as the difference between pretest and posttest skill and was based on the mean of three scores to account for interrater variability. The video allocation table was only available to P.P. until the end of the evaluation. After completion of the study, all data were anonymized according to the study protocol. Several unexpected events occurred. Although prior training in a surgical suturing course was denied by all participants, some students ( n =7; 2 for e-learning, 3 for HMD-based, and 2 for tutor-led courses) had proficient performance (OSATS GRS ≥24) during the pretest, which was against expectations, and they failed to meet the inclusion criteria of being a novice, according to the study protocol (Fig. A–C). Reasons for this may be the result of previous experience in sewing, talent, or not remembering previous suturing training. Due to the anonymous nature of the study, a follow-up survey was not possible. Therefore, we had to exclude these students for the final assessment, who showed a median ∆GRS of 1. For a more objective assessment of surgical skills based on the recorded videos, the optical flow (OF) (corresponding to the movement between individual frames) was calculated at a rate of 1 Hz (every 30th frame) for all videos and summed up. The OF was then divided by the number of sutures to obtain a score. The script (Supplementary Code) was programmed in MATLAB (version 2022b) and run on a workstation (AMD Ryzen 3900X, 64 GB memory and RTX 2080 TI OC, 11 GB memory). Statistical analysis Statistical analyses were performed using the programming language R (version 4.2.2). P -values <0.05 were considered significant. A power analysis performed at a difference of 3 with a standard deviation of 4 and a power of 80% yielded a total sample size of 108 participants. During the peer review of the study, the power was increased to 95%, giving a total sample size of 150 participants, resulting in additional recruitment until the sample size was reached. According to the Shapiro–Wilk test, the primary endpoint (ΔGRS) was not normally distributed, and because of the ordinal nature of the GRS, a Kruskal–Wallis test was performed. A pairwise Mann–Whitney U test was used for subanalysis. Adjustment for multiplicity was applied according to Hochberg’s method . The correlation between the secondary endpoints and the influence of the number of sutures performed during training and MRT was examined using linear regression. Self-assessment was examined using Fisher’s exact test. Parametric and nonparametric tests, as well as linear regression, were used to examine the additional data in an explorative manner. In addition, linear regression analysis was performed to investigate the relationship between GRS and the number of sutures performed during training. Fisher’s exact test or the χ 2 test was used to test group distributions. 

The cohort was stratified by study program (medicine or dentistry) and randomly divided into three trial arms by block randomization (block sizes of 3, 6, and 9). The random assignment sequence was generated by a statistician. Recruitment was done by P.P., and assignment to the intervention groups was done by B.P. There were no changes in the allocation of the groups. After course assignment and obtaining consent to participate, the visual-spatial ability of the subjects was measured with a Mental Rotation Test (MRT) . MRT was used to analyze the relationship described in the literature for other surgical tasks that involve a relationship between visual-spatial ability and surgical learning. Afterward, the students were given brief written instructions on how to perform the single-button suture and were asked to perform as many single-button sutures as possible within 5 min according to the written instructions. The procedure was recorded and documented with a camera (GoPro Hero 5, GoPro Inc., San Mateo, California, USA). A pre-study questionnaire was administered to collect general and demographic data, such as age, sex, handedness, visual ability, and previous experience with HMDs (Fig. ). Subsequently, a 40-min theoretical session was conducted. To ensure that the learning content was identical in all three trial arms, a standardized presentation was produced for the tutor-led course with important information on wound healing, suture material, various instruments, the correct execution of the single-button suture, 2D videos on the surgical instruments, suture materials, presentation of the main suturing methods, and how to perform the single-button suture. The 2D instructional video showed a bird’s eye view of the suturing procedure. An online teaching application identical to the standardized presentation made during the tutor-led course for the e-learning course was derived and presented via a learning platform ( https://emedia-medizin.rwth-aachen.de ). The web application (a digital lecture) included texts, pictures and videos, and subjects were provided with a computer to visualize the e-learning program to learn independently and in a self-directed manner (Fig. A). Students on the HMD-based course received a video version of the standardized lecture presentation, which was displayed on a screen. In addition, after a 2-min adaption period, each participant learned independently through an immersive stereoscopic instructional video using the Oculus Quest 2 VR-HMD (Meta Platforms Inc., United States) and the preloaded video player (Meta Quest TV). For this purpose, a 180° stereoscopic video from a first-person perspective that conveyed the correct way to perform the surgical suture with explanations and instructions from the tutor was recorded (Insta360 EVO, Arashi Vision Inc., China) (Fig. C). After that, the participants underwent 60 min of training. Participants of the e-learning course and the HMD-based course were allowed to continue using the respective platforms for as long as necessary during the learning session (Fig. B, D). Only students in the tutor-led course received direct feedback, including instruction, from a tutor (P.P.) (Fig. F). The tutor was an oral surgery resident (dental degree), who gained extensive suturing experience while working as a surgical technician for 5 years. For the suturing procedure, an ethylene vinyl acetate (EVA) patch 4 mm in thickness (Infimedix GmbH, Osteorode, Germany) was provided as a low-fidelity bench model . Ethilon 5-0 (Johnson & Johnson, New Brunswick, New Jersey, US) was used as the suture material. A Hegar needle holder, surgical forceps, and scissors were provided, and disposable gloves were worn by all participants throughout the course. Finally, all participants were asked to perform the demonstrated single-button suture technique again for 5 min under video recording. To maintain the subsequent anonymous evaluation, both the pretest and posttest video recordings included only the work environment so that only the gloves were worn and parts of the participants’ forearms were visible. Finally, all students were surveyed about their attended courses using a questionnaire. The quality of surgical sutures as recorded in the videos was evaluated in a blinded fashion by experienced residents with a dual degree in medicine and dentistry (1× second-last and 2× final year of training) from our department (B.P., M.O., and M.U.) using the Global Rating Score (GRS). The GRS is the summary of eight items (respect for tissue, time and motion, instrument handling, suture technique, procedure of surgery and advance planning, knowledge of specific procedure, quality of final product, and overall performance; see Table ) of the Objective Structured Assessment of Technical Skills (OSATS) and has been used since the late 1990s , . The primary endpoint (ΔGRS) was calculated as the difference between pretest and posttest skill and was based on the mean of three scores to account for interrater variability. The video allocation table was only available to P.P. until the end of the evaluation. After completion of the study, all data were anonymized according to the study protocol. Several unexpected events occurred. Although prior training in a surgical suturing course was denied by all participants, some students ( n =7; 2 for e-learning, 3 for HMD-based, and 2 for tutor-led courses) had proficient performance (OSATS GRS ≥24) during the pretest, which was against expectations, and they failed to meet the inclusion criteria of being a novice, according to the study protocol (Fig. A–C). Reasons for this may be the result of previous experience in sewing, talent, or not remembering previous suturing training. Due to the anonymous nature of the study, a follow-up survey was not possible. Therefore, we had to exclude these students for the final assessment, who showed a median ∆GRS of 1. For a more objective assessment of surgical skills based on the recorded videos, the optical flow (OF) (corresponding to the movement between individual frames) was calculated at a rate of 1 Hz (every 30th frame) for all videos and summed up. The OF was then divided by the number of sutures to obtain a score. The script (Supplementary Code) was programmed in MATLAB (version 2022b) and run on a workstation (AMD Ryzen 3900X, 64 GB memory and RTX 2080 TI OC, 11 GB memory). 

Statistical analyses were performed using the programming language R (version 4.2.2). P -values <0.05 were considered significant. A power analysis performed at a difference of 3 with a standard deviation of 4 and a power of 80% yielded a total sample size of 108 participants. During the peer review of the study, the power was increased to 95%, giving a total sample size of 150 participants, resulting in additional recruitment until the sample size was reached. According to the Shapiro–Wilk test, the primary endpoint (ΔGRS) was not normally distributed, and because of the ordinal nature of the GRS, a Kruskal–Wallis test was performed. A pairwise Mann–Whitney U test was used for subanalysis. Adjustment for multiplicity was applied according to Hochberg’s method . The correlation between the secondary endpoints and the influence of the number of sutures performed during training and MRT was examined using linear regression. Self-assessment was examined using Fisher’s exact test. Parametric and nonparametric tests, as well as linear regression, were used to examine the additional data in an explorative manner. In addition, linear regression analysis was performed to investigate the relationship between GRS and the number of sutures performed during training. Fisher’s exact test or the χ 2 test was used to test group distributions. 

A total of 150 novices, including 84 medical students and 66 dental students, participated in the study. This group included 109 females and 41 males. Finally, 51 students participated in the e-learning course, 49 students in the HMD-based course, and 50 students in the tutor-led course. On average, participants had an age of 22.2 years and were second-year students, with 62.7% of students reporting prior experience with stereoscopic media. Prior use of HMDs was affirmed by 14.6% of respondents. There were no observable systematic differences between the groups (Table ). Suturing performance The correlation in the GRS between the three investigators in the pairwise comparison was strong, with a Pearson’s correlation coefficient between 0.8 and 0.83. This is consistent with the pooled interrater correlation for OSATS GRS of 0.8 found in the literature [95% confidence interval (CI): 0.72–0.86] . According to the students’ self-assessments, the HMD-based course was comparable to the tutor-led course and significantly better than the e-learning course (Mann–Whitney U test, P =0.003). In this regard, 37% of students in the HMD-based course, 22% in the tutor-led course, and only 10% in the e-learning course reported that they had improved beyond their expectations (Fig. A). In contrast, the median ΔGRS was 12.4 (95% CI: 10.0–12.7) for the e-learning course, 14.1 (95% CI: 13.0–15.0) for the HMD-based course, and 12.7 (95% CI: 10.3–14.2) for the tutor-led course. In the subgroup analysis, the unadjusted ΔGRS of the HMD-based course was significantly higher than that of the e-learning course (Mann–Whitney U test, P =0.039), which was not the case after correction for multiple testing (Mann–Whitney U test, adjusted P =0.114; Fig. C). Therefore, the different course modalities were comparable (Kruskal–Wallis test, P =0.15; Fig. C). However, a multivariable analysis (adjusted for group, age, and sex) showed that ΔGRS was significantly correlated with the number of sutures practiced during the training time [ β =0.15±0.49 (estimate±SD), P =0.002] and with the number of sutures in the pretest ( β =−1.80±0.29, P <0.001). In a subsequent multivariable analysis (adjusted for age and pretest GRS), there was a significant correlation with the male sex ( β =−2.85±1.27, P =0.028), the HMD-based ( β =3.12±1.39, P =0.026) and tutor-led courses ( β =2.76±1.37, P =0.046), and the number of sutures achieved in the pretest ( β =2.03±0.74, P =0.007). Despite this, 31.4% [ n =16, risk ratio (RR) 1.43, 95% CI: 0.60–2.43, P =0.37] of students in the e-learning course, 26.5% ( n =13, RR 1.21, 95% CI: 0.74–2.76, P =0.64) in the HMD-based course, and 22% ( n =11, RR=1.0) in the tutor-led course did not achieve a competent GRS (≥24) in the posttest (Fig. D). Correlations were found between the ΔGRS and the achieved GRS in the pretest ( R 2 =0.22, P <0.001; Fig. A) and the achieved number of sutures in the pretest ( R 2 =0.15, P <0.001; Fig. B). In contrast, no correlation was shown between ΔGRS and MRT ( R 2 <0.01, P =0.615; Fig. C). The number of sutures performed during the training period affected the posttest GRS ( R 2 =0.18, P <0.001; Fig. D) and the number of sutures in the posttest ( R 2 =0.46, P <0.001; Fig. E). Additionally, a correlation between the OF per suture in the posttest could be observed ( R 2 =0.21, P <0.001; Fig. F). In an additional computerized analysis, we extracted the OF from the recorded videos, revealing a linear correlation between the GRS and the sutures performed in each test ( R 2 =0.73, P <0.001), as well as a weak correlation with the OF ( R 2 =0.16, P <0.001). Interestingly, there was a strong decrease in OF per suture performed ( R 2 =0.61, P <0.001), starting with an increase in the GRS (Fig. ). Qualitative evaluation The students considered a surgical sutures course to be very important ( x̄ =4.9, 1 ≙ strongly disagree, 5 ≙ strongly agree) and agreed that it should be firmly integrated into the curriculum ( x̄ =4.9). The impairment in the acquisition of manual skills as a result of the COVID-19 pandemic was rated high ( x̄ =4.0). Text-only instructions and explanations of how to perform surgical sutures were generally disliked ( x̄ =1.3), and students rated the added value of watching an experienced surgeon or trained instructor perform the surgical procedure as high ( x̄ =4.4). In this context, live demonstrations ( x̄ =4.7) were rated to be slightly more helpful for the acquisition of manual skills than 2D videos or stereoscopic 180° videos (both x̄ =4.4). Furthermore, live demonstrations were considered the most important factor for teaching manual skills and should be permanently integrated into the curriculum (Table ). In general, a tutor-led course was rated significantly better in its ability to teach surgical suturing techniques ( x̄ =4.8). However, an HMD-based course ( x̄ =3.6) was perceived to be clearly better than an e-learning course ( x̄ =2.5) (Mann–Whitney U test, P <0.001; Table ; Fig. B). The actual course format was rated positively in terms of the amount of information disseminated and the length ( x̄ =4.6), the usefulness of the instructions ( x̄ =4.0), and understandability ( x̄ =4.6) for all three course types, with the presentation or live demonstration scoring the highest. The courses were rated good in terms of the memorability of the information conveyed ( x̄ =4.8 and 4.5, respectively) (Table ). In the self-directed courses, significantly more students in the e-learning course (47.1%, n =24) compared to the HMD-based course (18.4%, n =9) felt that feedback would improve learning ( χ 2 =9.30; P =0.002). Regarding perspective, students in tutor-led courses most frequently chose the opposite perspective (48%) or the side perspective (42%), followed by the over-the-shoulder perspective (10%). The chosen perspective on the course content in tutor-led ( x̄ =4.6) and e-learning courses ( x̄ =4.7) was rated as good; however, the HMD-based course received a lower rating ( x̄ =4.0). A perspective from the opposite was rated best by the students who participated in the tutor-led course ( x̄ =4.0), while agreement was lower in the HMD-based ( x̄ =3.0) and e-learning ( x̄ =2.7) courses. For all course types, a high potential of putting oneself in the position of the practitioner was reported ( x̄ =4.2–4.5). Denial of discomfort, fatigue, dizziness or nausea was similar in the corresponding courses ( x̄ =1.4–1.6) and thus comparable (Table ). Regarding the use of HMDs, students stated a preference for the use of HMDs in university teaching in private settings ( x̄ =4.4 vs. x̄ =3.2). Wearing HMDs in the presence of others ( x̄ =4.2) and being shielded from the environment was rated as a positive experience ( x̄ =4.3) and therefore perceived as an improvement in concentration ( x̄ =4.0). However, the use of the HMD for a longer period of time (more than 30 min) was rated as neutral ( x̄ =3.3). 

The correlation in the GRS between the three investigators in the pairwise comparison was strong, with a Pearson’s correlation coefficient between 0.8 and 0.83. This is consistent with the pooled interrater correlation for OSATS GRS of 0.8 found in the literature [95% confidence interval (CI): 0.72–0.86] . According to the students’ self-assessments, the HMD-based course was comparable to the tutor-led course and significantly better than the e-learning course (Mann–Whitney U test, P =0.003). In this regard, 37% of students in the HMD-based course, 22% in the tutor-led course, and only 10% in the e-learning course reported that they had improved beyond their expectations (Fig. A). In contrast, the median ΔGRS was 12.4 (95% CI: 10.0–12.7) for the e-learning course, 14.1 (95% CI: 13.0–15.0) for the HMD-based course, and 12.7 (95% CI: 10.3–14.2) for the tutor-led course. In the subgroup analysis, the unadjusted ΔGRS of the HMD-based course was significantly higher than that of the e-learning course (Mann–Whitney U test, P =0.039), which was not the case after correction for multiple testing (Mann–Whitney U test, adjusted P =0.114; Fig. C). Therefore, the different course modalities were comparable (Kruskal–Wallis test, P =0.15; Fig. C). However, a multivariable analysis (adjusted for group, age, and sex) showed that ΔGRS was significantly correlated with the number of sutures practiced during the training time [ β =0.15±0.49 (estimate±SD), P =0.002] and with the number of sutures in the pretest ( β =−1.80±0.29, P <0.001). In a subsequent multivariable analysis (adjusted for age and pretest GRS), there was a significant correlation with the male sex ( β =−2.85±1.27, P =0.028), the HMD-based ( β =3.12±1.39, P =0.026) and tutor-led courses ( β =2.76±1.37, P =0.046), and the number of sutures achieved in the pretest ( β =2.03±0.74, P =0.007). Despite this, 31.4% [ n =16, risk ratio (RR) 1.43, 95% CI: 0.60–2.43, P =0.37] of students in the e-learning course, 26.5% ( n =13, RR 1.21, 95% CI: 0.74–2.76, P =0.64) in the HMD-based course, and 22% ( n =11, RR=1.0) in the tutor-led course did not achieve a competent GRS (≥24) in the posttest (Fig. D). Correlations were found between the ΔGRS and the achieved GRS in the pretest ( R 2 =0.22, P <0.001; Fig. A) and the achieved number of sutures in the pretest ( R 2 =0.15, P <0.001; Fig. B). In contrast, no correlation was shown between ΔGRS and MRT ( R 2 <0.01, P =0.615; Fig. C). The number of sutures performed during the training period affected the posttest GRS ( R 2 =0.18, P <0.001; Fig. D) and the number of sutures in the posttest ( R 2 =0.46, P <0.001; Fig. E). Additionally, a correlation between the OF per suture in the posttest could be observed ( R 2 =0.21, P <0.001; Fig. F). In an additional computerized analysis, we extracted the OF from the recorded videos, revealing a linear correlation between the GRS and the sutures performed in each test ( R 2 =0.73, P <0.001), as well as a weak correlation with the OF ( R 2 =0.16, P <0.001). Interestingly, there was a strong decrease in OF per suture performed ( R 2 =0.61, P <0.001), starting with an increase in the GRS (Fig. ). 

The students considered a surgical sutures course to be very important ( x̄ =4.9, 1 ≙ strongly disagree, 5 ≙ strongly agree) and agreed that it should be firmly integrated into the curriculum ( x̄ =4.9). The impairment in the acquisition of manual skills as a result of the COVID-19 pandemic was rated high ( x̄ =4.0). Text-only instructions and explanations of how to perform surgical sutures were generally disliked ( x̄ =1.3), and students rated the added value of watching an experienced surgeon or trained instructor perform the surgical procedure as high ( x̄ =4.4). In this context, live demonstrations ( x̄ =4.7) were rated to be slightly more helpful for the acquisition of manual skills than 2D videos or stereoscopic 180° videos (both x̄ =4.4). Furthermore, live demonstrations were considered the most important factor for teaching manual skills and should be permanently integrated into the curriculum (Table ). In general, a tutor-led course was rated significantly better in its ability to teach surgical suturing techniques ( x̄ =4.8). However, an HMD-based course ( x̄ =3.6) was perceived to be clearly better than an e-learning course ( x̄ =2.5) (Mann–Whitney U test, P <0.001; Table ; Fig. B). The actual course format was rated positively in terms of the amount of information disseminated and the length ( x̄ =4.6), the usefulness of the instructions ( x̄ =4.0), and understandability ( x̄ =4.6) for all three course types, with the presentation or live demonstration scoring the highest. The courses were rated good in terms of the memorability of the information conveyed ( x̄ =4.8 and 4.5, respectively) (Table ). In the self-directed courses, significantly more students in the e-learning course (47.1%, n =24) compared to the HMD-based course (18.4%, n =9) felt that feedback would improve learning ( χ 2 =9.30; P =0.002). Regarding perspective, students in tutor-led courses most frequently chose the opposite perspective (48%) or the side perspective (42%), followed by the over-the-shoulder perspective (10%). The chosen perspective on the course content in tutor-led ( x̄ =4.6) and e-learning courses ( x̄ =4.7) was rated as good; however, the HMD-based course received a lower rating ( x̄ =4.0). A perspective from the opposite was rated best by the students who participated in the tutor-led course ( x̄ =4.0), while agreement was lower in the HMD-based ( x̄ =3.0) and e-learning ( x̄ =2.7) courses. For all course types, a high potential of putting oneself in the position of the practitioner was reported ( x̄ =4.2–4.5). Denial of discomfort, fatigue, dizziness or nausea was similar in the corresponding courses ( x̄ =1.4–1.6) and thus comparable (Table ). Regarding the use of HMDs, students stated a preference for the use of HMDs in university teaching in private settings ( x̄ =4.4 vs. x̄ =3.2). Wearing HMDs in the presence of others ( x̄ =4.2) and being shielded from the environment was rated as a positive experience ( x̄ =4.3) and therefore perceived as an improvement in concentration ( x̄ =4.0). However, the use of the HMD for a longer period of time (more than 30 min) was rated as neutral ( x̄ =3.3). 

This RCT compared the benefits of using HMDs to teach surgical sutures against an e-learning course and the gold standard of a tutor-led course and included, to our knowledge, the highest number of participants ( n =150) among published RCT studies on this topic . Despite their distinctly different characteristics (self-directed/no feedback vs. tutor-led/with feedback; stereoscopic vs. monoscopic), all three types of courses were able to advance students’ surgical suturing skills (Fig. ). Contrary to our expectations and the desires of the students, feedback from a tutor in our RCT did not show a significant impact on the quality of surgical suturing techniques among novices. Rather, the number of individual sutures performed during training was significantly related to the final competency level. These results are surprising, given that the tutor-led course with feedback is described in the literature as the gold standard and the most common teaching modality . However, the lack of positive effects associated with feedback in the acquisition of basic suturing techniques (Fig. C, D) is consistent with a study on the learning of laparoscopic suturing techniques, which shows that only actual training and knowledge of the result are relevant. Further instruction provided offers limited additional benefit . This is also consistent with an RCT that found that feedback offered no benefit between a video course and a tutor-led course in terms of efficiency and tissue handling during suturing . Learning modalities were perceived very differently by the students (Fig. A, B). First, the self-assessment of improvement showed that the HMD-based course was best at advancing this factor, followed by the tutor-led course, and lastly, the e-learning course (Fig. A). Second, the students rated the tutor-led course best regarding the ability to teach surgical suturing techniques, followed by the HMD-based course. The e-learning course was largely seen as unsuitable (Fig. B). Therefore, feedback and the teaching modality seem to have a primary impact on the learning experience. This is consistent with a study showing that additional feedback via videoconferencing in an e-learning suturing course increased user satisfaction . Another study showed that feedback improves students’ confidence . Although the HMD-based course was very similar to the e-learning course and both did not provide feedback, the students’ ratings for each were significantly different (Fig. A, B). The HMD-based course, unlike the e-learning course, offers immersion and stereoscopy, and it is taught from a first-person perspective. Immersion describes the feeling of proximity to reality created by digital learning applications and constitutes an important aspect for understanding the effects of digital teaching media . Higher immersion is said to enhance the user’s sense of being much more of a participant than an observer in the constructed space, known as the individual presence. This, in turn, can promote more intensive learning . In line with this, students in this study described being shielded from the outside world by the HMD as a pleasant experience and reported being able to concentrate better. Similarly, the benefits of HMDs with non-stereoscopic 360° video (without spatial vision) for the acquisition of knot-tying skills have been demonstrated . Furthermore, Peden et al . showed an increase in satisfaction with a non-stereoscopic AR-HMD (Google Glass). An additional parameter influencing an enhanced learning experience through a deeper presence could be the spatial understanding of stereoscopic learning content. Stereoscopy is technically generated in HMDs through two different images for the right and left eyes, respectively (Fig. C) . This can lead to an expansion of ‘essential information’, such as improved visual acuity or depth perception . In this regard, extrinsic stereoscopy must be distinguished from intrinsic visual-spatial ability, the latter describing the ability to mentally manipulate objects in three dimensions , . For this purpose, the visual-spatial ability of subjects can be assessed with the MRT, in which figures must be rotated using the imagination and assigned to a corresponding initial graphic figure . Wanzel et al . have shown that such high-level tests of visual-spatial ability are appropriate for demonstrating novice skills, such as in demonstrating rigid fixations of mandibular bench models. Previous work has shown that there is a linear correlation between knot-tying performance and visual-spatial ability . Interestingly, stereoscopic displays were able to provide significant assistance in anatomical skill development to subjects with lower visual-spatial ability . However, we found neither a correlation between visual-spatial ability and the acquisition of surgical suturing techniques nor an advantage of stereoscopic displays in students with low visual-spatial ability (Fig. C). The reason for the lack of correlation could be the influence of our having students perform less complex procedures , such as basic suturing techniques. Aside from the stereoscopic presentation, the perspective on the teaching event also appears to play an essential role in promoting more intensive learning. For the past 100 years, the classic Halsted model of ‘see one, do one, teach one’ has been followed and applied in surgical teaching , . The theoretical basis of this type of learning is mirror neurons, which play a fundamental role in understanding movement . In our tutor-led courses, the most common position chosen by students was the opposite perspective (48%) or the side perspective (42%), which also corresponds to a typical situation in the operating room. Only 10% of the students chose an over-the-shoulder perspective, which never fully corresponds to a first-person perspective. However, a true first-person perspective further enhances learning and can be provided by videos . Unlike monoscopic 2D videos, stereoscopic videos through HMDs provide a much more uniquely realistic first-person perspective from the proper distance, viewing direction, and response to head movements. In this context, Nagayo et al . , who visualized the step-by-step execution of surgical suturing through holographic representations using AR-HMDs from a first-person perspective, showed that this method was rated very helpful by the users. It should be noted that the HMD-based course in our study did not offer an interactive VR environment but was based on immersive and stereoscopic 180° first-person videos. VR environments enable a change of viewpoint or even allow users to communicate with the simulations displayed, such as through direct digital feedback, to achieve a maximum form of immersion. The high level of technical preparation required makes these applications very complex. Therefore, it is very difficult to adapt to the needs of each specific medical school. In addition, current controllers for HMDs provide little haptic feedback, so the handling of instruments or tissue cannot be realistically reproduced in a VR environment. Therefore, the use of bench models is, to date, still the preferred approach for the teaching and acquisition of surgical skills. In this regard, studies have shown that the degree of fidelity has less influence on novices’ learning of surgical suturing skills . In this respect, even handmade tools proved to be useful during the COVID-19 pandemic . EVA, as used in our study, has been shown to be as suitable for learning surgical suturing and various related techniques as high-fidelity models, such as pig feet . Supporting this, when learning surgical suturing, the way the procedure is taught seems to have a higher impact on the learner than the material used in the model . Overall, our results suggest that the HMD-based course offers clear benefits over an e-learning course in terms of the learning experience. Despite the same training duration of 1 h, interestingly, students performed ~3.1 sutures more in the HMD-based course and ~2.8 more in the tutor-led course than in the e-learning course. However, the following limitations should be considered. Conclusions about the possible long-term retention of skills or the possible cumulative effect of longer training periods cannot be drawn from our study. In this regard, it is important that students train regularly by themselves, as there could be a decrease in learned skills , . Independent training and automated assessments as an alternative feedback method could enable this. However, most automatic systems for surgical performance assessment are based on kinematic data and require specific hardware , while video-based assessment relies on endoscopic videos . Although we showed that the OF correlates with the GRS and that there is a strong decrease per performed suture with an increase in the GRS (Fig. C), this alone would not be enough to replace detailed feedback . In this regard, artificial intelligence (AI) algorithms for automated surgical performance assessment have already been used very successfully for endoscopic videos based on public datasets , . However, we are not aware of any public dataset relevant to the evaluation of open surgical sutures such as those presented in our study . Therefore, as part of this study, we have published our dataset ( https://doi.org/10.5281/zenodo.7940583 ) for further development. The dataset includes 314 videos with a total length of 26 h and blinded OSATS annotated by three qualified investigators. This makes it one of the largest video datasets and to our knowledge the first in its field. 

The use of HMDs with stereoscopic and immersive video provides advantages in enhancing the learning experience and should be preferred over a traditional web application for e-learning. Contrary to expectations, feedback is not necessary for novices to achieve a sufficient level of competence in surgical suturing; only the number of surgical sutures performed during training is a good determinant of competence improvement. Nevertheless, feedback still enhances the learning experience. Therefore, automated assessment as an alternative feedback approach could further improve self-directed learning modalities. As a next step, the data from this study could be used to develop such automated AI-based assessments. 

The study approved by the Institutional Review Board (or Ethics Committee) of University Hospital RWTH Aachen (protocol code EK 352/21 and date of approval 09.09.2021 & protocol code EK 22-329 and date of approval 05.10.2022). 

Informed consent was obtained from all subjects involved in the study. 

This research was supported by the Medical Faculty of the RWTH Aachen University under the grant number SBE-22-2-5. Behrus Puladi was funded by the Clinician Scientist Program of the Medical Faculty of RWTH Aachen University. 

B.P., P.P., F.H., and A.B.: conceptualization; P.P., B.P., A.B., and M.Li.: methodology; P.P., B.P., M.Le., and R.R.: software; P.P., B.P., and A.R.: validation; P.P., B.P., F.H., M.Le., F.K., T.W.K., and R.R.: formal analysis; P.P., B.P., M.O., and M.U.: investigation; F.H., B.P., M.Le., M.Li., and R.R.: resources; P.P. and B.P.: data curation; P.P.: writing – original draft preparation; B.P., P.P., R.R., F.H., M.Le., A.B., M.O., M.U., F.K., A.R., M.Li., and T.W.K.: writing – review and editing; P.P. and B.P.: visualization; B.P.: supervision; B.P.: project administration; B.P.: funding. All authors have read and agreed to the published version of the manuscript. 

The authors declare no conflicts of interest. 

Name of the registry: German Clinical Trials Register. Unique identifying number or registration ID: DRKS00029307. Hyperlink to your specific registration (must be publicly accessible and will be checked): https://drks.de/search/de/trial/DRKS00029307. 

Behrus Puladi. 

The video dataset of this study, including the OSATS assessment by the three investigators, can be downloaded at https://doi.org/10.5281/zenodo.7940583 . The remaining data generated and/or analyzed as part of this study are available upon reasonable request from the corresponding author. 

We would like to thank Jakob van Eys, Daniel Fink, Patrick Buchta, and Uli Heuter from the Audiovisual Media Center of the Medical Faculty RWTH Aachen University for their technical support in the production of the teaching material and Dr. Max Taubert for preparing the random assignment sequence and providing statistical advice. 

