Designing a touchless physical examination for a virtual Objective Structured Clinical Examination
The COVID-19 pandemic forced many institutions to transition to virtual Objective Structured Clinic Examination (OSCE) formats for the assessment of clinical skills. One of the challenges with pivoting to a virtual format is how to assess physician examination (PE) skills. Thus far, approaches to assessing PE skills during virtual OSCEs have varied. One approach is to have examinees focus only on ‘inspection’ when examining standardized patients (SPs). , Another is to have examinees only verbalize what PE maneuvers they would want to conduct. Others have removed all PE components while some have incorporated multimedia to illustrate real clinical findings. , The use of high-fidelity virtual patients has also been reported. Studies have reported some challenges in assessing PE skills virtually, although it is unclear if this is due to a knowledge gap, the limitations of the format, or inadequate training with the technology. , While virtual OSCEs are feasible, there are limitations regarding what PE skills and competencies can and should be assessed with this format. , One can conceptualize a PE as four distinct competencies: selecting appropriate PE maneuvers, performing these maneuvers, identifying normal and abnormal findings, and interpreting the significance of the findings. One disadvantage of traditional OSCEs is that SPs typically do not have abnormal findings. While examinees may be able to demonstrate a PE, it is not clear that they would be able to interpret findings to make a diagnosis. Thus, although lower order skills such as knowledge recall are assessed (e.g. performing PE maneuvers by rote), skills related to application of knowledge, such as interpretation and synthesis (e.g. interpreting the significance of abnormal findings) may not be assessed. A virtual format may present an opportunity to target the assessment of these higher order skills, for example by incorporating multimedia depicting abnormal findings, something that can be challenging in a traditional setting. While previous studies have looked at studying virtual physical exam stations with or without adding abnormal findings, they either did so without the learners having to identify the abnormal findings independently, there were no abnormal findings and/or there was no discrimination between the different orders of learning. , , , , , , In this study, we developed and pilot-tested a touchless PE station within a virtual OSCE. We prompted examinees to verbalize their approach to a PE, interpret findings using images, audio, and video files, and formulate a diagnosis. We aimed to assess the feasibility of implementing a touchless PE station that assesses knowledge recall (i.e. information gathering) and application of knowledge (i.e. interpretation and synthesis) and to gather validity evidence related to scoring, generalization and extrapolation.   The Ottawa Hospital Research Ethics Board (OHREB) reviewed this project, which received an exemption in the context of a quality initiative/program evaluation. Participants At the University of Ottawa, Internal Medicine (IM) residents from post-graduate years one to three (PGY-1 to 3) participated in a mandatory formative OSCE in March 2021. As this is a progress test, residents from all years participate but the difficulty level is set at a PGY-3 level. OSCE format administration We used Microsoft Teams to implement the OSCE virtually. Participants logged in remotely. The OSCE consisted of nine 12-minute stations: four structural oral and four PE stations, and one communication station. Skills assessed included knowledge, PE skills, and communication skills. There were no SPs. We recruited physicians as examiners; they underwent training using recorded orientation sessions and detailed written instructions. For the pilot station, they received additional training on how to share PE findings (i.e. using images, audio files, videos). We used station-specific checklists, rating scales, and examiner oral questions to score examinees. Pilot station development The co-investigators developed the pilot station. The checklist consisted of 18 items, one station-specific rating scale, and a global rating scale (for standard-setting purposes). The pilot station used elements of Bloom’s taxonomy of learning where we sub-categorized individual checklist items two domains: knowledge recall or application of knowledge (i.e. interpretation and synthesis). Participants were unaware of item categorization. Examinees received directions to complete a touchless PE and to describe any normal or abnormal findings. As examinees progressed through the station, the station provided them with real-time images, audio files and videos of PE findings, and expected them to interpret and recognize their significance. In contrast, the other PE stations followed a more traditional format, i.e. examinees verbally describing PE maneuvers with examiners reporting any findings. Analyses We calculated total scores for the overall OSCE using scores on all 9 stations (weighted equally). We calculated the reliability (internal consistency) of the OSCE using Cronbach’s alpha. We calculated cut scores using the borderline group regression method, which incorporates a linear regression approach using the scores from all examinees. We used univariate ANOVA to compare differences in scores between training levels for the overall OSCE and for the pilot station scores, domain sub-scores and rating scale scores. To explore pair-wise differences, we used Tukey’s honestly significant difference test. We calculated partial eta squared (η p 2 ) to measure effect size. We hypothesized that overall and application of knowledge scores would increase with training level, while knowledge recall scores would be similar across training levels. We calculated station-total correlations between each station and item-total correlations for each item on the pilot station. To compare total scores, domain sub-scores and rating scale score for the pilot station, we analyzed Pearson’s correlation coefficients. We used SAS, version 9.4 (SAS Institute Inc) for statistical analyses.   At the University of Ottawa, Internal Medicine (IM) residents from post-graduate years one to three (PGY-1 to 3) participated in a mandatory formative OSCE in March 2021. As this is a progress test, residents from all years participate but the difficulty level is set at a PGY-3 level.   We used Microsoft Teams to implement the OSCE virtually. Participants logged in remotely. The OSCE consisted of nine 12-minute stations: four structural oral and four PE stations, and one communication station. Skills assessed included knowledge, PE skills, and communication skills. There were no SPs. We recruited physicians as examiners; they underwent training using recorded orientation sessions and detailed written instructions. For the pilot station, they received additional training on how to share PE findings (i.e. using images, audio files, videos). We used station-specific checklists, rating scales, and examiner oral questions to score examinees.   The co-investigators developed the pilot station. The checklist consisted of 18 items, one station-specific rating scale, and a global rating scale (for standard-setting purposes). The pilot station used elements of Bloom’s taxonomy of learning where we sub-categorized individual checklist items two domains: knowledge recall or application of knowledge (i.e. interpretation and synthesis). Participants were unaware of item categorization. Examinees received directions to complete a touchless PE and to describe any normal or abnormal findings. As examinees progressed through the station, the station provided them with real-time images, audio files and videos of PE findings, and expected them to interpret and recognize their significance. In contrast, the other PE stations followed a more traditional format, i.e. examinees verbally describing PE maneuvers with examiners reporting any findings.   We calculated total scores for the overall OSCE using scores on all 9 stations (weighted equally). We calculated the reliability (internal consistency) of the OSCE using Cronbach’s alpha. We calculated cut scores using the borderline group regression method, which incorporates a linear regression approach using the scores from all examinees. We used univariate ANOVA to compare differences in scores between training levels for the overall OSCE and for the pilot station scores, domain sub-scores and rating scale scores. To explore pair-wise differences, we used Tukey’s honestly significant difference test. We calculated partial eta squared (η p 2 ) to measure effect size. We hypothesized that overall and application of knowledge scores would increase with training level, while knowledge recall scores would be similar across training levels. We calculated station-total correlations between each station and item-total correlations for each item on the pilot station. To compare total scores, domain sub-scores and rating scale score for the pilot station, we analyzed Pearson’s correlation coefficients. We used SAS, version 9.4 (SAS Institute Inc) for statistical analyses.   Sixty-seven residents participated in this OSCE (25 PGY-1, 21 PGY-2, and 21 PGY-3 residents). The mean score for the overall OSCE was 66.2% ±9.2 and the cut score was 58.2. The reliability for the OSCE, measured by Cronbach’s alpha, was 0.79. The overall pass rate for the OSCE was 60% (mean=59.1 ±6.0), 90.5% (mean=67.4 ±7.7) and 100% (mean=73.6 ± 7.3) for PGY1, PGY2 and PGY3 residents, respectively. For the pilot station, the total mean score was 6.8 ±1.4 (out of 10). Total scores on the overall OSCE were significantly different between training levels (F=24.828, p < 0.001, η p 2 = 0.437). Post-hoc analyses revealed differences between all levels, with scores increasing significantly with each year of training. Total scores and rating scale scores on the pilot station were also significantly different between training levels ( F = 3.936, p = 0.024, η p 2 =0.11; F=11.76, p < 0.001, η p 2 0.27, respectively) with differences between PGY-1 and PGY-2, and also between PGY-1 and PGY-3 residents. Domain sub-score analyses revealed that residents performed better on tasks relating to application of knowledge than to knowledge recall (see ). There was no significant difference in sub-scores by training level (see ). When looking at individual items, however, there were differences in scores between senior (PGY-2 and 3) and junior (PGY-1) examinees for five items, all of which assessed application of knowledge. For the overall OSCE, station-total correlations ranged from 0.38 to 0.75. The pilot station station-total correlation was 0.558 and the “total but excluding pilot station” correlation was 0.443 ( p < 0.001). This similarity in station-total correlations suggests that the pilot station was measuring similar constructs as other stations. Pilot station item-station correlations ranged from 0.12 to 0.57. The three most discriminating items assessed skills relating to application of knowledge (see ). Total score and all sub-scores significantly correlated with the global rating scale score (see ).   The adaptation of OSCEs to a virtual format presented an opportunity to easily incorporate technology to assess different PE skills (i.e. interpretation of abnormal findings) than are traditionally assessed. The assessment of PE skills is important because previous studies have only demonstrated a modest correlation between physical examination technique and diagnostic accuracy and that trainees are often unable to detect abnormal findings in an OSCE setting after completing medical school. , Thus, traditional OSCEs, in which examinees interact with a standardized patient with no abnormal findings, may limit the conclusions that one can infer based on examinee performance. Although other studies have incorporated multi-media in their OSCEs, this is one of the few studies that we know of that has leveraged the virtual format to incorporate abnormal PE findings in an OSCE for medical residents. In addition to demonstrating feasibility, careful attention must be paid to validity evidence when adapting to a virtual OSCE format. In this study, a rigorous approach to the development of the content and scoring instruments was used. Furthermore, no significant technical issues were encountered, and the overall reliability and station-total correlation were acceptable. Finally, this station was able to discriminate between learners at different stages of training. There was an unexpected lack of difference in sub-scores by year of training which may be a consequence of only having one touchless PE station. It is also possible that there may truly be no difference in the acquisition of these skills in more senior residents. Interestingly, the most discriminating items were those that did assess knowledge application skills. This study has several limitations: due to the nature of the OSCE being virtual, we did not assess psychomotor skills; this was a single-center study with only one touchless PE station; and we did not compare findings to those of a traditional OSCE.   This study demonstrates that the incorporation of a touchless PE station in a virtual OSCE is feasible and can be used to assess knowledge application skills while discriminating between learners at different levels of training.   