A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study
In this mixed-method study, we collected quantitative and qualitative data from exam participants via an anonymous, voluntary online questionnaire. Our primary objective was to assess the level of organisation, ease of use, usefulness and overall acceptance of the v-OSCE. Qualitative data explored contexts related to participants’ perceptions and views. Setting The Academic Unit of Psychiatry and Addiction Medicine is based in Canberra, ACT. Due to COVID-19 restrictions, final (4 th ) year students attended online lectures and interactive workshops. Assessments were conducted using Zoom , a secure, video-conferencing platform. The assessment consisted of a 20-min role-play interview where the students obtained a relevant history from the simulated patient. To ensure vertical equity, potentially disadvantaged students could use the medical school’s dedicated high-speed internet and computer facilities. Participants and procedures In early 2022, medical students, examiners and simulated patients (paid actors) were informed of online, summative v-OSCEs. After the examinations (October 2022), all three groups were sent an email invitation to participate in the voluntary survey, study information sheet and a survey link (see Supplemental Table S1 ). Return of the survey was approved by the Ethics Committee to constitute informed consent. Survey questions measured user perceptions, acceptance and attitudes towards the v-OSCE, as well as open-ended questions related to beneficial aspects, challenging aspects and improvements ( ). Quantitative analysis We computed summary descriptive statistics of survey responses using Microsoft Excel. Likert-type ordinal data were dichotomised into 0 = none or a little, and 1 = moderately, very or extremely (with respect to sample strata of the relevant items). Gender and age bracket string data were transformed into numeric data with missing data excluded. We present results as frequencies and proportions of respondents. Qualitative analysis Thematic analysis of the qualitative online survey responses was performed by two of the researchers, according to Braun and Clarke’s six-phase process. Researcher one led phases 2-6. Data familiarisation involved independently reading transcripts, making notes of meaningful and relevant aspects of the data. Initial codes were assigned and collated to produce a shared coding framework, using NVivo 12.0. Coding was driven by data and semantics, recognising the contribution of each researchers’ subjectivity of data interpretation. In phase 3, the two coders applied this framework to the transcripts, working independently to search for appropriate themes and sub-themes. Conferencing achieved a ‘thematic map’. During Phases 5 and 6, themes were further explored iteratively by both coders, who reached a final framework of four overarching themes by consensus. Reflexivity statement Researcher one is a senior research coordinator with the academic unit. Researcher two is a consultant psychiatrist with experience in conducting and examining OSCEs. Thus, the results of the qualitative analysis may have been influenced by our own judgements, practices and belief systems. Ethics approval The project was approved by the ACT Health Human Research Ethics Committee (Protocol 2022/504). 

The Academic Unit of Psychiatry and Addiction Medicine is based in Canberra, ACT. Due to COVID-19 restrictions, final (4 th ) year students attended online lectures and interactive workshops. Assessments were conducted using Zoom , a secure, video-conferencing platform. The assessment consisted of a 20-min role-play interview where the students obtained a relevant history from the simulated patient. To ensure vertical equity, potentially disadvantaged students could use the medical school’s dedicated high-speed internet and computer facilities. 

In early 2022, medical students, examiners and simulated patients (paid actors) were informed of online, summative v-OSCEs. After the examinations (October 2022), all three groups were sent an email invitation to participate in the voluntary survey, study information sheet and a survey link (see Supplemental Table S1 ). Return of the survey was approved by the Ethics Committee to constitute informed consent. Survey questions measured user perceptions, acceptance and attitudes towards the v-OSCE, as well as open-ended questions related to beneficial aspects, challenging aspects and improvements ( ). 

We computed summary descriptive statistics of survey responses using Microsoft Excel. Likert-type ordinal data were dichotomised into 0 = none or a little, and 1 = moderately, very or extremely (with respect to sample strata of the relevant items). Gender and age bracket string data were transformed into numeric data with missing data excluded. We present results as frequencies and proportions of respondents. 

Thematic analysis of the qualitative online survey responses was performed by two of the researchers, according to Braun and Clarke’s six-phase process. Researcher one led phases 2-6. Data familiarisation involved independently reading transcripts, making notes of meaningful and relevant aspects of the data. Initial codes were assigned and collated to produce a shared coding framework, using NVivo 12.0. Coding was driven by data and semantics, recognising the contribution of each researchers’ subjectivity of data interpretation. In phase 3, the two coders applied this framework to the transcripts, working independently to search for appropriate themes and sub-themes. Conferencing achieved a ‘thematic map’. During Phases 5 and 6, themes were further explored iteratively by both coders, who reached a final framework of four overarching themes by consensus. 

Researcher one is a senior research coordinator with the academic unit. Researcher two is a consultant psychiatrist with experience in conducting and examining OSCEs. Thus, the results of the qualitative analysis may have been influenced by our own judgements, practices and belief systems. 

The project was approved by the ACT Health Human Research Ethics Committee (Protocol 2022/504). 

Of the participants in the 2022 v-OSCE, 47.5% ( n = 47) of medical students, 77% ( n = 10) of examiners, and 100% of simulated patients ( n = 5) returned the survey. summarises participant demographic data. Quantitative results Most participants (93.5%) rated the level of organisation of the v-OSCE favourably, and its ease of use and usefulness as at least moderately favourably, (ease of use = 98.4%, usefulness for demonstrating interviewing skills = 86.6%, and usefulness for interacting with other participants = 83.9%). Overall, 82.3% of participants reported the v-OSCE format was at least moderately acceptable (students = 83%, examiners = 80%, and simulated patients = 80% (see ). Qualitative freeform feedback – thematic analysis The results are supported by quotations from study participants (S = student, E = examiner, SP = simulated patient). Thematic analysis identified four main themes extracted from the survey. Theme 1: Easy to use, efficient and convenient Students reported that the online OSCE was easy to use: ‘everything was set out well and very easy to understand/follow’ (S7). Others highlighted its ‘smooth organisation and explanation of the OSCE structure and expectations’ (S38). The technology supported them ‘to adequately demonstrate all aspects of psychiatric interview’ (S43). The examiners appreciated the ‘efficient transition from one student to the next’ (E6), observing the ‘students were mostly relaxed which may have been contributed by being in their own familiar and comfortable environment’ (E6). The convenience of working from home and reduction in stress levels were common themes. Participants highlighted, ‘it is much less anxiety provoking and I feel I can demonstrate what I’m capable of much more easily’ (S7) and (working) ‘from home was a huge advantage, both in terms of preparation and low stress on the day’ (SP5). Theme 2: Challenges of online OSCE Challenges to the online exam were experienced by some. Several students reported that the online technology made it ‘more difficult to immerse yourself in the scenario’ (S27) and ‘difficult to read expression, body language, and responses’ (S37), making it harder ‘to smoothly conduct the interview’ (S13). One student observed: It was difficult to pick up on non-verbal cues from the patient, which would have made the scenario more realistic. It was hard to demonstrate rapport and engagement with the patient, which we’ve been taught are important aspects of managing psychiatric patients (S20). One examiner commented that the online technology did not allow them ‘to view the student’s facial expression when the actor is portraying distress or seeking empathy’ (E6). Although this problem could be overcome by adjusting the software settings in future, examiners observed that ‘the virtual space does not allow the same degree of professional intimacy and connectedness as face-to-face setting’ (E6). Examiners and simulated patients also commented on some distractions and variations in students’ home environments. Theme 3: Technical and connectivity issues Technical problems were reported, including poor sound quality, interrupted internet connection and bandwidth issues. One student stated the ‘audio broke out a few times interrupting the flow of a difficult discussion [making it] hard to stay “in character”’ (S8). Others found they ‘would often speak over the top of each other due to some lag time with the audio’ (S30). One student expressed concern about the standardization and fairness of v-OSCE and its impact on exam performance: [It’s] hard to have a fair playing field. Some people have different home setups and availability to sit them at home, in person means we all are doing them in the same environment… Its just overall its more difficult to really shine over Zoom. It’s harder to interact with patients and the examiners (S35). According to one examiner, the ‘occasional Wi-Fi drop-out disrupted the continuity of the interview’ (E6), however, others described them as ‘minor technical issues’ (E5) that ‘can be adjusted for with backups and flexible timing’ (E3). One simulated patient observed, ‘the students all handled any of those moments well’ (SP5). Despite these challenges, students provided positive feedback: ‘I really enjoyed the vignette which included everything and gave me enough time to really think about it’ (S15) including that Psychiatry and Addiction Medicine stations: ‘are, in my opinion, the only stations that can be completed appropriately online’ (S24). Theme 4: Improvements to the OSCE The detailed feedback from participants provided helpful suggestions for improving the online OSCE experience. A consistent recommendation was the use of better-quality microphones, sound quality and internet connection. Examiners recommended participants be offered a practice telehealth session which includes advice on the ideal, professional set up. One simulated patient appreciated the reserve actor who rapidly stepped in during some technical difficulties. Others described ‘it worked about as seamlessly as could be expected’ (SP5), feeling ‘impressed with how these online OSCE’s are organized and how generally they run very smoothly’ (SP3). On the other hand, some students (6/47) and examiners (3/10) overtly stated they preferred face-to face examinations. summarises our recommendations for conducting v-OSCEs. 

Most participants (93.5%) rated the level of organisation of the v-OSCE favourably, and its ease of use and usefulness as at least moderately favourably, (ease of use = 98.4%, usefulness for demonstrating interviewing skills = 86.6%, and usefulness for interacting with other participants = 83.9%). Overall, 82.3% of participants reported the v-OSCE format was at least moderately acceptable (students = 83%, examiners = 80%, and simulated patients = 80% (see ). 

The results are supported by quotations from study participants (S = student, E = examiner, SP = simulated patient). Thematic analysis identified four main themes extracted from the survey. 

Students reported that the online OSCE was easy to use: ‘everything was set out well and very easy to understand/follow’ (S7). Others highlighted its ‘smooth organisation and explanation of the OSCE structure and expectations’ (S38). The technology supported them ‘to adequately demonstrate all aspects of psychiatric interview’ (S43). The examiners appreciated the ‘efficient transition from one student to the next’ (E6), observing the ‘students were mostly relaxed which may have been contributed by being in their own familiar and comfortable environment’ (E6). The convenience of working from home and reduction in stress levels were common themes. Participants highlighted, ‘it is much less anxiety provoking and I feel I can demonstrate what I’m capable of much more easily’ (S7) and (working) ‘from home was a huge advantage, both in terms of preparation and low stress on the day’ (SP5). 

Challenges to the online exam were experienced by some. Several students reported that the online technology made it ‘more difficult to immerse yourself in the scenario’ (S27) and ‘difficult to read expression, body language, and responses’ (S37), making it harder ‘to smoothly conduct the interview’ (S13). One student observed: It was difficult to pick up on non-verbal cues from the patient, which would have made the scenario more realistic. It was hard to demonstrate rapport and engagement with the patient, which we’ve been taught are important aspects of managing psychiatric patients (S20). One examiner commented that the online technology did not allow them ‘to view the student’s facial expression when the actor is portraying distress or seeking empathy’ (E6). Although this problem could be overcome by adjusting the software settings in future, examiners observed that ‘the virtual space does not allow the same degree of professional intimacy and connectedness as face-to-face setting’ (E6). Examiners and simulated patients also commented on some distractions and variations in students’ home environments. 

Technical problems were reported, including poor sound quality, interrupted internet connection and bandwidth issues. One student stated the ‘audio broke out a few times interrupting the flow of a difficult discussion [making it] hard to stay “in character”’ (S8). Others found they ‘would often speak over the top of each other due to some lag time with the audio’ (S30). One student expressed concern about the standardization and fairness of v-OSCE and its impact on exam performance: [It’s] hard to have a fair playing field. Some people have different home setups and availability to sit them at home, in person means we all are doing them in the same environment… Its just overall its more difficult to really shine over Zoom. It’s harder to interact with patients and the examiners (S35). According to one examiner, the ‘occasional Wi-Fi drop-out disrupted the continuity of the interview’ (E6), however, others described them as ‘minor technical issues’ (E5) that ‘can be adjusted for with backups and flexible timing’ (E3). One simulated patient observed, ‘the students all handled any of those moments well’ (SP5). Despite these challenges, students provided positive feedback: ‘I really enjoyed the vignette which included everything and gave me enough time to really think about it’ (S15) including that Psychiatry and Addiction Medicine stations: ‘are, in my opinion, the only stations that can be completed appropriately online’ (S24). 

The detailed feedback from participants provided helpful suggestions for improving the online OSCE experience. A consistent recommendation was the use of better-quality microphones, sound quality and internet connection. Examiners recommended participants be offered a practice telehealth session which includes advice on the ideal, professional set up. One simulated patient appreciated the reserve actor who rapidly stepped in during some technical difficulties. Others described ‘it worked about as seamlessly as could be expected’ (SP5), feeling ‘impressed with how these online OSCE’s are organized and how generally they run very smoothly’ (SP3). On the other hand, some students (6/47) and examiners (3/10) overtly stated they preferred face-to face examinations. summarises our recommendations for conducting v-OSCEs. 

The majority of study participants rated the online technology as an acceptable format for the exam. Most perceived the v-OSCE as well organised, easy to use and useful for demonstrating their skills and interacting with others. The main challenges reported were difficulties in assessing body language and emotional (affective) prosody, defined as the non-verbal aspects of language that allow people to recognise or convey emotions. This included a lack of access to the full range of facial expressions, posture, gesticulations, and nonverbal cues which would be accessible with an in-person interaction. Similar constraints have been reported in telepsychiatry and further research could investigate how various v-OSCE configurations (e.g. station setup, close ups) impact on student examination performance. There were also internet connectivity challenges that de-synchronised the interaction and negatively impacted on participant immersion in the role-play examination. According to feedback, immersion could be improved by using high-quality headsets and microphones as well as wired internet access. Contingency plans are recommended for unavailability of examiners and simulated patients and for connectivity issues (e.g. rescheduling and back-up access options). Students indicated they lacked practice opportunities to develop their digital and interpersonal skills with telehealth, referred to as one’s ‘webside manner’. We recommend establishing telepsychiatry tutorials and peer-assisted mock v-OSCE sessions in the curriculum to enhance student preparedness and confidence with online assessments. Credibility refers to the believability of research findings, enriched by the convergence of multiple sources of data (triangulation). Wilkes et al. recently published an evaluation of student summative OSCE results for each year of the pandemic (compared to the 3 years prior). Results showed the technology was effective for assessing examination competence, with a high degree of internal consistency, reliability and validity. Our recommendations for practical measures to improve the examination process concord with the findings from recent systematic reviews of health professional and medical student online examination adaptations during the pandemic. – , Despite the limitations of the virtual examination, we conclude that it is an acceptable, efficient and useful modality for medical student examinations. Limitations and future considerations Our mixed-methods study surveyed a small sample of medical students, examiners and simulated patients, which may limit the generalisability of the findings. Although the e-OSCE was a response to pandemic restrictions and not an initiative of the authors, it’s important to note we were investigating our own examination processes. Future research could investigate whether improvements to the teaching curriculum (e.g. e-learning modules, tutorials and/or peer-assisted mock-OSCEs) could enhance student exam confidence and performance. A controlled trial of face-to-face compared to v-OSCE is warranted. 

Our mixed-methods study surveyed a small sample of medical students, examiners and simulated patients, which may limit the generalisability of the findings. Although the e-OSCE was a response to pandemic restrictions and not an initiative of the authors, it’s important to note we were investigating our own examination processes. Future research could investigate whether improvements to the teaching curriculum (e.g. e-learning modules, tutorials and/or peer-assisted mock-OSCEs) could enhance student exam confidence and performance. A controlled trial of face-to-face compared to v-OSCE is warranted. 

Medical students, examiners and simulated patients found that v-OSCE examinations are convenient, easy to use, efficient and acceptable. To move towards a level playing field, we need to ensure familiarity with professional telehealth interviewing, access to high quality videoconferencing and high-speed internet connections. This is likely to better simulate the in-person experience and, therefore, acceptance of online technologies. Lastly, telepsychiatry is increasingly being normalized, becoming an important tool for psychiatrists to delivery health care. This necessitates that medical schools teach and assess telehealth skills. The integration of telehealth competencies and assessment tools in medical schools is essential to equip students with the required skills for their future careers. 

Supplemental Material - A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study Supplemental Material for A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study by Rebecca E Reay, Paul A Maguire and Jeffrey CL Looi in Australasian Psychiatry. 

