Curating COVID-19: A Digital Internship in a Rapid Response Archive

In May 2020, Arizona State University’s history department offered its first remote, digital internship to graduate students. Students completed a 180-hour internship between May and August 2020. The internship involved weekly meetings, curation, collecting, journaling, and marketing. Over the summer, the interns worked to identify a silence in the archive and address it by creating a collection plan targeting the perceived silence. The interns drew on their own networks to build the collection, created a collection plan, conducted oral history interviews, wrote a blog post, and completed a final portfolio.

Normally, the word "archive," conjures the image of a brick and mortar building with stacks of semi-organized contents, filing cabinets, and finding aids. In JOTPY, the contents are organized using similar principles and metadata, but all of its items are digital. While interns had no set physical location to report to, JOTPY inhabits an extensive digital space that includes the archive's website hosted by Omeka and its digital workspace operated on Slack. An archive can be enhanced or hindered by its organization, and graduate students proved crucial to managing its contents.
Of all the changes that occurred in 2020, one positive shift was the value and recognition of remote labor. For students, this led to an inevitable rise in what Jade Ryerson and Katherine Crawford-Lackey termed, "Quar-interning." 1 Arizona State University's COVID-19 Archive naturally lent itself to this role: a flexible internship opportunity, completed on the student's time and schedule. Successful interns completed 180+ hours of service. Their workload included weekly meetings, curating stories in the archive by adding metadata and following best practices in public history, collecting stories about pandemic experiences, identifying and addressing archival silences, data cleaning, categorizing stories, adding tagging and geolocation information, conducting oral histories, and writing three reflection pieces based on primary sources submitted to the archive.

Curation
Curation forms the backbone of a rapid response archive and frames the digital repository in real-time. 2 Data curation connects items to a larger global context comprised of libraries, publishers, scholars, and the public. 3 Its information channels users through the archive. 4 JOTPY uses the Dublin Core Metadata Element Set, which includes 5. Sue Barrett, "Metadata for All," in Presentation,   fifteen "core" elements to describe items. The same basic metadata is applied to every pandemic story. 5 Each entry includes a title, description (the story), date, and the contributor. Dublin Core offers controlled vocabulary, and in addition, curators add folksonomy terms. A controlled vocabulary is an organized list of words and phrases used to index content. Controlled vocabulary indicates an initial relationship between an item and thousands of other stories in the same category. After gaining a sense of the story, curators add additional, more specific words to describe submissions. Additional terms provide more paths into the archive and identify precise connections. For example, a political cartoon depicts Peru's national government ending the prohibition against children going outside on May 18, 2020. The image shows two panels. On the left, it reads: "First day children can go outside," (Shows a father chasing his son). On the right: "First day of exercise," (the father is collapsed in his chair sleeping). 6 Its controlled vocabulary phrases include "home and family life," "art and design," and "national government." The curator's tags added: son, father, meme, Twitter, hijo, padre, exercise, and tired ( Figure 1).
The submission form allows contributors to add their own "tags," or metadata. User-driven tags counter the traditional top-down taxonomy and allow for socially built metadata. 7 When uploading a story, participants are encouraged to tag it with keywords. These terms are saved as "contributor tags," a folksonomy, (a portmanteau of 'folk' and 'taxonomy), 8 forged from public input. Users create their own language of categorization; each term becomes its own field-an access point that breaks out from the traditional schema. Every COVID story, therefore, is contextualized by the contributor and the curatorial team.

Collecting
After spending a few weeks working in the archive, the interns established a collection plan to solicit stories related to specific topics like lost graduations or canceled sports seasons. The team reviewed collection plans from various institutions: the New Bedford Whaling Museum, the Smithsonian, and others. 9 After assessing the genre, they wrote a collection mission statement. In other words, what was the purpose of this collection, what does it do well, and why does it matter? Furthermore, how does this collection bolster and serve the mission of the COVID-19 archive at large? Next, they developed a plan for soliciting and collecting stories.
To capture digital and print audiences, the interns wrote and advertised the collection for a target audience. Next, they composed a social media call for submissions and a more traditional newspaper press release. These exercises drew on the interns' ability to conceptualize, communicate, and market a collection. As the internship progressed, they searched and aggregated submissions into collections. Toward the end, they wrote a blog post centered on the prompt: what does this collection tell us about the pandemic?  Finally, they combined these materials with their journal and their 5 to 10 favorite items into a digital portfolio. The portfolio was built and hosted within the archive and highlights the work and skills they acquired.

Silences
Archives are not neutral spaces. They are a corpus of items that have been collected, organized, prioritized, and cataloged by human actors. 10 They influence how stories are remembered. Simon Fowler writes, "Sources and archives are neither neutral nor natural. . .this. . .is the reason for so many silences." 11 Rapid response archives are no exception. A rapid response archive operates on the premise that contributors and curators collect around a historical moment, an episode deemed important enough to stop what we were doing and document instead. Yet, even with our best efforts and a team comprised of students, faculty, and curators around the world, JOTPY inherently creates silences. Silences reflect sources that either were not created in the first place, or not preserved. 12 The curatorial team can address some of these issues in real-time. Interns process track, and organize content and are trained to think critically about what JOTPY does well and what it might be missing. Technology increases inclusion. For example, oral histories help capture the ways people discuss and process their lived experiences like delivering a newborn during a pandemic. Moreover, memories reflect an individual's political, moral, and existential views of the present. 13 Recording these experiences captures subtle dynamics within a public health crisis.
Visual records dominate the archive, but interns and students can reflect on and submit items related to changing soundscapes, smells, physical spaces, and tactile experiences. For example, intern Shanna Gagnon wrote ". . .there is an odd tension of noise inside my home and silence the second I step outside. I find myself needing a quiet space when I'm in my house. Yet the second I walk outside, it's way too quiet . . . I start to feel uncomfortably lonely and in need of human connection." 14 Beyond the sources collected, the interns also identified an archival silence and created a collection targeting that absence. Their labor amplified distinct perspectives.
The summer interns zeroed in on several collection ideas: healthcare, incarceration, LGBTQ+, COVID brides, rural voices, the San Francisco Bay Area, children, the performing arts, foodways, and social justice (Figures 2 and 3). These collections reflected their individual strengths, areas where they might pull together their networks to fill information gaps. In the fall cohort, curators managed those collections and added motherhood, law enforcement, New Mexico narratives, and mental health.

How to Run a Digital Internship: Communication Is Key
The ability to communicate and receive feedback is critical. Interns met weekly as a group, attended larger project-wide town hall meetings, and navigated between the Omeka website, Canvas, and Slack. Intern Shanna Gagnon noted, "Our team's ability to directly communicate with each other, to offer constructive feedback, to step in to help others when needed, and to adapt to ever-constant changes characteristic of working in a rapid response live archive, was simply incredible." 15 Slack facilitated effective communication. It allowed the team to divide up work into different channels and promoted file sharing, threaded conversations, direct messages, and audio/ video calls.

What Interns Gain
The internship goes beyond traditional academic learning by introducing students to practical, desirable workforces skills. The interns learned to work remotely and asynchronously as a team using Slack. They learned how to use new software to create stable files. For example, they navigated to the Wayback Machine to capture websites   and save them for future use. 16 To save ephemeral social media posts, they used screen capture or recording software to create screenshots or MP4 files. And, when processing oral history audio files, interns generated AI transcriptions with Otter.ai. Some participants seized the opportunity to emerge as leaders within the group. Whether distinguished by critical thinking, attention to detail, consistency, or experience, almost everyone had a moment where they became the expert and taught the group about a topic or issue. Intern Lawson Miller wrote, ". . .the internship has given me practical experience in being a leader among my peers, communicating effectively through press releases and social media, and developing standardized practices." 17 After working on the project for several months, Miller trained new interns on how to curate and reviewed their work for quality and consistency.
Finally, and perhaps most importantly, the internship built community. Sometimes online coursework can feel contrived. In the internship, collaboration is necessary. Working in teams made the workload a social experience. Intern Alexis Walker reflected, "Before the internship, I shied away from group projects as a necessary evil to be avoided at all costs however the internship showed me just how fun working with others could be and to value a great team." 18 Echoing this sense of collaboration, intern Chris Twing added, "By the end of our fifteen-week internship we had become even more than a team, we had become a family. I believe this was partially due to a shared crisis. . .For many of us, our weekly Zoom. . .was nearly the only face-to-face contact, though virtual, we had with anyone." 19 As weeks of lockdown dragged on, the curatorial team's weekly meetings felt like a hybrid of work and socialization.
Directing the internship taught me to value the distinct perspectives everyone brought to the team. No two were exactly the same, but they all added something the project needed. The interns committed themselves to an extensive internship experience and broke down digital barriers. They walked away with an unexpected comradery uniting a geographically and demographically disparate team. Furthermore, the JOTPY internship created a sense of purpose, spurred by the sense of doing work that matters and work that will matter for future generations. We each became part of the archive, and it left a legacy with us. Years from now we will look back on the historical shift created by the COVID-19 pandemic and remember the dedication and care that went into curating pandemic stories.

Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author received no financial support for the research, authorship, and/or publication of this article.


ONLINE EDUCATION OF MARGINALIZED CHILDREN IN NORTH MACEDONIA AND ITALY DURING THE COVID-19 PANDEMIC

Online Education of Marginalized Children in North Macedonia and Italy During the COVID-19 Pandemic The COVID-19 pandemic abruptly interrupted the traditional education process and imposed the need to switch to online education. Children living in poverty without proper IT infrastructure have been substantially excluded from the educational process amidst the COVID-19 pandemic. During this challenging pandemic, education has been placed in quarantine. This research paper aims to qualitatively investigate the potential social exclusion and further marginalization within remote learning during the pandemic in a very particular local context, focusing on the areas of Bitola (North Macedonia) and Treviso (Italy).

EDUCATION ON THE MARGINS
About 91% of the world's student population did not go to school in spring 2020. The rapid spread of the virus responsible for COVID-19 and the resulting declaration of the global pandemic imposed the need for online education, leading to the exclusion of marginalized groups of children. UNICEF reported 463 million students worldwide being generally unable to access online education due to lack of technological infrastructure or remote learning policies for online education. 1 Italy was the first country after China to be caught by the unforeseen virus, which rapidly spread across the entire European continent, eventually reaching throughout the world. These circumstances led to forced lockdowns, curfews, remotely working from home, and remote schooling. However, approximately 54% of the students in Italy declared to have had difficulties in reaching the online didactics, while 7% had no access at all. 2 In North Macedonia, the Ministry of Education confirmed in September 2020 that 40,000 children are not able to follow online classes due to a lack of IT infrastructure. 3 This research poses the question How have teachers, parents, and students responded to the emergency online learning during spring 2020? Furthermore, What do they think has been the situation's impact on marginalized children in Bitola and Treviso? Bearing in mind the world's situation, this research examines how online education has impacted children, especially those in already marginalized positions, in the regions where we received our compulsory education. This paper examines the potential further marginalization of pupils within remote learning during the COVID-19 pandemic in Bitola (North Macedonia) and Treviso (Italy). Although there are enormous differences between the two countries -North Macedonia is a candidate state to become a member of the European Union with a population of approximately two million, and Italy is a European Union member state with a population of about sixty million -as well as different living standards and social composition, the marginalization in education amidst the pandemic is not quite disparate.

MARGINALIZING THE MARGINALIZED?
The key concepts underpinning the research are poverty, marginalization, social exclusion, and online education/remote learning. More precisely, the latter refers throughout the paper to traditional compulsory education merely transferred online. The abrupt switch to online learning during the COVID-19 pandemic denied, indeed, the right to education for poor children who are without proper IT infrastructure and electricity. According to the United Nations (UN), this move to online education or remote learning has been the largest disruption to the education system in history. 4 Poverty, in both its absolute and relative meanings, could be multidimensional and deprives around one billion children worldwide of basic needs, including nutrition and clean water, as well shelter, health care, and education, 5 thus, leading to social exclusion. Poverty has an impact on children's education no matter where they live and learn. Placed in the context of North Macedonia, poverty represents a problem that affects many social spheres. Nearly one-third of the population lives below the national poverty line having more negative effects on children than adults. 6 One in three children lives in poverty, comprising over 100,000 children in the country, 11% being in the Pelagonia region (Bitola and Prilep). 7 Nevertheless, poverty does not equally affect all ethnic communities. Among Roma, the poverty rate is three times the national average (Petrovska Mitrevska, Tuna 2017: 29). According to UNICEF North Macedonia, 1 in 300 poor children in the country attends preschool, and 2 out of 3 children attend secondary school. 8 Whereas in a decade, Italy has tripled the number of children living in poverty, 9 and these figures are increasing since child poverty is strongly connected to investments in human capital. So, individuals with a low level of education, especially those without a lower secondary school diploma, are disadvantaged and have limited access to the formal labor market, relying on less-safe jobs and yet at greater risk of poverty (ILO 1995;World Bank 2020). Unfortunately, a significant proportion of people living in poverty are migrants. For example, in Italy, they account for 24.4% in contrast to 4.9% for Italians. 10 Furthermore, migrants are a substantial part of the country's population, and thus, their children nowadays compose a significant portion of the Italian student 4 United Nations Policy Brief (2020 population (9.7%). 11 Yet, educational inequalities -in terms of unequally distributed opportunities -persist (Azzolini, 2012;Mura et al. 2020: 15) since the low socioeconomic resources, experienced above all by migrants, play a significant role in accessing those chances. Thus, there is an urgent call for policy interventions to fight this gap in educational achievements between native Italian and immigrant populations (Azzolini 2012: 177;Barbanti 2016). The other key concept present in this research is marginalization, as the educational gap represented by inequalities is strongly connected to the population belonging to the lower socio-economic classes and, thus, to some extent, marginalized (Azzolini 2012: 13). Marginality could be defined as "involuntary position and condition of an individual or group at the margins of social, political, economic, ecological and biophysical systems, preventing them from access to resources, assets, services, restraining freedom of choice, preventing the development of capabilities and eventually causing extreme poverty" (Gatzweiler et al. 2011: 3). "Marginalization describes both a process and a condition that prevents individuals or groups from full participation in social, economic, and political life." 12 The fact that marginalization represents a dynamic social phenomenon has made it quite challenging to define marginalization as a restricted category and should be observed as a "collection of changing relationships" (Howitt 1993: 6), sometimes referred to as social exclusion. Although the role of schools should be to act as drivers of inclusion, since they are the major vehicles capable of overcoming inequalities and promoting diversities (Jalušič, Bajt, Lebowitz 2019: 44), and education should be considered the way out of marginalization, it could, however, as Munn and Lloyd (2005) contend, become an agent of marginalization with unadapted curriculum to individual needs, inflexible structures, and inconsistency between norms and the capacity of students to meet them (Mowat 2015: 460). Amidst the pandemic, the role of teachers remains significant as "every teacher has an impact on the successful inclusion of immigrant children and developing intercultural education" (Vižintin 2018: 93). Nevertheless, the absence of proper conditions for online education has significantly challenged this obligation.
Remoteness, exclusion, and extreme poverty are related, so the poorest are often encountered in rural areas and belong to ethnic minorities and socially excluded groups (Gatzweiler et al. 2011: 2). Social exclusion, as defined by Razer et al. (2013), is a state when individuals or groups "lack effective participation in key activities or benefits of the society in which they live" (Mowat 2015: 457). However, as Mowat 11 According to data provided by the Ministry of Education University and Research (MIUR), the population of students without the Italian citizenship is about 9,7% in Italy, see also further contends, marginalization also encompasses a sense of not belonging and not being valued (Ibid 2015: 457). Hence, extreme poverty is rooted in marginality and represents another layer in its complexity introducing the sub-concept "marginalized poor," referring to those affected by marginalization and poverty. 13 For this study, the term refers to the economic situation of the families, including their access to technological devices and infrastructure 14 in the context of public schools and education in Italy and North Macedonia.

METHODOLOGY
The research employs a purely qualitative methodology conducting participatory and subject-focused research with an ethnographic approach. Through qualitative research, we aim to represent an open view capable of understanding human concerns. We analyze self-reflections and narratives of educators, students, and parents experiencing these uncertain times of global pandemic. We try to understand their perspectives on challenges that marginalized children, particularly with minority and migrant backgrounds, have been facing during the lockdown. Furthermore, in Bitola's context, 15 storytelling as a supplementary qualitative methodology has been employed. 16 Narratives and testimonies from lived experiences of parents and students have been collected from the field in the period between July and August 2020. The role of teachers and other subjects involved in the educational process in creating inclusivity within schools and implementing an education system in which achievements and success are available to all is crucial. Educators are required to accept their responsibilities in promoting participation and reducing underachievement, particularly with marginalized children, due to different reasons affecting their learning outcomes (Rouse 2008).
The research involves our personal experience as members of the researched community and familiarity with the education sector. This reality could represent a personal bias because it is not an easy process to "make the familiar strange" (Holliday 2007). Nevertheless, qualitative research settings are in any case difficult to control, and, following Holliday's suggestion, "we have to capitalize on those that are available to us" (Ibid. 2007: 22

Research Design and Ethical Considerations
We did not select our informants with premeditation; they were chosen through a snowball sampling method. They all granted their informed consent, which enabled us to record responses and successively transcribe all the interviews and surveys in their original languages (Italian and Macedonian). Therefore, in the analysis, we will use our translations into English of the significant statements. Concerning another ethical aspect of the study, we have preserved the anonymity of the respondents.

Data Collection
Data collection was conducted with open-ended semi-structured interviews, whereas in North Macedonia, we employed structured surveys and storytelling as an additional method. Due to social distancing measures and the inability to conduct interviews solely, structured surveys were sent via email directly to educators. Moreover, we collected testimonies from parents sharing their experiences. These testimonies served to assess the perception, beliefs, and experiences of everyone involved in the educational process. The questions were intended to guide the informants, rather than restricting them to answering what they are just asked. Thus, through dialogues, valuable information beyond the scope of the questionnaires was collected as well. In Italy, we interviewed 11 teachers from different school cycles: primary (4), lower secondary (3), and upper secondary schools (4). In North Macedonia, we gathered data from structured surveys (8), semi-structured interviews (4), and field visits (4). There were 14 participants involved, including primary school teachers (1), secondary school teachers (5), high-school teachers (1), special educators (2), social workers (1), and parents (4).

FIELDWORK TESTIMONIES Online Education of Marginalized Poor Children in Bitola
When schools were closed and education transferred online, marginalized poor students were confined at home, and their education rights were substantially placed in quarantine. Many of these children belong to low-income families who have been receiving emergency relief during the pandemic. Some do not have basic living conditions, including electricity, and particularly proper IT infrastructure to access online education: There was no way they could follow online lectures; they were at home. We do not have the conditions, not even internet or mobile. (Single mother of three children) In my school, above 50% of the students were without proper technical conditions to follow online classes. […]. The majority of students without conditions for online education are Roma, whether devices or the internet. (Secondary school teacher) The fact that certain families and children could not manage to participate in online education might have completely interrupted online education in certain schools and classes: "Online teaching lasted for almost two weeks. Later some people started canceling because they did not have the internet, some did not have proper means, even mobile phones. That is why online classes were eventually canceled" (Father of a second grader). Some of the schools, teachers and special educators managed to obtain internet and technology donations to supply the students who were not able to afford them: The school secured donations from NGOs of a certain number of tablets distributed to the students who did not have the technical means to follow online lectures. (Secondary school teacher) I had a low-income student in my class. Free internet was provided at my request, which unfortunately arrived after the end of the school year. The same student will receive and use it in the new school year. (Teacher/Special Educator and rehabilitator) Teachers also consider alternatives to address the student's absences in their online classroom. They implemented different teaching methods to make sure no student is excluded. The teaching methodology used by an elementary school teacher involved "sending recorded lessons so the students could watch when he/she was able to." While the internet and technical devices were secured for some children, others' education remained in quarantine. Moreover, in some cases, it was insufficient to provide only a device, mobile phone, or tablet if the home was without an internet connection or access to electricity. Educational programs were available on certain television channels for students who did not have the proper devices to follow online education. Students from families with low income who are without computers or internet at home "were given printed materials, which were sent to their parents and several families were visited in their homes by experts from our educational institution" (Social worker).
As education was brought into the homes, parents absolved a more significant role. Moreover, as an elementary school teacher stated: "unlike classes with a physical presence, online teaching is much harder as parents must control all of the schoolrelated assignments" thus, parents' illiteracy and lack of IT knowledge represented a problem. It was particularly difficult for students being dependent on their parents during the process of online education, particularly those who did not have a proper internet connection and devices or whose parents' lacked IT knowledge and were thus unable to assist with the assignments.
What I discovered about my students and their families are very sad pictures in general. IT literacy, more precisely, parents' lack of knowledge of computers and the internet. As they were also stating: "We do not know, we cannot log in on the platform, we have already reached the internet limit […], we do not have money to buy them new phones, the old one does not function well, the camera is broken, and similar things. " (Secondary school teacher) September 2020 marked the beginning of the new school year. The government's policy included online education and education with a physical presence in school for first-, second-, and third-grade students, as well as for other children who were unable to follow online classes with previous parental consent. Throughout the COVID-19 pandemic in spring 2020, an evident effort was made by some educators to find alternatives to online education or provide the necessary conditions for students to follow online classes. However, access to quality education was significantly disabled. Whether the absence of proper technology, weak internet connection and interruptions, or lack of knowledge, the educational process has been significantly challenged. This education crisis might have a significant impact in the future, particularly on the academic performance of marginalized children who were unable to access online education properly. The lesson this pandemic taught the educational system is that there is an imminent need to address poverty for the successful digitalization of education.

Online Education of Marginalized Migrant Children in Treviso
Interviewed teachers were asked to share their stories on how they experienced online teaching during the pandemic, especially regarding their relationship with students from migrant backgrounds and with less knowledge of the Italian language. Particularly about how they reached out to them and through which strategy they included them or not in the classroom.
The answers received were quite similar. All referred to the fact that communicating with foreign 17 students has been a lot more difficult if not interrupted during the online schooling period. These students belong to the category of the most 17 Foreign is here referred only to their citizenship status. absent to classes and sometimes also to the worst performers in the evaluation. Besides, difficulties due to lack of face-to-face communication with the families of newly arrived and non-native Italian speakers have dramatically arisen, leaving those students falling behind the rest of the class in most cases.
In one of my classes, I had seven students out of eighteen who were unreachable. One was even not in Italy, as we found out later. Eventually, some of those students connected to the class, but it was impossible to communicate, especially with their families. (Lower secondary school teacher) Almost all my foreign students were absent and never showed up to my video meetups. I've never received their homework. So, I did not know how to evaluate them.
[…] I am expecting high dropout cases next year. (Upper secondary school teacher) Even though teachers recognized the many-layered difficulties affecting migrant students, they did not always proactively solve hindrances. Especially in secondary education, where students are expected to possess a certain degree of freedom and responsibility, teachers left them alone in most cases. While, to a great extent, primary school education has been delegated to parental education, in the case of a family with a migratory background, parents do not always sufficiently know the language to teach their kids.
Well, foreign students were surely the most absent in my classes. In the beginning, it was impossible to communicate with them. Then, we tried every possible strategy to reach the parents. I used WhatsApp, even though the principal explicitly told us not to use it for school communications. [...] As soon as it was possible to go out from home, I brought copies of the exercises to their houses. Some of them had no computer at home and used their parents' smartphones only when possible. Others have six siblings, and the youngest were penalized. It was very hard for some children to find a quiet place to connect. (Primary school teacher) Even though schools offer tablets and internet connection to students without technological devices, according to the teachers' sample, only a small percentage of the potential beneficiary students have profited from it. Nonetheless, according to ISTAT (Italian National Statistics Institution), in Italy, 33.8% of the families have neither a PC nor a tablet; 47.2% have just one, while only 18.6% have two or more. 18 Primary school teachers have highlighted the loss of the solidarity network surrounding the students from migrant backgrounds. The Italian education system does not systematically provide extra classes for foreign students to learn the new 18 ISTAT Report on Families and Technological Devices, https://www.repubblica.it/tecnologia/ 2020/04/06/news/istat_un_terzo_delle_famiglie_non_ha_pc_o_tablet_in_casa_sud_ penalizzato-253279893/ (25 Jan. 2021).
language (Machetti et al. 2018), which is consistently found to represent one of the most relevant hindrances for immigrant children (Azzolini 2012). Also, in spring 2020, for security reasons, none of the extra activities meant to support migrants' integration occurred.
Everything has stopped. I've seen a decline both in the performance and attendance of the class.
[…] I oversee one F.A.M.I. 19 project, but if the school has stopped, you can imagine extra projects … In Olmi, 20 we have many foreign students and the only way to support them properly, but also to help teachers, is to send them to extracurricular class projects where they learn the language and are tutored in doing their homework. As soon as the school closed, this network stopped. It was three times more difficult to teach, students regressed [...] Now (July) that Piccolo Carro 21 has started again with the extra tutoring, you see how important it is for them to keep regularity in the learning process not only of the language.
If we look at school dropout rates within the migrant student population, they are still significantly high (30.1%) compared to the native population (12%) and affect immigrants' occupational possibilities. 22 A recent study (2020) promoted by Ipsos Italy collected a sample of students aged 14-18 years old attending Italian public schools during the first outbreak of the COVID-19 pandemic and reports 28% of them thinking one of their classmates will drop out of school in 2021. 23 The primary cause is a lack of IT infrastructure and devices, but also the insufficient knowledge of the Italian language is playing a significant role. Language skills deeply affect immigrants' performance in the labor market and reduce their employment rate by about 30% (Pieroni et al. 2019 A small village in the periphery of Treviso, Italy, with a high incidence of foreign students (e.g., 39% of the children in primary school do not have the Italian citizenship). 21 Piccolo Carro is a not-for-profit association, volunteer based and active in the fields of education and inclusion. 22 Eurydice Report: Integrating Students from Migrant Backgrounds, https://eacea. ec.europa.eu/national-policies/eurydice/sites/eurydice/files/integrating_students_from_ migrant_backgrounds_into_schools_in_europe_national_policies_and_measures.pdf (25 Jan. 2021). 23 Ipsos for Save The Children, https://s3.savethechildren.it/public/files/uploads/pubblicazioni/i-giovani-ai-tempi-del-coronavirus.pdf (25 Jan. 2021).

CONCLUSION
Technology is rapidly advancing worldwide, yet even some of the most affordable technological means remain inaccessible for children living in poverty. Switching abruptly to online education has revealed how the educational system in both countries has been unprepared to sustain and implement proper public instruction. Online schooling has stressed the marginalization of students who already live in an economically more difficult situation and could not access their right to education mainly due to a lack of technological infrastructures and devices. However, the lack of a structure aimed at including marginalized poor students, particularly migrant children without sufficient knowledge of the language of the lessons, has contributed to this marginalization. Consequently, many students have suddenly inter rupted their education processes and gone missing from the class. The reasons behind their absences are, nevertheless, multiple and complex. Dropout rates are likely to increase in the next few months, 24 even though the situation's aftermath will be clearer only in a few years. Moreover, online schooling has been perceived as a pure emergency solution and not as an occasion to improve the digitalization of the education proposals. If only given the proper infrastructure, online schooling has great potential in inclusivity, offering concrete means of reducing children's marginalization. Hence, the lessons drawn from COVID-19 emphasize educational reforms, moreover "building an education system that is more resilient, adaptable to student needs, equitable, and inclusive, with a strong emphasis on the role of technology in teacher's training at scale and ensuring learning continuity between the school and the home" 25 should be a priority.


The possibilities of criterion-oriented testing in verifying the achievements of teachers

The article presents the results of empirical research on perception of teachers from military engineering university about the features of didactic testing and on methodologies of diagnostics used by experienced and young teachers. It also presents the results of the survey on students’ learning motivation according to the achievement goal theory. Students have been identified as being oriented towards mastery goal, and concentrate on getting new knowledge and skills that lead to professional and personal growth. It has been proved possible to develop methods for determining the effectiveness of pedagogical action and communicative interaction between the teacher and the student using didactical criterionoriented testing. One of the goals is to assess teacher’s performance as a factor influencing students’ cognitive activity, their results and motivation. The method of formalization by mathematical means of the modeling process and representation of criterion-oriented testing results as a communicative task of interaction between the teacher and the student is described. Prospects for use of theoretical bases, ideas and methods of educational evaluation in criterion-oriented testing and in verifying teachers’ achievements are discussed.

Introduction
Testing represents a form of pedagogical diagnostics as a method of obtaining information about the educational process effectiveness. In line with the competency-based paradigm the student is regarded as motivated active participant of the educational process. His/her activity becomes dominant and determines the possibility to learn, upgrade and use knowledge independently and consciously. Teacher's work does not lose its managerial character. Teacher should create certain conditions in class and organize students' work so that they help to unleash the potential of the learner and increase awareness of knowledge and skills for further professional and personal development. Learners' success as research and educational practice show are largely determined be their motivation and teacher's skills. Adequate (correct) targets and methods of diagnostics, its criteria under certain conditions can not only help assess the performance of learners, but also verify teacher's achievements.
Russian researchers correctly point out that on the basis of test results, it is possible to compare the quality of different textbooks and methodological systems. In this context the problem of diagnostics in higher education becomes relevant as it becomes voluminous and multifaceted. The educator's performance and impact on learners should also be assessed, because it contains many important tasks such as: ability to plan and design a lesson, analyze their own activities, implement different innovative technologies and monitor their effectiveness, motivate students, organize and stimulate their cognitive activity, etc. However, there is currently a lack of methods and criteria of diagnostics of teachers' performance results. The criterion-oriented testing may be used for these purposes.

Materials, methods, results
Testing as a modern diagnostic method relies on scientifically sound statistical patterns in examining large number of inspections of educational results. It ensures the technological soundness of diagnostic procedures, allows storage and comparison of the results. Systematic objective monitoring provides information on the evolution of students' education quality in case of making changes to educational system. It makes possible comparing the quality of education in different countries on the basis of the same criteria. However, criteria-oriented testing is recommended in ongoing didactic testing.
Three main methods were used in the study: analysis of data from researchers on the reported problem, teacher survey, and mathematical simulation of student testing.
The sample survey of teachers from humanitarian departments of the Military Telecommunications Academy (78 respondents) showed that they devote no more than 10-20 % of teaching time to pedagogical diagnostics ( Fig. 1). Teachers are not sufficiently aware of the proper diagnostic instruments to use in accordance with objectives of diagnosing educational outcomes, and are not fully familiar with the concept of pedagogical diagnostics. They generally identify pedagogical diagnostics with didactic control, which mostly focuses on the identification of student knowledge, skills and competencies acquired in the educational process. The individual and personal qualities of students are hardly taken into account. Testing as a method of pedagogical diagnostics is not fully explored. There is a lack of systematic knowledge in the field of testing, including knowledge on use of testing to correct teaching under ongoing pedagogical control. The problem that researchers noted in the eighties of the last century remains relevant today -it is widely believed that tests only measure the outcomes of students, but can't help identifying gaps, and moreover gaps in training [1].
The problem of underestimation of the didactic possibilities of testing, in particular by young teachers, is essential in military education. Thus, according to our research results in the Military Telecommunications Academy, only 0,04 % of adjuncts (beginning teachers) consider testing the most effective method of pedagogical diagnostics (Fig. 2). Both experienced and young teachers use oral and written interviews as traditional ways to assess educational results (Fig. 3). Despite the significant popularity of using tests as an objective measure of educational results testing does not prevail among methods of diagnostics reported by teachers, however experienced teachers use testing more often than young teachers, alternating with traditional surveys.  In Russian pedagogy pedagogic diagnostics appears as an independent theoretical and applied field. It examines the rules of the diagnostic procedure, principles, methods and forms of diagnostics [2]. Pedagogic control is considered an integral part of pedagogic diagnostics -a system of scientifically based verification and evaluation of the results of education, training and personal development. There are instrumental and subjective methods of pedagogical control distinguished. Instrumental methods include testing (to assess knowledge) and instrumental observation (to assess actions). Subjective methods include oral survey, validation interview, written paper, pass/fail exam, control observation, etc. Four main functions of pedagogical control are identified: diagnostic, educative, organizing and disciplinary [3].
Diagnostic function involves regular analysis of learning outcomes to obtain objective and reliable information on the education process, results of learning activities and trends both positive and negative. This makes it possible to make the necessary adjustments to the teaching process itself as well as the monitoring and evaluation of learners' knowledge. Educative function is not only about acquiring knowledge, skills and competencies but also about their maintenance. It usually takes place in the course of repetition, refinement and synthesis of the content studied. This function, which is one of the most important, contributes to deepening, broadening and sustaining knowledge, skills and competencies, and also promotes interdisciplinary linkages. Organizing function is carried out by orienting learning towards a specific goal. This function is oriented to creating positive motivations in the learning process, creative atmosphere of competition and favorable conditions for cognitive expression. It is also aimed at stimulating the initiative of learners. The central organizing point of this function is the teacher's commitment to the development and use of content, forms, methods, techniques and tools to use for the goal. Disciplinary function is aimed at disciplining and educating students, and training them in systematic work. It is oriented to the development, in the process of pedagogical control, of the moral responsibility for the results of their own learning and the perseverance in achieving excellence. It also addresses building of professional qualities and the desire for continuous learning.
The review of the works of Russian educators devoted to the functions of pedagogical diagnostics is presented in the Table 1. Pedagogical control in secondary vocational education system: functions, forms, types. [6] 2018 Diagnostic, educative, organizing, disciplinary 4. Dairi N.G. Content and the logic of its study [7] 1987 Control, evaluative, educative, evolutive, disciplinary 5.
Remedial control as a factor for improving the quality of teaching (on humanitarian cycle materials) [3] 2004 Diagnostic, educative, organizational, disciplinary Testing and evaluation functions in training [8] 1980 Orientation, diagnostic, educative, disciplinary, evolutive, control, administrative 7.
Pedagogical psychology [11] 1998 Diagnostic, control and evaluative, feedback, organizing, disciplinary Research on quality measurement in education [12] shows that even a slight increase in the motivation of learners influence the quality of educational process more than introduction of innovative learning technologies. Traditionally external motivation and internal motivation have been singled out in Russian pedagogy [13]. In modern studies two main theoretical approaches prevail where motivation is associated with different types of objectives. In the achievement goal orientation theory, there are two basic goals: mastery goal and performance goal [14]. Students who focus on the mastery goal concentrate on knowledge and skills that lead to personal and professional growth. Performance oriented students seek to demonstrate their skills and knowledge [15], obtain from others desired assessment of their competencies or avoid negative evaluations [16]. Result orientation is usually associated with superficial attitudes to learning and is considered to have an adverse effect on the emotional, cognitive and behavioral components of learning [17]. Mastery goal orientation is seen as stimulating learning activities and positive emotions towards them. All things been equal, this type of orientation has a positive effect on student achievements as expressed in the test results. And it is this kind of value orientation that the most successful teachers stimulate in their students.
N.G. Maloshonok et al. [18] indicate that their studies of Russian students reveal the existence of all these types of goal orientations and suggest to share in mastery goal orientation the goals of professional and personal development, as they directly influence the student's learning strategy. From our sample survey of second and third-year students from Military Telecommunications Academy (100 respondents, 2020) it can be seen that the predominant purpose of their higher education is to acquire professional skills (88 %). Result orientation (getting a high rating, or a high-ranking position in the rating system) is characteristic of only 10 % of respondents. Thus, the training motivation of students is mainly related to their interest in and aspiration to the future profession.
Selection of the theoretical framework to verify students' achievements using criterionoriented testing depends on which aspects of motivation it is most important to study. The choice of empirical methods of motivational research is also broad [19,20] and can be recommended in verifying the achievements of teachers.
To what extent and by what means all functions in ongoing control are implemented depends on teacher's personal qualities, activities, motivation and possession of appropriate methodologies. Generally, all the factors that affect the educational outcomes of students might be diagnosed. And criterion-oriented tests with a clear focus on objectives and criteria should be selected as a tool.
Criteria-oriented testing requirements are justified in the works of Pechnikov A.N., etc. [21,22]. These are the requirements of reliability, diagnosticity, efficiency, formalizability, and multicriteriality. The reliability requirement includes the independence of the procedure from the examiner's opinion; using the scale with the resolution which corresponds to the resolution of the form of the test to present the results; the consistency of the procedure for evaluating test results for all subjects; interval estimation requirement [21]. The multicriteriality defines that control results should reflect all factors influencing the achievement of such results. It gives the possibility of test results evaluation (correctness and time of test tasks) and other subject's parameters that affect the test result.
To illustrate the application of the mathematical tools of modeling and interpreting the results of criterion-oriented testing of students, let us cite the following. The research report [23] by one of the authors (with co-authors) shows that in ongoing pedagogical control the following assumptions can be made: 1) the complexity j I of each test task is determined by reference (normative) complexity ýò j I of a tested educational element, i.e. ýò j j I I = ; 2) each of the educational elements was presented to the learners in the test with the same instruction; 3) the learning and motivation of all learners in the process of study of all educational elements remained the same.
Then the result of any test performed according to accepted assumptions can be presented as: The search of sets   where the values of the coefficients As a result of criterion-oriented test data processing using conventional method the teacher gets information about the only characteristic that is useful for achieving the goals of ongoing pedagogical control. The characteristic represents the estimations ( )

Discussion
M.V. Guskova draws attention to the fact that educational results evaluation is oriented to the competencies with meta-latent nature and delayed nature of display. Another thing is that the process of competency building is influenced by contextual factors and students' learning activity development background and learning motivation. Therefore, modern ideas and methods of evaluation should be used in diagnostics [24]. Evaluation is an area of assessment, collection and analysis of information, and, at the same time, of management decision-making apparatus and external monitoring [25]. A hierarchy of factors can be identified in educational evaluation (updating the content of education, adopting innovative organizational and technological pedagogical decisions, introduction of new teaching methods, changing the style of interaction with students, improving the quality of the monitoring system in educational institutions, strengthening the involvement of students, etc.). Independent variables and predictor variables are identified for the study of subjects of educational evaluations (for which groups of evaluation subjects differ). For example, if one needs to figure out the effects of competencies on follow-up activities of graduates of educational institutions, it is possible to consider the requirements of Federal State Educational Standards in different areas as predictors, etc. Dependent or criterion variables are those that can measure the differences between groups of subjects achieved during the training program. In education, learning achievements and other educational results serve as criterion variables. To solve the problems identified in our study, indirect causation can be used in evaluation. For example, if the quality of the measures influences student motivation to acquire the competencies, the proxy variable will be the magnitude of the measurement error, because subjective estimates obtained with great measurement error reduce student learning motivation.
Thus, integrating the idea and methodology of evaluation into the system of diagnostic brings the true meaning to the diagnostics itself as a procedure that helps to identify the causes of test failures and errors, and to effectively implement training functions based on diagnostic results.
Note that national standardized tests as well as exams, portfolios, interviews, debates, discussions, professional work experience, descriptive methods, simulations, and observations can be used in verification of informal and nonformal education results in the teacher qualification system and validation of teachers' professional experience. The experience and communication skills of an examiner play a crucial role in this case to ensure validity, reliability and fairness of the process [26]. Verification of learning outcomes is the scope of criterion-oriented testing application. Non-trivial task of development of the system of verification teachers' professional achievements, especially the results of their informal and nonformal education, is now been addressed in modelling regional systems of teacher professional development in Russia [27] Question of creating the conditions for the conduct of such a procedure for teachers may need to be addressed in the development of knowledge and competency verification for any professional area. In such testing, it is important to determine: whether the assessment tools are adaptable to personal features of a teacher; to which extent the assessment tools are robust and sound; what set of criteria/standards are used and how accurate they make possible to record an individual set of acquired experience; to which extent the evaluation procedures, tools and standards are clearly defined and explained to teachers, employers and educational institutions [28].

Conclusion
As can be seen from the above, pedagogical diagnostics as a knowledge-based technology should be based on a number of principles serving as its starting point and at the same time as the conditions for its organic incorporation into the holistic pedagogical process: consistency and specificity; scientific soundness of diagnostic theory and methodology; combination of diagnostic detection and correction functions; consistency and continuity of the diagnostic system; accessibility of diagnostic procedures to teachers and students; optimization of diagnostic forms and methods; comprehensiveness and predictiveness of diagnostics.
Motivation has a strong influence on learning outcomes, including those remote, important for professional development, determines the choice of educational strategies by students. On the other hand, it is teachers who have a significant influence on motivation. Through special studies, it is possible to identify, for didactic and educational purposes, the influence of the teacher in charging the motivation of the course, and thus take into account the impact of a particular type of motivation on diagnostic result. And back again, studying test strategy or using special tests, investigate student motivation and then its impact on academic success. On the other hand, knowledge of factors influencing criterion-oriented testing results, allows to refine mathematical models of its design and evaluation results. Provisions developed in the study on usability and features of criterion-oriented testing and the findings can be used to verify pedagogical achievements and in similar further studies.


Most significant barriers and proposed solutions for medical schools to facilitate simulation-based undergraduate curriculum in OBGYN

Despite having a good understanding of medicine, doctors lack clinical skills, problem-solving abilities, and the ability to apply knowledge to patient care, particularly in unanticipated circumstances. To overcome this, medical education has evolved into a system-oriented core curriculum with cognitive, psychomotor, and affective learning goals. With an emphasis on problem-based learning, the educator’s aim is to establish a long-term, predetermined improvement in the learner's behavior, acquired skills, and attitudes (Datta R, Upadhyay KK, Jaideep CN. Simulation and its role in medical education. Med J Armed Forces India. 2012;68(2):167–172. https://doi.org/10.1016/S0377-1237(12)60040-9). However, teaching these disciplines to real patients is almost impossible; this is where simulation comes in. This opinion paper will discuss the relevance and necessity of a simulation-based undergraduate curriculum in obstetrics and gynecology. What are the biggest obstacles that medical schools face in making the most of simulation-based learning, and how can they be overcome?

Introduction
One of the most fundamental strategies in this era to teach crisis resource management skills to promote the professional development of frontliners is introducing the simulation-based medical curriculum, which will provide comprehensive and realistic training and safer patient care in several ways. It is a process, not a technology, for promoting experiential and reflective learning, intending to instill the proper mindset and skills to cope competently with reallife sensitive situations in a planned and prescribed manner while respecting patients' moral and legal rights [1].
To prevent any potential breach of patient privacy, students must not involve directly in any patient care; however, there is a need to attend an orientation before beginning clinical practice. The simulation model assists medical students in gaining clinical experience and understanding the progression and clinical manifestations of diseases in specific patients. Simulation allows students to gain valuable experience in areas where resources are scarce but technological skills are essential. Effective simulator models are available for teaching shoulder dystocia, vaginal breech delivery, obstetric emergencies, operative hysteroscopy, and laparoscopy [12].
What are the challenges medical schools facing to integrate simulation-based learning for undergraduates? Furthermore, what are the potential options for overcoming such obstacles?
One of the most important roadblocks is the expense of implementing simulation-based medical education. A well-conducted economic evaluation can assist stakeholders such as program directors, regulators, and curriculum designers determine the best use of resources when choosing the simulation modality or assessment method. For example, a great deal of practice is necessary for procedural competencies as we do not want medical students to improve this ability with patients in the early training stages. So, we can use everything from citrus fruits to board games to hand-crafted corduroy and neoprene versions of vaginal cuffs. These low-fidelity simulation models do not include artificial intelligence; nevertheless, they allow students to practice throwing nodes before entering the operating room [13]. In specific training scenarios, 1 3 hybrid simulation, which involves using a patient actor in conjunction with a task-trainer during the same session, may be as effective as high-fidelity simulators while also providing superior training contexts to enhance patient learners' relationships with caregivers and better immerse trainees in the feelings and emotions of the scenario. There is evidence that a hybrid-birthing simulator enables quick maneuvers for handling a shoulder dystocia delivery and is beneficial for non-technical skills like communication and team training [16]. High-fidelity modeling recreates a scenario to replace actual patient encounters with a supervised clinical practice that mimics significant aspects of the real world in a completely immersive approach. Highfidelity birthing simulators are ideal for teaching both natural and assisted vaginal deliveries.
To conclude, the potential option for getting over the expense roadblock is to carefully evaluate which type of fidelity simulation model is suitable for optimal patient safety outcomes by the clinical simulationist because money invested in the medical industry directly affects patient lives. Task coaches, scenario manikins, or human patient simulators of full-fidelity may achieve various goals, depending on the desired outcome. Economic assessment in medical education research is still in its early stages. However, there is much room for the state-of-the-art application of these methods in this field [9]. If you are looking for a more detailed breakdown, try "Comprehensive simulation in healthcare: operations, technology, and innovative practice" written by Scott B. Crawford, MD, FACEP, CHSOS, the Director for Simulation at TTUHSC El Paso, is a practical guide to help practitioners prepare for a wide range of simulations in healthcare. The book starts with an introduction to the simulation of healthcare, including staff, curricula, and space. Eight core areas of knowledge/skill discuss the fundamental aspects of healthcare simulation technology specialist in the following chapters. Finally, it addresses the advantages of establishing a relationship with business partners and provides best practices and innovations.
The second most difficult challenge in simulation-based education is maintaining an SP Program. As per William Osler, "For the junior student in medicine and surgery, it is a safe rule to have no teaching without a patient for a text, and the best teaching is that taught by the patient himself." Simulated patients trained in a structured manner are known as low-technology resources, but running this program is costly as their compensation rate is $20.00 per hour [14]. However, SP's professional input is the secret to the success of the simulation-based training, and they have a high level of realism and potential for teaching general and specialized communication skills. So, it is best to utilize standardized patients in training scenarios where peer role-play is inappropriate, such as breaking bad news or providing a trainee assessment alternative.
The emerging trend of using simulation as a teaching method for assessing and preparing students is reflected in medical schools and their curricula as they are responsible for designing the curriculum and ensuring that adequate educational facilities are available to shape the futures of today's undergraduates-tomorrow's physicians. In order to facilitate both outcomes-based and competency-based medical education, there is a need to develop a simulationbased procedural skills curriculum and this could be one of the difficulties encountered in the simulation-based medical education of OBGYN undergraduates. To make this obstacle easier to overcome a non-profit, membership-based organizations with the ultimate aim of delivering optimal health care to women like APGO for women's health educators are now offering contemporary, applicable educational resources for physician-educators and learning strategies for faculty, students, and residents. The Undergraduate Medical Education Committee (UMEC) of the Association of Professors of Gynecology and Obstetrics (APGO) provides educators with an overview of the use of simulation in undergraduate medical education in the field of obstetrics and gynecology in the "To the Point" series and the APGO Medical Student Educational objectives, 11th edition, provides a succinct overview of the goals and objectives of a complete Obstetrics and Gynecology clinical program. Members of the UMEC are responsible for assisting APGO in achieving its mission and objectives; the primary responsibilities of this committee are to provide consultation service to departments concerning undergraduate medical education programs and establish, build, and evaluate a variety of teaching and learning [2].
Faculty expertise and competencies in debriefing and simulator logistics are critical components of successful SBME delivery. Several studies have shown that debriefing preparation workshops are a feasible and effective method of faculty training. Debriefing experience can be gained by completing simulation instructor training courses provided by various simulation programs, attending seminars, at conferences, and pursuing fellowship training or advanced degrees in simulation [4].
In situ simulation drills during clinical ward postings are among the most successful ways to introduce simulationbased education for undergraduates as sitting and listening to lectures does not allow for significant activation, which is the essential learning and memory retention element. Simulation involves the team, but it also allows them to practice in a more supervised environment. An in-situ simulation is much more rewarding than an off-site simulation, which means simulation occurs in the actual clinical environment. This method helps the learners to learn technical skills, teamwork, and communication in a real-world setting. It enables the detection of possible facility problems and obstacles to treatment which, of course, is challenging to gain in a simulation lab or other off-site locations. By this, we not only enhance communication and coordination between doctors and midwives by including training at the undergraduate level, as well as break down professional barriers before they develop. This is especially important in low-middle-income countries with high rates of maternal and perinatal mortality and a deeply ingrained hierarchical healthcare system [6]. Obstetrics is a team-based, high-risk specialty with a high level of litigation. Failures in teamwork lead to poor outcomes for mothers and babies; therefore, effective multiprofessional teamwork is important. Furthermore, obstetric team simulation preparation shows healthy neonatal outcomes after obstetric complications. Simulation training is increasingly being used for undergraduate training because it helps students to consolidate their learning and apply their expertise and experience in practice [3]. Simulation for obstetric emergencies is a relatively recent area of interest, but it increases patient safety and outcomes. It is becoming an essential component of both education and patient care. The Council on Patient Safety in Women's Health Care has released "Practicing for Patients: In-Situ Simulation Program Manual", whose main agenda is that if all members of the labor and delivery team practice and simulate medical emergencies on their actual labor and delivery unit, they will reduce obstetric related morbidity and mortality by improving the team's communication and response in a medical emergency [11].

How to promote simulation-based medical education in the COVID-19? Being ready for post-pandemic transformation
Above all obstacles, one of the most significant challenges that the entire world faced during the COVID 19 pandemic was on the educational system. Computerized simulation education products are in high demand for healthcare training environments, which eventually necessitate vast infrastructures. Significant obstacles to using virtual simulation in clinical education started from clinical simulation training operating systems, system architecture technology, encrypted data, cloud-based eLearning, and a professional technical team. Therefore, investing in simulation-based instructional management technologies and resources to keep clinical education and competency training on track during the pandemic is an expeditious need today [15]. The use of digital technologies for educational purposes, such as virtual, mixed, and augmented reality, are likely to be the most significant aspects of the transformative and post-COVID medical education [5]. A possible solution for quickly adapting to these new technologies is to develop courses for teachers and lecturers on how to prepare courses for AR/VR and a structure that allows teachers to adapt their material for AR/VR quickly [10]. Not only advance simulation, but according to one study, OBGYN students, residents, and fellows who lack hands-on experience due to the COVID-19 pandemic benefit from the use of homemade simulator models as well [7].
One of the authors stressed the importance of reopening simulation labs and facilities by following simple and doable strategies like establishing a COVID-19 response task force, using space, establishing access conditions to the simulation facility, personal hygiene measures, use of protective devices, physical distancing, management during training activities, personnel management, cleaning/ disinfection, and review of emergency plans/procedures. The mentioned are realistic focus points with operational tips and mitigation advice to prepare the reader to reopen SF after the lockdown safely. So plan the future event with protection and versatility in mind [8]. In addition to this, including medical students in the healthcare simulation center's staff is one way to support undergraduate simulation-based medical teaching [17].

Conclusion
According to the points raised, it is clear that there are many opportunities for facilities to integrate simulationbased curriculum, especially for OBGYN undergraduates including: the appropriate use of resources like low, hybrid, and high fidelity simulators; the use of contemporary simulation models like VR/AR and MR for teaching basic medical sciences; utilization of standardized patients programs in training scenarios where the alternate strategy is not possible; by designing evidence of curriculum documentation; the strategy of curriculum review with the help of APGO Medical Student Educational Objectives Guide; overcoming the barrier of simulation training by running workshops to train faculty; effective use of in situ simulation drill and last but not the least be innovative and ready to adapt to changes during the pandemic situation and have sound knowledge and urge to get the latest updates in simulation technology. However, there are still some gray areas with room for improvement and much scope for researchers to assess how best to incorporate simulators in a more primary phase of medical education.
Funding This study is supported by University of Dundee.

Declarations
Conflict of interest I have no conflicts of interest to disclose.


extfieldsofstudy:
Computer Science
Physics

text:
Continuous black-box optimization with quantum annealing and random subspace coding

A black-box optimization algorithm such as Bayesian optimization finds extremum of an unknown function by alternating inference of the underlying function and optimization of an acquisition function. In a high-dimensional space, such algorithms perform poorly due to the difficulty of acquisition function optimization. Herein, we apply quantum annealing (QA) to overcome the difficulty in the continuous black-box optimization. As QA specializes in optimization of binary problems, a continuous vector has to be encoded to binary, and the solution of QA has to be translated back. Our method has the following three parts: 1) Random subspace coding based on axis-parallel hyperrectangles from continuous vector to binary vector. 2) A quadratic unconstrained binary optimization (QUBO) defined by acquisition function based on nonnegative-weighted linear regression model which is solved by QA. 3) A penalization scheme to ensure that the QA solution can be translated back. It is shown in benchmark tests that its performance using D-Wave Advantage$^{\rm TM}$ quantum annealer is competitive with a state-of-the-art method based on the Gaussian process in high-dimensional problems. Our method may open up a new possibility of quantum annealing and other QUBO solvers including quantum approximate optimization algorithm (QAOA) using a gated-quantum computers, and expand its range of application to continuous-valued problems.

A black-box optimization algorithm such as Bayesian optimization finds extremum of an unknown function by alternating inference of the underlying function and optimization of an acquisition function. In a high-dimensional space, such algorithms perform poorly due to the difficulty of acquisition function optimization. Herein, we apply quantum annealing (QA) to overcome the difficulty in the continuous black-box optimization. As QA specializes in optimization of binary problems, a continuous vector has to be encoded to binary, and the solution of QA has to be translated back. Our method has the following three parts: 1) Random subspace coding based on axis-parallel hyperrectangles from continuous vector to binary vector. 2) A quadratic unconstrained binary optimization (QUBO) defined by acquisition function based on nonnegative-weighted linear regression model which is solved by QA. 3) A penalization scheme to ensure that the QA solution can be translated back. It is shown in benchmark tests that its performance using D-Wave Advantage TM quantum annealer is competitive with a state-of-the-art method based on the Gaussian process in high-dimensional problems. Our method may open up a new possibility of quantum annealing and other QUBO solvers including quantum approximate optimization algorithm (QAOA) using a gated-quantum computers, and expand its range of application to continuous-valued problems.

I. INTRODUCTION
Given a black-box function f (x), x ∈ d , whose closed-form expression is unknown, Black-box optimization (BO) identifies the extremum via an iterative procedure of input design and evaluation [1,2]. Figure 1 shows the procedure of software-based black-box optimization. First, the black-box function is estimated using a machine-learning prediction model f ML (x) trained by known dataset D = {(x 1 , y 1 ), ..., (x n , y n )}. Using a trained model, an acquisition function g(x) is defined, and the next input x * is designed by optimizing it. Next, the value of black-box function at the recommended point y * = f (x * ) is evaluated by a black-box system, e.g., simulation or experiment. A new data (x * , y * ) is added to D. These steps are iterated as long as the budget allows. Black-box optimization has been applied to many physics applications including autonomous X-ray scattering experiments [3], inverse scattering [4], crystal structure prediction [5], design of organic synthesis experiments [6], and effective model estimation [7].
It is expected that BO accelerates extremum identification in comparison to random sampling. Namely, at the same number of evaluations, the best input vector by BO should be closer to the extremum than that of random sampling on average. It is known that, however, the advantage of BO quickly diminishes as the number of dimensionality increases [8]. There are two possible reasons, statistical and computational. In statistical terms, it is difficult to estimate f (x) with a limited number of training data in a high-dimensional space. In computational terms, the optimization of a non-convex acquisition function becomes increasingly difficult as the number of dimensionality increases.
In this paper, to solve computational problem for continuous black-box optimization, we propose a hardwareassisted black-box optimization algorithm called CON-BQA (CONtinuous Black-box optimization with Quantum Annealing, pronounced like kombucha) [9] that employs a quantum annealing [10,11] as an acquisition function optimizer. Here, we use quantum annealers developed by D-Wave Systems, Inc. [12] which are special hardware for searching the ground state of Ising model or solving quadratic unconstrained binary optimization (QUBO). For our method, instead of quantum annealer, we can use other Ising machines established using characteristic physics systems such as, digital circuit [13], CMOS [14], laser oscillator [15,16], GPU [17], and so on. Recently, Kitai et al. [18] proposed a method that solves a discrete black-box optimization problem with the help of quantum annealing, which is called FMQA algo-  [19,20]. Figure 1 illustrates the procedure of CONBQA. 1) Continuous data x are encoded to binary data z. 2) Machine learning is performed and an acquisition function g(z) is prepared.
3) The QUBO defined by g(z) and additional constraints to ensure decodability is solved by a quantum annealer to obtain maximizer z * . 4) The solution is decoded back to a continuous vector x * . 5) The value of black-box function y * = f (x * ) is evaluated. 6) A new data point is added to the dataset.
Continuous-to-binary encoding is a common subject of research in scientific fields including neuroscience [21,22] and machine learning [23,24]. Among existing methods, we employ random subspace coding [22] due to its simplicity and high resolution. Several coordinates are randomly chosen and an axis parallel hyperrectangle is sampled randomly in the subspace. A continuous vector is mapped to a binary variable by projecting it to the subspace. If the mapped point is included in the hyperectangle, the vector is mapped to one, otherwise zero. By using m hyperrectangles, a continuous vector is mapped to an m-dimensional binary vector. In Fig. 2 (a), example of random subspace coding in the two-dimensional continuous space is shown. Here, two-dimensional coordinates are selected to generate hyperrectangles, and m = 2 case is considered. Two rectangles divide the whole space into four regions. It corresponds to an encoding to a binary vector with two bits, i.e., if the continuous vector is included in the left blue hyperrectangle, the first bit is 1, while for the right red rectangle, second bit becomes 1 if continuous vector is located. This example is surjective encoding. On the other hand, it is worth noting that the random subspace coding is not necessarily surjective. Figure 2 (b) is an example of non-surjective encoding. In this case, there may exist binary vectors that cannot be decoded to continuous vectors, i.e., '11' in this figure. In addition, for three-dimensional case, an example of the random subspace coding is shown in Fig. 2 (c). Here, the continuous vector located at the red point is mapped to the binary vector where the bits expressing bold hyperrectangles are 1 and the others are 0.
In CONBQA, as the machine learning model in the binary space, a nonnegative-weighted Bayesian linear model is employed, and the acquisition function based on the trained model is defined. We prove that the global optimal solution of the acquisition function under constraints to ensure decodability is always decodable via Helly's theorem. Interestingly, nonnegativity constraints in the prediction model are essential in proving the property. To use quantum annealer, a QUBO is formulated with the acquisition function and additional constraints. Note that it is not really necessary to obtain global optimal solution of QUBO by quantum annealing, because decodable solutions are exist even if not global optimal solution. Thus, the task of quantum annealing is to search decodable solutions with as large acquisition function as possible. In numerical experiments using benchmark functions, it was shown that our method is competitive with Add-GP [25], which is one of the state-of-the art method for high-dimensional Bayesian optimization [8].

A. Random Subspace Coding
In random subspace coding in a d-dimensional space, the dimensionality of subspace d s is first determined. Let I = {i 1 , . . . , i ds } denote the indices of randomly chosen coordinates from [1, d], and {[l i , u i ]} i∈I be closed intervals in respective coordinates. Here, l i < u i , l i ∈ and u i ∈ . Random sampling of each interval [l i , u i ] is performed using the Dirichlet process method [26]. Given a positive integer N , this method can generate an interval such that any point in [0, 1] along each selected coordinate is included in the interval with probability 1/N . Throughout this paper, N is fixed as 3. Then, an axis-parallel hyperrectangle R is determined as (1) Later on, we denote it as "rectangle" for simplicity. Using m rectangles R 1 , . . . , R m , the random subspace coding is defined as z k = I(x ∈ R k ), where I(·) is one if the condition is satified and zero otherwise. Given a binary vector z = {z k } k=1,...,m , a point x ∈ d corresponds to z, if x is included in all rectangles with z k = 1 and not included in those with z k = 0. Let P denotes the intersection of all rectangles with z k = 1, Let N denote the union of all rectangles with z k = 0, Since the preimage of z is described as

B. Black-Box Optimization
In CONBQA, the training dataset D are encoded to binary dataset D = {(z 1 , y 1 ), ..., (z n , y n )} through random subspace coding. Here, {y i } i=1,...,n is normalized by min-max normalization, and thus these values are nonnegative. Then, a QUBO is prepared from D and its optimal solution z * is obtained with a quantum annealing. Next, z * decoded to a continuous vector x * , and fed to the black-box system.
To model an underlying function from D , a nonnegative linear model: is employed, and trained with least-squares fitting to D . Let w * = {w * k } k=1,...,m denote the trained parameter vector. Thus, the acquisition function is defined as The unique solution with maximum of g(z) has z k = 1 for ∀k. However, this solution may be a non-decodable solution, because it is almost impossible to exist the region, where all rectangles are intersected. That is, unconstrained maximization of the acquisition function may lead to a non-decodable solution. To keep the optimal solution decodable, we add the following constraint, z ∈ C where The meaning of this constraint is that the data point do not belong to R i and R j when two rectangles are not intersected. The constrained problem is then described as In Sec. III, we will present a proof that the optimal solution of Eq. (7) is decodable. The problem by Eq. (7) is treated as a QUBO and solved by a quantum annealing. As done in many application studies of quantum annealing, constraints are replaced by a penality term as follows, where A and B are positive hyperparameters. To make contributions from two terms comparable, we use A = 1/ max k {w * k } and B = 1. Once the solution z * of Eq. (8) is obtained by quantum annealing, a point x * is randomly sampled from the preimage P (z * ) − N (z * ), evaluated to yield a new outcome y * by the black-box system. The new data point (x * , y * ) is then added to the dataset D, and the next iteration is put forward.

III. DECODABILITY
To understand Eq. (7) better, we formulate it as a graph-theoretic problem. Let us define m nodes corresponding to all rectangles, and make an edge if the corresponding rectangles are overlapped. Put a nonnegative weight w * k to each node. Then, the acquisition function defined by Eq. (5) is the sum of all node weights of the subgraph induced by the nodes with z k = 1. Let us call the rectangles with z k = 1 and 0 positive and negative rectangles, respectively. The nodes corresponding to positive and negative rectangles are called positive and negative nodes, respectively.
Consider the following two conditions. 1) There is no edge between node i and j. 2) Both of the nodes are positive. If both of the conditions are satisfied for z, it is not decodable, because the intersection of two nonoverlapping rectangles is empty, hence P (z) = ∅. The constraint of Eq. (6) ensures that this situation does not happen, and the positive node set always forms a clique. Assume that the positive set does not form a clique. Then, there is at least one pair of positive nodes without an edge, violating the constraint. In summary, the optimization problem corresponds to a maximum clique problem [27], where the weight of a clique is defined as the sum of all node weights.
Let us prove that the global optimal solution z * of Eq. (7) is decodable. As shown before, the decodability condition is written as P (z * ) − N (z * ) = ∅. To prove it, it is sufficient to show P (z * ) = ∅ and P (z * ) ∩ N (z * ) = ∅. Since the optimal solution corresponds to a clique in the graph theoretic problem, all pairs of positive rectangles overlap. Due to Helly's Theorem of convex sets, the intersection of all rectangles P (z * ) is nonempty. Let us assume that P (z * ) ∩ N (z * ) = ∅. Then, P (z * ) has to have overlap with one or more negative rectangles. If a negative rectangle has overlap with P (z * ), it has overlap with all the positive rectangles. In that case, one can build a larger clique than the current one by including the negative rectangle. Since node weight is positive, larger clique should have larger weight. It contradicts the assumption that the current clique has maximum weight, thus P (z * ) ∩ N (z * ) = ∅.

A. Results with D-Wave Quantum Annealer
Using D-Wave Advantage, we implement a hardwareassisted system for continuous black-box optimization. In the previous section, we proved that, if the hardware optimizer finds the optimal solution of Eq. (7), it is always decodable. In reality, however, a quantum annealer and other algorithms solve the QUBO problem defined by Eq. (8) and do not always return the ground state. Thus, there are the following three possible cases when the solution z * is calculated by quantum annealing: 1. Empty: P (z * ) = ∅.  For a decoder, if z * with decodable or admissible is recommended, the next data point x * is randomly sampled from the region with P (z * ) = ∅. On the other hand, when the solution is empty, x * is randomly selected from the whole search space.
To test the performance, we employ a 6-dimensional function called Hartmann-6 [28] that is often used for testing global optimizers. In this testing function, whole search spaces are determined as [0, 1] in all coordinates. For all experiments, the parameters of random subspace coding are determined as m = 60 and d s = 2. The performance of CONBQA is measured by regret S T , which is the difference between the optimal value of Hartmann-6 and the best one observed so far in the black-box optimization. Thus, the purpose of the present continuous black-box optimization is to obtain small S T with as few iterations as possible. Figure 3 shows the regret in logarithmic scale against the number of iterations for CON-BQA, AddGP based on UCB score, and random search. In the 15 initial steps, x is randomly sampled from the whole search space, and afterwords, iterations controlled by CONBQA or Add-GP are performed. Here, the average over 10 independent runs, where initial randomlysampled data are different, is shown with the error indicating the standard deviation. CONBQA with D-Wave Advantage with hybrid solver service performs better than CONBQA with simulated annealing and that with greedy search. It outperforms Add-GP based on UCB score and random search as well. 10 independent runs are performed and shaded region is the error.
As a QUBO solver used in CONBQA, we employ D-Wave Advantage with hybrid solver service[29], simulated annealing using dwave-neal with default setting[30], and greedy search. It is remarkable that when the number of iterations is small, regardless of QUBO solvers, CONBQA outperforms Add-GP which performed best in a previous benchmark study [8]. Of course, in comparison with random search, better optimization performance of CONBQA is confirmed. This result indicates that our framework to solve continuous black-box optimization using random subspace coding and QUBO solvers as acquisition function optimizer is work well. In addition, among the CONBQA variants, the quantum annealer performed best. In particular, optimization result by quantum annealer is quite better than that by greedy search, indicating that the performance of continuous black-box optimization can possibly be improved by a better QUBO solvers, i.e., specialized hardware.
Next, we address the decordability in optimization process. Table I shows the fraction of empty, admissible, and decodable solutions in the iterations in CONBQA depending on the QUBO solvers. As an important fact, we have never observed empty solutions by three methods to solve QUBO. Simulated annealing and greedy search had a higher fraction of decodable solutions than the quantum annealer despite that they performed poorly in optimization (see Fig. 3). This result indicates that the solutions obtained from the simulated annealing or greedy search are well satisfying the constraint of Eq. (6), but tend to stuck at local minima with larger values of the first term in Eq. (8). On the other hand, quantum annealer recommends many admissible solutions, which were not fine-tuned for the constraint, but better optimization is realized. This means that the solutions by quantum annealer have smaller value of the first term in Eq. (8) instead of fulfilling the constraint exactly, and CONBQA could get closer to the global optimal solution.

B. Benchmarking with Simulated Annealing
The performance of CONBQA is tested further with three additional benchmark functions from 2013 IEEE CEC Competition on Niching Methods for Multimodal Optimization [31]. Among the 20 functions provided, three high-dimensional functions of F 12 (5D), F 11 (10D), and F 12 (10D) are chosen. Their dimensionality is 5, 10, and 10, respectively. To try some experimental settings with different number of encoding bits, m, we perform CONBQA with simulated annealing. In these testing functions, whole search space are determined as [0, 1] in all coordinates. Figure 4 shows the regret in logarithmic scale against the number of iterations for CONBQA with m = 20, 60, and 100, Add-GP, and random search. For random subspace coding, d s = 2 is used. The average over 10 independent runs, where initial 15 randomlysampled data are different, is shown with the error indicating the standard deviation.
For relatively low-dimensional dataset of F 12 (5D), CONBQA is superior in finding better solutions in comparison with Add-GP regardless of m. In this case, m = 60 shows the better optimization performance. If m is increased, the resolution of random subspace coding becomes high, which is discussed in the next subsection, and we can get closer to the optimal solution in principle. On the other hand, the regression model becomes complicated, and in an early stage of iterations, an accurate prediction model is difficult to construct by statistical problems. Thus, an appropriate value of m will be determined by this trade-off and is strongly depending on the target problem.
For 10-dimensional datasets (F 11 (10D) and F 12 (10D)), CONBQA outperforms Add-GP in an early stage, but slightly worse than Add-GP at the final stage of optimization. This is because CONBQA cannot get exactly to the optimal point due to the finite resolution of random subspace coding. Note that this resolution problem can be resolved by using large m, but the tradeoff described above should be appeared. Thus, it is not always necessary to make m larger to obtain better op- timizations. In applications such as experimental and simulation conditions tuning in physics, chemistry, and materials science, the number of iterations is severely limited. That is, the early stage of optimization is more important for these applications. In such cases, CONBQA would be preferred over Add-GP.

C. Higher Dimensionality
In the benchmark, CONBQA was applied to up to 10 dimensional problems. Since the ability of Ising machines is growing rapidly, CONBQA will likely be applied to problems with higher dimensionality in the future. To estimate how many bits are necessary for CONBQA, we investigated the resolution of random subspace coding for d = 10, 100, and 1000. The resolution is characterized by the average size of the intersection of all rectangles enclosing a randomly chosen point which is represented by R I . The size of a rectangle is measured by the average of its edge lengths. If R I is small, in a decoder, the possible region of x * is small when z * is given. Thus, the high resolution by random subspace coding is realized when R I is small enough. Figure 5 shows R I depending on the number of bits for d = 10, 100, and 1000 for d s = 2. Here, the average over 50 randomly chosen points is shown with the error indicating the standard deviation. In our benchmarking in Sec. IV B, if 100 bits are used for 10-dimensional problems, a better performance than Add-GP in an early stage of optimization is realized in CONBQA. According to this fact, R I 0.4 would be enough to work CON-BQA effectively for d = 10. Thus, with 10,000 bits, it is possible to reach sufficiently high resolution for 1,000dimensional problems. For 100-dimensional problems, 1,000 bits will be sufficient. On the other hand, to get more closer to the optimal solution at the final stage of optimization, it is important to improve a resolution. In this case, we will need ten times more bits for each dimension of problems at least. Notice that statistical aspects are totally disregarded in this analysis: solving a highdimensional problem may require a prohibitive number of black-box evaluations. Then, the required number of bits is strongly depending on target problems. Nevertheless, CONBQA has a potential to solve high-dimensional continuous black-box optimization problems that are difficult to solve by conventional methodologies when more bits will be used in quantum annealer.

V. CONCLUSION
To summarize, we presented that a quantum annealing can be used to solve continuous black-box optimization problems, if a coding-decoding scheme and a machine learning algorithm are properly designed and combined. In this study, for a coding-decoding scheme, random subspace coding was applied. Mathematical foundation about decodability was laid out with intriguing connection to Helly's Theorem. It was shown that CON-BQA is competitive with a state-of-the-art black-box optimization algorithm, encouraging applications to various domains. In addition, random subspace coding will be one of the effective methods to handle continuous problems by Ising machines.
CONBQA is the first of its kind and there remains many possibilities for improvement. First, uncertainty quantification [32] can be combined with CONBQA. In black-box optimization, it is customary to compute uncertainty of prediction and incorporate it in the aquisition function, while CONBQA does not have such a mechanism. A special aspect about CONBQA is that optimization of the acquisition function is done by a quantum annealer, hence the solution is affected by noise. It is interesting to investigate how uncertainty quantification and quantum noise affect overall performance. Second, CONBQA currently make one suggestion at a time to the black-box system, but, in many cases, it is more desirable to make multiple suggestions simultanously. A quantum annealer is basically a sampler and generates many solutions at a time. Currently, only the best solution is used, but it may be possible to use other solutions as multiple suggestions. In our algorithm, not only quantum annealer but also all QUBO solvers including quantum approximate optimization algorithm (QAOA) [33][34][35] using a gatedquantum computers are adopted. Thus, CONBQA may open up a new possibility of these machines to continuous valued-problems.