Learning lessons from the COVID-19 pandemic for real-world evidence research in oncology—shared perspectives from international consortia
The coronavirus disease 2019 (COVID-19) pandemic caused a dramatic disruption in clinical practice and undermined the timely delivery of optimal care for patients with cancer worldwide. In order to address many of these challenges and to provide insights into how cancer treatment could be safely delivered to a clinically vulnerable group during the pandemic, several research groups created registries from real-world evidence (RWE). These aimed to understand features and factors associated with COVID-19 overall mortality and severity in patients with cancer, optimal management and the impact of the infection in terms of cancer care delivery. , , , , , , This collaborative process was conducted at high speed, with several adjustments required over time as new data, questions and challenges emerged, such as disruptions in the health care systems and patient care globally, new virus strains/variants, COVID-19 treatments and vaccination, and the demonstration of long-term effects of the infection. During this dynamic period, many important lessons were learned in the design and conduct of RWE studies, spanning different countries, which could be used for future RWE research in oncology and pandemic-related research. Herein, a group of global experts from several registries , who collaborated and studied the effects of COVID-19 in patients with cancer, share their experiences and provide perspectives on how these insights could inform and collaborate on future studies. 

Most health care systems as well as registry/database infrastructures were not yet prepared to study the rapidly evolving landscape of the pandemic, and either significant adjustments were needed or new databases had to be created. Voluntary ‘crowdsourcing’ was broadly used to rapidly capture the continuously changing experience of COVID-19 and its management. Some groups such as the COVID-19 Risk in ONcology Evaluation Tool (CORONET) opted to start with very basic data capture tools (DCTs) and as studies increased in ambition, more sophisticated DCTs were developed to ensure data homogeneity and high quality. Other groups such as the European Society of Medical Oncology (ESMO) COVID-19 and Cancer Registry (ESMO-COCARE), COVID-19 and Cancer Consortium (CCC19), Belgian Society of Medical Oncology (BSMO) and Sociedade Portuguesa de Oncologia (SPO) decided to start with more sophisticated DTCs and larger number of variables collected from the start of the pandemic. One of the most commonly used tools was the Research Electronic Data Capture (REDCap) system, which enabled rapid prototyping and rollout of sophisticated, secure survey instruments, often across different countries. This also allowed future data sharing to increase the number of patients into a larger database. Having a large electronic case report form (eCRF) to capture different variables from a new disease entity, with adjustments for relevant clinical and biological parameters over pandemic phases and the addition of collaborative centers, created several challenges for data completeness, optimal quality and interoperability. Moreover, in order to add flexibility and recognize the variability of RWE, several groups added numerous free text data variables, such as for medication lists, strain/variant of the virus and/or disease symptoms. This increased the amount and potential utility of data; however, this brought its own challenges and tradeoffs between precision and recall. Databases that favored selected options over free text were more effective in the delivery of studies at scale. Furthermore, the DCTs had to be flexible in order to add additional fields as the pandemic shifted in terms of new variants, treatments, vaccines and other relevant variables. Thus, professional DCTs became critical to optimize data collection and would be recommended for use in future RWE research. A frequent obstacle observed during the pandemic was the use of different definitions, grading and timing for the capture of several clinical, pathologic and biologic variables and clinical endpoints/outcomes across different institutions. This was particularly true as new collaborations developed, which required standardization of data fields in order to ensure consistent collection and proper interpretation of data. For instance, the ESMO-CoCARE, CCC19 and the Portuguese ONCOVID initiatives cooperated from the outset to have nearly overlapping data dictionaries, anticipating collaborative analysis. Later when ESMO-CoCARE and BSMO started to collaborate, differences in the data dictionaries required additional manual data curation to merge the datasets. A homogenous metadata for at least core variables under collection or automatic conversion, when different units are used, could mitigate this challenge. In addition, early agreement and definition by key stakeholders of optimal terms and outcome measures can facilitate the acquisition of knowledge over time. One example of this challenge is the use of composite and/or ordinal outcomes, both of which can easily become particular to a singular effort, hampering inter-exchange of ideas and actual data. Another challenge in an RWE study is managing missing, unknown or unreliable data and this was particularly true for the COVID-19 studies. For instance, CCC-19 chose to implement a universal ‘unknown’ option for all variables, such that missingness at the survey level could be distinguished from underlying missingness in electronic health records. To mitigate the incomplete data issue, regular eCRF revisions and the implementation of centrally determined ‘quality scores’, taking into account missingness of critical data elements, such as survival or key prognostic factors were utilized by different groups. Thus, in developing eCRFs, investigators should contemplate not only the variables and outcomes that could optimally address the main research questions under study, but also the feasibility to harmonize the process of data collection and possible adjustments needed over time. It is also important to invest time interacting with different collaborating centers in order to identify potential pitfalls to data collection before commencing, and estimate the workload (e.g. average time per patient data collection). For example, the CCC19 survey took on average 30 min per case, while for ESMO-CoCARE it was around 1 h. Moreover, careful review of data variables for selection of the clinical relevance can help reduce such burden via exclusion of unnecessary information that may not contribute meaningfully to the data analysis. Finally, artificial intelligence or machine learning methods, commonly used in different domains, were also applied during the pandemic to improve data analysis and the development of some prognostic or predictive scores. , , These methods also have huge potential to aid data collection, although in reality natural language processing remains challenging for practical implementation. In the context of clinical prediction efforts, particular emphasis should be placed on the reliability, robustness and fairness of the proposed models as well as its transparency, utility and acceptability for health care professionals. , Various ongoing initiatives may contribute to improving the standards of RWE infrastructures, data sharing and research quality globally, such as the European Health Data Space (EHDS) or the Data Analysis and Real World Interrogation Network (DARWIN EU). 

Frequently, oncology RWE studies are generated from small cohorts, often from single-center experiences and there is commonly parallel research being carried out at different institutions addressing similar questions. This results in a lower level of evidence and various biases, and increases research inefficiencies. During the pandemic many centers were motivated to jointly study the effects of COVID-19. Intriguingly, the collaborative process was not only between major institutions, but also with regional and smaller health institutions, even if they were able to provide only a few clinical cases. Thus, the pandemic highlighted the potential of collaborative efforts to generate larger, broader and therefore more powerful datasets. Furthermore, it revealed that when supported, smaller centers are enabled to contribute to a common research goal. This hub-and-spoke type model where larger centers provide an easy-to-use platform to enter data, the administration, logistical and statistical support, thereby facilitating smaller centers to contribute, could be more used in RWE oncology projects. In addition, bringing together a multi- and inter-disciplinary team of oncologists, virologists, immunologists, public health specialists/epidemiologists, biostatisticians, informaticians and data analysts, among other experts, is essential to improve interpretation of COVID-19-related data. This could be considered for this and other types of RWE research in oncology, with the required set of skills clearly identified to optimally address the aims in each case. 

Before the pandemic, bureaucracy was already a common obstacle affecting the ability to conduct research in a timely manner. , During the pandemic there was variability in the approach of different regulators toward these processes. For example, in the UK the research ethics committee removed the requirement to apply for specific approval to use patient data without written consent. However, in continental Europe most of the institutions kept their procedures and timings to approve new studies, taking weeks or even months until data could be shared or published. In addition, many institutions required data-sharing agreements to be in place, resulting in the inevitable major delays from legal teams. Thus, it was complex or even impossible for many institutions to share data quickly, compromising the ability to carry out research at a fast pace. It is therefore critical that regulators, ethics committees and institutions review their processes, taking into account the views of all stakeholders, to achieve a better balance between ensuring optimal ethical conduct and data protection while reducing barriers to the conduct of research in a timely manner. The pandemic also highlighted the value of central registries, which significantly reduced the barriers to data transfer. For instance, the ESMO-CoCARE and CCC19 initiatives had a unique eCRF and a centralized data management from one institution, which was essential to increase efficiency in all processes of data collection. For institutions that wanted to store additional local data, partial federation was also enabled and was for the most part, successful. Going forward, centralization is probably one of the most practical approaches to reducing bureaucracy. Recent progress in fully federated approaches such as ‘Swarm Learning’ in the biomedical domain is intriguing, although the practicality of such approaches remains unproven; sites must still agree to adopt a common data model in any case. The creation of trusted research environments with emphasis on data security is another approach. In addition, it is also important to have proper, dedicated administrative, regulatory and legal teams to efficiently handle the bureaucratic issues that may arise, especially in cross-national collaborations, enabling researchers to focus on addressing the research questions. These COVID-19 collaborative efforts relied on health care professionals providing their time voluntarily, in the majority of cases unsupported by research funding. This was at a time when there were also huge demands on them to deliver care for patients and thus their dedication to research during this period should be applauded. However, this is not a sustainable model for the long term and many of the consortia reported waning of effort over time as COVID-19 data collection became de-prioritized and the ‘routine’ practice resumed. Thus, proper infrastructure supported by dedicated research funds and protected time for health care professionals within institutions are required in order to more effectively deliver RWE. This could range from using electronic health record (EHR) forms that are completed at critical steps along a patient’s journey, to valid wearable devices which constantly collect patient data or patient apps which are completed in real time. , Artificial intelligence techniques, such as natural language processing, may support a more flexible data interpretation paradigm, where events of interest can be extracted from free text into a target schema. In this new age of data-driven decision making, establishing this infrastructure will be critical for automatic, comprehensive and high-quality data collection. 

The COVID-19 pandemic has been one of the fastest changing areas of clinical knowledge that we have ever seen. From an unknown illness in December 2019 to an extremely well characterized disease in March 2023 with many approved treatments and vaccines, it has shown how research can be rapidly carried out and translated into clinical practice. , Various clinical trials were launched to assess treatments and vaccines for COVID-19 largely in the general population; therefore, such evidence was complemented with prospective and retrospective RWE studies in specific oncology populations, providing additional evidence for clinical and public health decisions. For instance, before 2020 the estimated time between the start of a vaccine development and approval was around 10-15 years, and that period was shortened to less than a year for COVID-19. This resulted in challenges for journals to rapidly disseminate findings in order for the research community to quickly learn from each other. Often COVID-19 articles were fast-tracked to enable rapid publication, and were made open-access at reduced or no cost to share the results with the widest possible audience. However, one of the major drivers of timely research dissemination was the utilization of pre-print servers, which had not otherwise gained much traction in the clinical domain before 2020. These enabled researchers to share their results for comment while in parallel going through the peer review process. However, it also created challenges in that many poor-quality studies with often contradictory results were also widely available. , It is therefore even more important that health care professionals are educated in the critical evaluation of studies in order to ensure they base their decisions on high-quality evidence. The implementation of reasonable scientific rigor and principles is of utmost importance not only during standard care, but also during pandemics and other crises. Guidelines are an opportunity to distil the huge amounts of available information and highlight high-quality findings with the input from experts to inform the best standards in patient management. However, as COVID-19 revealed, they can quickly become outdated. A possible solution is to have regular mechanisms to capture the highest level of evidence for specific clinical decisions, such as the ‘living guidelines’ considered by the World Health Organization for COVID-19 clinical decisions. , ESMO is also embracing this approach and is currently developing online, interactive and regularly updated living guidelines for different tumor settings, , and more recently the American Society of Clinical Oncology (ASCO) also moved into this innovative direction. The COVID-19 pandemic has been a catalyst for development of many research areas, but has also highlighted how barriers to RWE need to be overcome. It is now critical that we use this experience to improve the design and conduct of oncology studies in order to rapidly provide high-quality evidence to inform the best practice in oncology . It is our hope that many of the lessons that we have learned during the COVID-19 years can be utilized to build rigorous and informative RWE registries at speed, and at a global level. 

