Clinical trials offer a fundamental opportunity to discover new treatments and advance the medical knowledge. However, the uncertainty of the outcome of a trial can lead to unforeseen costs and setbacks. In this study, we propose a new method to predict the effectiveness of an intervention in a clinical trial. Our method relies on generating an informative summary from multiple documents available in the literature about the intervention under study. Specifically, our method first gath-ers all the abstracts of PubMed articles related to the intervention. Then, an evidence sentence, which conveys information about the effectiveness of the intervention, is extracted automatically from each abstract. Based on the set of evidence sentences extracted from the abstracts, a short summary about the intervention is constructed. Finally, the produced summaries are used to train a BERT-based classifier, in order to infer the effectiveness of an intervention. To evaluate our proposed method, we introduce a new dataset which is a collection of clinical trials together with their associated PubMed articles. Our experiments demonstrate the effectiveness of producing short informative summaries and using them to predict the effectiveness of an intervention. Clinical Trials (CT) present the basic evidence-based clinical research tool for assessing the effectiveness of health interventions. Nevertheless, only a small number of interventions make it successfully through the process of clinical testing. Approximately, 39%-64% of interventions actually advance to the next step of each phase of clinical trials (DiMasi et al., 2010). The uncertainty of a CT outcome could lead to increased costs, prolonged drug development and ineffective treatment for the participants. At the same time, the volume of published scientific literature is rapidly growing and offers the opportunity to explore a valuable knowledge. Therefore, there is a need to develop new tools which can i) integrate such information and ii) enhance the process of intervention approval in CT. Predicting the approval of an intervention, a task that describes the ability of a system to predict whether an intervention will reach the final stage of clinical testing, is a topic that has been studied before (Gayvert et al., 2016; Lo et al., 2018). The majority of these studies use various traditional machine learning methods and rely on structured data from various sources, including biomedical, chemical or drug databases (Munos et al., 2020; Heinemann et al., 2016). However, only a few studies take into account the textual information that is available online, and mostly in a supplementary manner (Follett et al., 2019; Geletta et al., 2019). In fact, employing natural language processing (NLP) techniques to address the outcome prediction task has been hardly explored. Recognising this lack of related studies, the work presented here addresses the task of predicting intervention approval with the use of NLP. Particularly, we relied on generating concise and informative summaries from multiple texts that are relevant to the intervention under evaluation. In a sense, we built an intervention-specific narrative which combines key information from multiple inter-connected documents. The benefit of using multiple articles to generate summaries is that they can cover the inherently multi-faceted nature of an intervention's clinical background. More precisely, given an intervention, our system retrieves all PubMed abstracts that are relevant to the intervention and refer to a clinical study. It then extracts the evidence sentences from each abstract using a BERT-based evidence sentence classifier, in a similar fashion to (DeYoung et al., 2020). This set of evidence sentences, which captures the consolidated narrative about the intervention, can 1947 grow gradually, as new articles become available. Thus, further analysis is necessary in order to select the most important information. Using the set of evidence sentences for each intervention, we generate short summaries by leveraging the power of language models (BERT or BART). The resulted summaries are then fed to a BERT-based binary sequence classifier which makes a prediction about the likely approval or not of the intervention. Overall, the main contributions of the paper are the following:  We propose a new approach for predicting the approval of an intervention which is based on a three-step NLP pipeline. We provide a new dataset for the task of intervention approval prediction that consists of 704 interventions and 15,800 PubMed articles in total. We confirm through experimentation the effectiveness of the proposed approach. Intervention Success Prediction The prediction of intervention approval belongs to a broader category of medical prediction tasks. Relevant work includes clinical trial outcome prediction (Munos et al., 2020; Tong et al., 2019; Hong et al., 2020), drug approval (Gayvert et al., 2016; Lo et al., 2018; Siah et al., 2021; Heinemann et al., 2016), clinical trial termination (Follett et al., 2019; Geletta et al., 2019; Elkin and Zhu, 2021), predicting phase transition (Hegge et al., 2020; Qi and Tang, 2019). All these studies rely either on specific types of structured data or on combining structured data with limited unstructured data. Differently from this line of work, the authors of (Lehman et al., 2019) proposed an approach that employs NLP to infer the relation between an intervention and the outcome of a specific clinical trial. Their method is based on extracting evidence sentences from unstructured text. An extension of this work suggests the use of BERT-based language models for the same task (DeYoung et al., 2020). Another closely related study (Jin et al., 2020), performs a large-scale pre-training on unstructured text data to infer the outcome of a clinical trial. Our approach builds upon this related work, aiming to incorporate information from multiple articles. This extension is motivated by the assumption that the inter-connected clinical knowledge, coming from multiple sources can provide a more holistic picture of the intervention, facilitating more precise analysis and accurate prediction. Although all these prior efforts tackle, more or less, the problem of intervention approval, none of them attempted to predict the effectiveness of an intervention using summarization methods. Summarization The goal of summarization is to produce a concise and informative summary of a given text. There are two main categories of approaches: i) extractive , which tackles summarization by selecting the most salient sentences from the text without changing them, and ii) abstractive , which attempts to generate out-of-text words or phrases instead of extracting existing sentences. Early systems were primarily extractive and relied on sentence scoring, selection and ranking (Allah-yari et al., 2017). However, both extractive and abstractive approaches have advanced significantly due to the novel neural network architectures, such as Transformers (Vaswani et al., 2017). The Transformers architecture is utilized by the BERT (De-vlin et al., 2018) and BART (Lewis et al., 2019) language models which are used by the state-of-the art solutions for multiple NLP tasks, including summarization. Although most of the summarization literature focuses on single-document approaches, there is also a line of work that applies summarization on a set of documents, i.e. multi-document summarization (Ma et al., 2020). Such approaches are of particular relevance to our work, as we aim to summarize a set of sentences about a particular intervention. Summarization in the Medical Domain Summarization has been used to address various problems in the field of medicine. These include electronic health record summarization (Liang et al., 2019), medical report generation (Zhang et al., 2019; Liu et al., 2021), medical facts generation (Wallace et al., 2021; Wadden et al., 2020) and medical question answering (Demner-Fushman and Lin, 2006; Nentidis et al., 2021). Our work is inspired by recent work on multi-document summarization of medical studies (DeY-oung et al., 2021). Apart from introducing a new summarization dataset of medical articles, that work also proposed a method to generate abstractive summaries from multiple documents. Their model is based on the BART language model, appropriately modified to handle multiple texts. Our 1948 model differs in the way it handles the input texts. Instead of concatenating all texts into a single representative document, we order them chronologically and split them into equal-size chunks. Doing so, we expect the clinical studies that were conducted during a similar time period, to reside in the same chunk. According to the U.S. Food and Drug Administration (FDA), a CT addresses one of five phases of clinical assessment: Early Phase 1 (former Phase 0), Phase 1, Phase 2, Phase 3 and Phase 4. Each phase is defined by the study's objective, the interventions under evaluation, the number of participants, and other characteristics 1 . Notably, Phase 4 clinical trials take place after FDA has approved a drug for marketing. Therefore, we can assume that a CT in Phase 4 assesses effective intervention. On this basis, our task is to predict whether an intervention will advance to the final stage of clinical testing (Phase 4), as shown in Figure 1. We model the task of predicting the success or failure of an intervention as a binary classification task. All data relevant to Phase 4 are omitted from the training stage. In this work, we introduce a new dataset 2 for the task of predicting intervention approval. The dataset is a collection of structured and unstructured data in English derived from clinicaltrials.gov and PubMed during May-June 2021. As a first step in the construction of the dataset, we retrieve all available CT studies from clinical-trials.gov that satisfy some criteria. Then, we associate each CT with PubMed articles based on the CT study identifier. Following some cleaning process (i.e. deduplication and entity resolution) we generate the final dataset. at clinicaltrials.gov . We focused on cancer related clinical testing and we retrieved approximately 85,000 studies related to this topic using a list of associated keywords 3 . From this set, we were interested in interventional clinical trials and specifically in two categories that indicate the status of the trial: i) Com-pleted , meaning that the trial has ended normally, and ii) Terminated , meaning that the trial has stopped early and will not start again. The resulting set of studies contains 34,517 completed and 6,872 terminated trials. Interventions Dataset Using the selected CTs, we associated each intervention with its corresponding trials. Therefore, a clinical trial record was formed for each intervention. Then, we selected all interventions that are assessed in at least one Phase 4 CT to form our positive target class (i.e. approval). Likewise, we built our negative target class (i.e. termination) using interventions that led to a trial termination. In total, our dataset contains 404 approved and 300 terminated interventions. For each intervention, we collect all articles from PubMed that are explicitly related to one of the CTs of the intervention. To achieve this, we combine two approaches. First, we search for eligible articles (or links to articles) in the corresponding structured results of clinicaltrials.gov . Secondly, we use the CT unique identifiers to query the PubMed database. Then, the selected PubMed articles are associated with the intervention. This way an intervention is linked with multiple studies that are inter-connected, and thus an intervention-specific narrative is developed. In our dataset, an intervention is associated on average with 22.4 pubmed articles, though for terminated interventions this number is just 1.4. This is because terminated interventions are usually not assessed in many CTs. Overall, our dataset contains 15,800 pubmed articles. The details of the dataset are presented in Table 1. In addition, we attempted to evaluate 4 our approach on a previously used dataset (Gayvert et al., 2016), which consists of 884 (784 approved, 100 terminated) drugs along with a set of 43 features, including molecular properties, target-based properties and drug-likeness scores. 3 The complete list of the keywords used is: cancer, neoplasm, tumor, oncology, malignancy, neoplasia, neoplastic syndrome, neoplastic disease, neoplastic growth and malignant growth 4 The results on this dataset are presented in appendix A 1949 Figure 2: Overview of the proposed approach for classifying an intervention. In Figure 2, we illustrate the proposed approach, which consists of three main steps. Initially, we use the abstracts of the intervention's clinical trial record to extract evidence sentences. These sentences are then used to generate a short summary that contains information about the efficacy of the intervention. The summary is then processed by a BERT-based sequence classifier to make the final decision about the intervention. Each of the three steps is detailed in the following subsections. Identifying evidence bearing sentences in an article for a given intervention is an essential step in our approach. Differently from other sentences in an article, evidence sentences contain information about the effectiveness of the intervention (Figure 3). Therefore, it is crucial that our model has the ability to discriminate between evidence and non-evidence sentences. First, all abstracts related to the given intervention are broken into sentences. The sentences of each abstract are then processed one-by-one by a BERT-based classifier that estimates the probability of each sentences containing evidence about the effectiveness of the intervention. For the classifier, we selected a version of the PubMedBERT (Gu et al., 2020) model, which is pre-trained only on abstracts from PubMed. We tested several models, including BioBERT (Lee et al., 2020), clinicalBERT (Alsentzer et al., 2019) and RoBERTa (Liu et al., 2019), but PubMedBERT performed the best in our task. On top of PubMedBERT, we trained a linear classification layer, followed by a Soft-max, using the dataset from (DeYoung et al., 2020). This dataset is a corpus especially curated for the task of evidence extraction and consists of more than 10,000 annotations. The classifier is trained with annotated evidence sentences (i.e. positive samples) and a random sample of non-evidence sentences (i.e. negative samples). Regarding the ratio of positive to negative samples, cross-validation on the training set showed 1:4 to be a reasonable choice. The evaluation of the different BERT-based models was done based on the same data splits (train, test and validation) as in (DeYoung et al., 2020). sentence is selected from each abstract. Therefore, for each intervention we extract as many sentences as the number of abstracts in its clinical record. explore both extractive and abstractive approaches. Extractive Summaries were based on the evidence sentences extracted in the previous step. Specifically, we re-rank them and choose the top k ( k = 5) to compose our final summary. The model we use here is the same BERT-based model as in Section 5.1. Abstractive Considering that an intervention is linked to multiple abstracts and thus to multiple evidence sentences, we first order all evidence sentences chronologically and combine them into a single text. Then, we split them to equal chunks 5 and each chunk then is fed to a BART-based model to produce the final summary. BART has been shown to lead to state-of-the-art performance on multiple datasets (Fabbri et al., 2021). Specifically, we used the pre-trained distilBART-cnn-12-6 model which is trained on the CNN summarization corpus (Lins et al., 2019). Since abstractive summarization produces out-of-text phrases, it needs to be fine-tuned with domain knowledge. In our case, we fine-tuned the BART model with the MS2 dataset (DeYoung et al., 2021), which contains more than 470K articles and 20K summaries of medical studies. We limited the length of the output summary to 140 words. For the extractive setting, in case the top k sentences exceeded this limit, we removed the extra words. For the abstractive setting we iteratively summarized and concatenated the chunks for each intervention until the expected number of 140 words was accomplished. We model the task of inferring the approval of an intervention as a binary classification task. In our approach, each intervention is represented by a short summary. For the classification of the summaries, we used again a PubMedBERT model. On top of it, we trained a linear classification layer, followed by a sigmoid, using the summaries generated in the previous step: Our positive training instances were the summaries of interventions that 5 A chunk has length equal to the maximum input length of the BART model (1024). have been approved, and correspondingly, the negative ones were the summaries of interventions that have been terminated. Hence, the model decides on the approval of the interventions. All models were pre-trained and fine-tuned for the corresponding task. The maximum sequence size was 512 and 1024 for BERT-based and BART-based models respectively. The Adam optimizer (Kingma and Ba, 2015) was used to minimize the cross-entropy losses with learning rate 2e-5 and epsilon value 1e-8 for all models. We trained all models for 5 epochs, with batch sizes of 32, except the abstractive summarizer for which the batch size was decreased to 4 due to RAM memory limitations of our system. The implementation was done using the HuggingFace library (Wolf et al., 2020) and Pytorch(Paszke et al., 2019). We followed different training approaches for the different trainable components of our pipeline. For the evidence sentence selection and the abstractive summarization models we split the data into development and test and then split the development set further into training (90%) and validation (10%). We kept the model that performed best on the validation set and evaluated it on the held-out test set of each task respectively, averaged over three random data splits. Considering the small size of the interventions dataset, we applied a 10-fold cross validation for the final classification task. For this task, we report macro averages of the evaluation metrics over the ten folds. Our experimentation started with a comparison of different variants and choices that were available for the various modules of our approach. Evidence Classifier Coming early in the pipeline, the performance of the evidence classifier can play a significant role in downstream tasks. The chosen approach relied on domain-specific BERT models. As domain-specific training that can affect the performance of BERT-based models, we conducted a comparison between different variants of BERT. The results in Table 2 demonstrate that the performance of the models is comparable, with all models obtaining scores over 90% in terms of F1 and AUC. PubMedBERT model achieved the best 1951 scores and was used in the rest of the experiments. Summarization Adequacy We assess the performance of the summarization methods on the MS2 dataset which is a collection of summaries extracted from medical studies. The task of the summarizers is to produce texts that approximate the target summaries. We measure the performance of the summarization methods using ROUGE and the results are presented in Table 3. As expected, the abstractive method achieves higher scores, as it has more flexibility in forming summaries. We also observed that domain-specific training improves performance. The abstractive no model is a generic BART model without fine-tuning in the domain. Comparing its performance to the abstractive model, which was fine-tuned on a small sample of the MS2 dataset that was excluded from the evaluation process, we notice a statistically significant improvement. Abstractive methods seem to provide better summaries, however, whether these are more useful than the extractive summaries for our donwstream task is still to be determined. Having made the choices for the individual modules, we now turn to the ultimate task, which is the prediction of the efficiency of the intervention. We evaluate two variations of our proposed method; i) with abstractive summarization denoted as PIAS abs and ii) with extractive summarization denoted as PIAS ext . BS : This is a PubMedBERT model that is trained with a single evidence sentence per intervention (instead of a summary). The sentence is extracted from the most recent PubMed article relevant to the intervention. BN : This is similar to BS but instead of using a single sentence for each intervention it is trained with n evidence sentences extracted from n different articles ( n = 3) . The articles are selected randomly among the ones referring to the intervention. The performance of all models is shown in Table 4. The proposed method outperforms the baselines independent of the summarization methods that is used. Interestingly, even selecting randomly selected evidence sentences seem to help, as BN achieved a higher performance than BS . Still, the use of summarization provides a significant boost over both baseline methods, validating the value of using short summaries to evaluate the efficiency of an intervention. Models that do not take advantage of the inter-connected documents suffer a significant drop in performance. Thus, this result justifies the design of the proposed method. We can also observe that the best performance of the proposed method is achieved when using the extractive summarization method. Extractive summaries have demonstrated low ROUGE scores in Section 6.1. Still, they can properly capture the properties involved in the data for the classification task. On the other hand, although the abstractive summarizer achieved better ROUGE scores, it seems that the generated summaries cannot discriminate the target classes (approved or terminated) as well as the extractive ones. This indicates that the quality of the summary, in terms of the ROUGE score, is not decisive in the classification of the intervention. Analyzing further the performance of our best model, PIAS ext , we report macro average scores for each target class in Table 5. We notice that the model is slightly better at predicting the approval of an intervention rather than its termination. This can be explained by the fact that the approved interventions are associated with a considerably larger number of articles than the terminated ones. This leads to richer summaries for the approved interventions and thus to a more informed decision. Early prediction of approval To build our models, we considered all the available data from Phase 1, Phase 2 and Phase 3. However, predicting the success of an intervention at the earliest phase possible is compelling. Therefore, we examine the ability of our model in making early predictions. More precisely, we evaluate PIAS ext model on the following three transitions: Phase 1 to Approval, Phase 2 to Approval and Phase 3 to Approval. To perform this experiment, we select the interventions that have CTs in various stages and there is least one article for each phase. In total, this subset contains 249 interventions (193 approved and 56 terminated). Then, we use 80% for training and 20% for testing. For each transition, we train our model only with training instances from the corresponding phase. In Table 6, we report the macro average scores over ten random splits of the data. from Phase 2 and Phase 3 to approval can be predicted with considerable success. The large gap in performance between Phase 1 and Phase 2, 3 transitions is explained by the lack of clinical evidence in early phases. Phase to Phase Another interesting and challenging task is to predict the transition of an intervention to the next phase of the clinical trial process. In this experiment, we want to predict Phase 1 to Phase 2 and Phase 2 to Phase 3 transitions. For each transition, we use data only from the former phase for training (e.g. for Phase 2 to Phase 3 transition we use data from Phase 2) for both target classes. Again, we use 80% for training and 20% for testing and present the average scores over ten random splits. Table 7 shows the results for the two transitions, which are comparable to the overall predictive performance of the model. Considering the small size of the datasets used in both phase transition tasks, these results can serve only as an indication of how our model behaves. Further analysis and experiments should be conducted for a more thorough evaluation. It is clinically very valuable to identify the factors that contribute most to a particular decision of the classifier. Interestingly, the summaries generated from our models can also serve that purpose very well. Table 8 illustrates some examples of interventions along with their abstractive and extractive summaries as produced by our pipeline. For the first intervention, pertuzumab , it is notable that both summaries report a improved median progression-free survival which somewhat explains the prediction. For the second intervention, taxane , the summaries mention the greater incidence of serious adverse events and lower median overall survival, which counts against the approval of the intervention. We also notice that many numerical entities are randomly placed or changed in 1953 Intervention PIAS abs PIAS ext pertuzumab (cid:51) the primary endpoint of the study is progressionfree survival. the abstractive summary. This contributes to the tendency of the abstractive methods to generate "hallucinated" evidence, as observed in the literature (Cao et al., 2018). However, the abstractive summaries look more readable. A more exhaustive analysis, including also a human evaluation, is needed to assess the ultimate explainability of these summaries. Predicting intervention approval in clinical trials is a major challenge with significant impact in healthcare. In this paper, we have proposed a new pipeline to address this problem, based on state-of-the-art NLP techniques. The proposed method consists of three steps. First, it identifies evidence sentences from multiple abstracts related to an intervention. Then, these sentences are used to produce short summaries. Finally, a classifier is trained on the generated summaries in order to predict the approval or not of an intervention. with 15,800 abstracts. This data was used to evaluate our pipeline against other baseline models. The experimental results verified the effectiveness of our approach in predicting the approval of an intervention and the contribution of each step of the proposed pipeline to the final result. Further evaluation on predicting phase transitions, showed that our model can assist in all stages of a clinical trial. Besides, the generated multi-document summaries can be naturally used to explain the predictions of the model. There are multiple ways to extend this work. In terms of multi-document summarization, there is room to explore more advanced summarization models, quality and performance metrics as well as better explainability assessment. In the bigger picture, we shall also consider to expand the dataset by extending its size and incorporating different types of resources (e.g. drug interaction networks). Finally, we are interested in enhancing the proposed method to incorporate temporal information associated with the CTs to maintain the history of clinical changes. We would like to thank the anonymous reviewers for their valuable and constructive comments on this research. This works was partially supported by the ERA PerMed project P4-LUCAT (Personal-ized Medicine for Lung Cancer Treatment:Using Big Data-Driven Approaches For Decision Support) ERAPERMED2019-163.
abstain abstain objective method objective abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain objective objective objective other other other other other abstain other objective other other other other other other other other other other objective other other abstain other other objective method method abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain method method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method method abstain method abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method objective result abstain objective abstain objective objective other other