Siamese Neural Networks have been widely used to perform similarity classification in multi-class settings. Their 4 architecture can be used to group the 5 clinical trials belonging to the same drug6 development pathway along the several 7 clinical trial phases. Here we present an 8 approach for the unmet need of drug9 development pathway reconstruction, 10 based on an Enhanced hybrid Siamese11 Deep Neural Network (EnSidNet). The 12 proposed model demonstrates significant 13 improvement above baselines in a 1-shot 14 evaluation setting and in a classical 15 similarity setting. EnSidNet can be an 16 essential tool in a semi-supervised 17 learning environment: by selecting 18 clinical trials highly likely to belong to the 19 same drug-development pathway it is 20 possible to speed up the labelling process 21 of human experts, allowing the check of a 22 consistent volume of data, further used in 23 the model's training dataset. Siamese Neural Networks (SNN) were developed 26 in the early 1990s (Bromley et al., 1994) to obtain 27 a similarity score from examples of signatures 28 with the goal of identifying forgery. From then 29 many applications used SNN, primarily on image 30 recognition tasks (Chopra et al., 2005). The basic 31 architecture of SNN consists of two identical 32 networks able to learn the hidden representation 33 of the inputs. A similarity function would then 34 compare the inputs hidden representations. The 35 similarity score was taken advantage of in 36 contexts like 1-shot learning in multiclass37 classification problems, where a single example 38 of a class was seen by the algorithm only once 39 before making inference (Koch et al., 2015). 40 Different architectures of SNN were developed in 41 time: Simo-Serra and colleagues developed a 342 inputs SNN (Simo-Serra et al., 2015), where the 43 neural network learned to rank the outputs and 44 identify whether the reference's hidden 45 representation is more similar to a positive or a 46 negative sample. 47 Another example involves the insertion of an 48 intermediate stage between the similarity score 49 layer and the final prediction layer (Subramaniam, 50 Chatterjee, and Mittal, 2016), allowing to increase 51 performance in person re-identification task 52 despite partial occlusion and difference in point of 53 view or illumination. 54 The first applications of SNN were based on 55 Convolutional Neural Networks (CNN) to obtain 56 similarity score on images (Simo-Serra et al., 57 2015), seeing SNN involved in different tasks 58 such as patch identification (Simo-Serra et al., 59 2015), person identification (Ahmed et al., 2015), 60 image matching from different angles (Vo and 61 Hays, 2016). SNN was also explored in Natural 62 Language Processing (NLP) contexts in tasks like 63 identifying sentence similarity (Mueller and 64 Thyagarajan, 2016) and support relation for 65 argumentation (Gema et al., 2017). These 66 applications highlight the flexibility of SNN to 67 identify similarities in different contexts. Here we 68 apply this architecture on an unmet healthcare 69 task: grouping clinical trials belonging to the same 70 drug-development pathway. 71 Before being released on the market a new drug 72 needs to go through several expensive and time73 consuming experiments, involving testing the 74 pharmacological characteristics of the drug in 75 EnSidNet: Enhanced Hybrid Siamese-Deep Network for grouping clinical trials into drug-development pathways Lucia Pagani Analytics Center of Excellence, IQVIA Via Fabio Filzi 29, Milan, Italy lucia.pagani@iqvia.com 255 2 biochemical, cellular, and animal models 76 (preclinical phase) and then on human volunteers 77 (clinical stage). The clinical stage is divided into 78 3 pre-approval phases (safety, efficacy, regulatory 79 proof) and a fourth post-market phase (Corr and 80 Williams, 2009). The experiments performed by 81 research or pharmaceutical companies to study a 82 drug in human subjects are called clinical trials. A 83 drug-development pathway is defined as all the 84 clinical studies performed on a drug for an 85 indication to obtain approval from the regulatory 86 agency. Example of a drug-development pathway 87 is presented in Supplementary Table 1. From 88 starting a phase 1 clinical trial to obtaining 89 approval from a regulatory agency, a drug can be 90 tested for over 10 years, and the process can cost 91 hundreds of millions of dollars, involving 92 thousands of subjects, including patients, doctors, 93 nurses and other personnel, with an approval rate 94 of around 10% (Wong, Siah, and Lo, 2019). 95 Information on most clinical trials is publicly 96 available. Pharmaceutical companies are asked to 97 share their information on ClinicalTrials.gov, a 98 U.S. National Library of Medicine resource. 99 Other companies such as DrugBank (Wishart et 100 al., 2006) or Citeline (Wong, Siah, and Lo, 2019) 101 parse the information from ClinicalTrials.gov and 102 add a hand-curation process in which human 103 labellers cross-reference certain information and 104 add additional labels to the trials, resulting in a 105 similar but more accurate database. 106 Although having information on the clinical 107 trials related to the development of a drug may 108 seem a very straightforward process, there are 109 many confounding factors: 110  Very often several trials of the same phase 111 are run, to obtain statistical power or on 112 slightly different protocols (country, 113 population, sample size, ) 114  The same trial can belong to two different 115 phases (e.g. phase 1-2 or 2-3) 116  The company may not share on public 117 databases the information of the trials it is 118 performing, or may share partial 119 information or not update them 120  Some phases may be skipped 121  Often subsequent trial phases from the 122 same drug-development pathway may 123 address slightly different diseases 124  The disease and the drug can be referred to 125 from different nomenclatures in different 126 trials 127 Grouping of clinical trials to the same drug128 development pathway is a requirement for many 129 different applications, such as analyzing the 130 success of a pharmaceutical company performing 131 trials and marketing new drugs, or calculating the 132 probability of success of a drug for a therapeutic 133 area, evaluating the number of pathway in a 134 therapeutic area, and investigating the futility of a 135 pathway. 136 Although there is a strong need for a large 137 freely-available dataset, only proprietary hand 138 curated datasets exist (Wong, Siah, and Lo, 2019). 139 A relatively small dataset of regulatory agency 140 approved pivotal trials could be parsed from Food 141 and Drug Administration Drug Trials Snapshots 142 (FDA Snapshot) 143 (https://www.fda.gov/drugs/drug-approvals-and144 databases/drug-trials-snapshots). The lack of 145 large publicly available datasets may be one of the 146 reasons why to our knowledge no algorithms to 147 group clinical trials in drug-development 148 pathways have been described in the literature. 149 The contributions of this paper are: (a) a novel 150 approach to group clinical trials in drug151 development pathways; (b) an iterative semi152 supervised learning pipeline to optimize the 153 grouping of clinical trials to the pathway. 154 The model proposed here is based on a SNN 155 architecture. The model learned the similarity of 156 trials belonging to the same pathway. The 157 advantage of using the proposed model in a semi158 supervised learning pipeline would lead to 159 decreased human-labelling effort; the proposed 160 pipeline can work in a de-novo mode (fresh start) 161 and in a primed mode (adding data to previously 162 scored pathways). 163 2 Methods 164 2.1 Data used to train and validate model 165 The ground truth pathways considered in this 166 experiment were pathways extracted by the 167 pivotal trials from the FDA Snapshot and 168 manually identified pathways (hand-curated). For 169 more details on the datasets composition and other 170 methods considered here see Supplementary 171 Methods. 172 256 3 2.2 Neural Network architectures 173 Three architectures were compared in the current 174 research, schematized in Supplementary Figure 1: 175 pure Siamese Neural Network architecture (SNN) 176 where only Siamese branches were present, a 177 hybrid Siamese and Deep Neural Network (SiD 178 NN) consisting of Siamese character-based 179 branches and an additional input branch, and an 180 enhanced version of the SiD NN, having a fully 181 connected layer before the prediction layer 182 (EnSidNet). Supplementary Methods contain the 183 detailed description of the 3 architectures. 184 2.3 Inputs of the model 185 The input features of the networks were: the drugs 186 used in the clinical trial (intervention), the disease 187 considered (condition), the phase of the trial 188 (phase), the countries where the clinical trial was 189 conducted (country), the sponsors of the trial 190 (sponsor), the start and end date of the trial 191 (expressed in days compared to an arbitrary 192 reference date, January 1 st 2000). Details of the 193 preprocessing of the inputs can be found on 194 Supplementary Methods. 195 2.4 Prediction Algorithm 196 Algorithm 1 contains the pipeline to apply the 197 Neural Network to group trials into pathways. 198 The details of the pipeline are reported in 199 Supplementary Methods. For schematic example 200 of the matching pipeline see Supplementary 201 Figure 2. 202 3 Experiments 203 In Supplementary Table 2 we report the number 204 of parameters of the networks and training time. 205 The three neural models have different number of 206 parameters to train, and the complexity of SNN 207 compared to the hybrid models made the training 208 time per epoch longer. In terms of time per epoch 209 the other two hybrid models had comparable time 210 per epoch, despite the slightly higher complexity 211 of EnSidNet compared to SiD NN. 212 3.1 Balanced datasets 213 Accuracy was tested on a balanced validation 214 dataset (see dataset splitting for details on 215 balanced dataset creation). It can be seen from 216 Table 1 that the best performing algorithm was 217 EnSidNet. 218 3.2 32-way 1-shot evaluation performances 219 One-shot evaluation was used to predict whether 220 a new trial belongs to established pathways. 221 The score expected from a random classifier is 222 3.125, due to the unbalanced 1:32 ratio of positive 223 couples versus negative. It can be seen in Table 2 224 that all neural models scored significantly higher 225 than a random classifier in a 32-way 1-shot 226 evaluation assay. 227 Algorithm 1 Input: trials to group in pathways and previously scored pathways Output: pathways containing development trials 1: divide trials in therapeutic areas 2: for every therapeutic area do 3: for every existing pathway do 4: predict similarity between 2 trials of a present pathway and a new trial 5: if probability > 0.8 for both couples do 6: add trial to present pathway 7: sort trials (common lead sponsor or condition) 8: divide trials into batches 9: for every trial in batch do 10: match all versus all and predict similarity 11: if probability > 0.8 do 12: group the trials in a pathway 13: group pathways with common trial 14: select 1 trial per pathway and repeat steps 9-13 15: return pathways Algorithm 1 Balanced dataset Accuracy SNN 0.763393 SiD NN 0.907738 EnSidNet 0.91369 Table 1: Accuracy of the best model on a balanced dataset 32-way 1-shot evaluation assay Neural Network 1-Nearest Neighbor Random Classifier SNN 66.67 81.82 6.06 SiD NN 93.94 69.70 0 EnSidNet 96.97 69.70 3.03 Table 2: Results of 1-shot evaluation assay 257 4 EnSidNet was the model with the highest 228 performance in the test set. On the contrary, the 229 SNN had the lowest performance between the 230 neural models. Surprisingly the input format of 231 SNN tested on the heuristic 1-Nearest Neighbor 232 gave a relatively high performance. 233 To understand the contribution of the different 234 features on the final EnSidNet prediction a SHAP 235 analysis was performed. As Supplementary 236 Figure 3 shows the most important feature to 237 distinguish between couples from the same or 238 different pathway is the number of common 239 sponsors. It is interesting to note that the most 240 contributing features belong to the additional 241 inputs branch of the NN, features that increased 242 the performance of the 32-way 1-shot learning 243 metric of almost 30% (see Table 2). 244 3.3 Metrics on imbalanced dataset 245 Table 3 shows the other metrics considered in this 246 research, calculated on the 1:32 unbalanced 247 dataset. 248 SNN had the worst performance on all metrics. 249 Despite Sid NN had performances comparable to 250 EnSidNet on precision and recall, ROC AUC and 251 PR AUC showed the higher performance of the 252 Enhanced model. 253 Figure 1 shows the probabilities associated to 254 couples belonging or not to the same drug255 development pathway for EnSidNet. The figure 256 shows that the algorithm can distinguish with 257 great certainty whether the trials belong to the 258 same pathway or not, and the higher recall than 259 precision. 260 3.4 Trials grouping in pathways 261 Algorithm 1 for grouping the trials in possible 262 pathways was applied to clinical trials present in 263 the DrugBank database. The clinical trials 264 included were those in phases 1, 2 and 3, with 265 industry lead sponsors and treatment' as the 266 purpose of the trial. Trials to match into drug267 development pathways were 34188. The 268 algorithm took less than 4 hours to run. 269 The therapeutic areas included in these 270 pathways were 27. 271 As presented in Table 4 the statistics of the 272 possible pathways obtained from Algorithm 1 is 273 overlapping with the statistics of the datasets used 274 to train the neural networks (Supplementary Table 275 3). 276 Despite the input of Algorithm 1 was more than 277 34,000 trials, less than 600 were matched in 278 pathways. However, the possible pathways 279 obtained were about 1.5 times the number of total 280 pathways in the dataset, suggesting new possible 281 pathways were discovered running Algorithm 1, 282 highlighting the potential of this semi-supervised 283 approach for the grouping of clinical trials in 284 pathways. 285 Unbalanced dataset F1 P R ROC AUC PR AUC SNN 0.16 0.09 0.76 0.85 0.61 Sid NN 0.90 0.86 0.94 0.97 0.89 EnSidNet 0.90 0.86 0.94 0.99 0.92 Table 3: Metrics of the neural models. The 73 predicted 287 pathways (2-49 trials long), for a total of 264 288 trials, gave rise to 165 different trials (1-11 trials 289 long). The different distribution of the predicted 290 versus confirmed pathways can be seen in 291 Supplementary Table 4. A total of 112 trials (42%) 292 were confirmed being assigned by the algorithm 293 to proper pathways. Only two of the trials selected 294 for human scoring were found also on the ground 295 truth datasets. Specifically, both trials belonged to 296 the FDA snapshot dataset and were single-trial 297 pathways. Interestingly, one of these trials was 298 assigned to 2 other trials, and this 3-trial pathway 299 was then confirmed by the human experts scoring. 300 This is a good example of the capability of 301 EnSidNet and the proposed algorithm to find the 302 contributing trials to a drug-development 303 pathway. 304 4 Conclusion 305 We present a new approach for the grouping of 306 clinical trials into drug-development pathways. To 307 meet this objective, we proposed 3 different 308 neural network architectures. The best performing 309 model was EnSidNet, an enhanced hybrid 310 Siamese-Deep Neural Network. 311 EnSidNet was used to develop a semi312 supervised learning pipeline using 1-shot 313 evaluation and classification to group trials into 314 existing or new pathways. Human scoring would 315 lead to the increase of the training size with ad316 hoc positive and negative samples. 317 Acknowledgements 318 We thank Gregory Lever and Joseph Heenan for 319 useful feedback. 320 References 321 Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard 322 Sckinger, Roopak Shah.
abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method other abstain abstain abstain other abstain abstain abstain other abstain other abstain objective abstain abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain