Evaluation of artificial intelligence systems for assisting neurologists with fast and accurate annotations of scalp electroencephalography data

Background Assistive automatic seizure detection can empower human annotators to shorten patient monitoring data review times. We present a proof-of-concept for a seizure detection system that is sensitive, automated, patient-specific, and tunable to maximise sensitivity while minimizing human annotation times. The system uses custom data preparation methods, deep learning analytics and electroencephalography (EEG) data. Methods Scalp EEG data of 365 patients containing 171,745 s ictal and 2,185,864 s interictal samples obtained from clinical monitoring systems were analysed as part of a crowdsourced artificial intelligence (AI) challenge. Participants were tasked to develop an ictal/interictal classifier with high sensitivity and low false alarm rates. We built a challenge platform that prevented participants from downloading or directly accessing the data while allowing crowdsourced model development. Findings The automatic detection system achieved tunable sensitivities between 75.00% and 91.60% allowing a reduction in the amount of raw EEG data to be reviewed by a human annotator by factors between 142x, and 22x respectively. The algorithm enables instantaneous reviewer-managed optimization of the balance between sensitivity and the amount of raw EEG data to be reviewed. Interpretation This study demonstrates the utility of deep learning for patient-specific seizure detection in EEG data. Furthermore, deep learning in combination with a human reviewer can provide the basis for an assistive data labelling system lowering the time of manual review while maintaining human expert annotation performance. Funding IBM employed all IBM Research authors. Temple University employed all Temple University authors. The Icahn School of Medicine at Mount Sinai employed Eren Ahsen. The corresponding authors Stefan Harrer and Gustavo Stolovitzky declare that they had full access to all the data in the study and that they had final responsibility for the decision to submit for publication.

Introduction
This decade has seen an ever-growing number of scientific fields benefitting from the advances in machine learning technology and tooling. More recently, this trend reached the medical domain [1], with applications ranging from cancer diagnosis [2,3], prediction of acute kidney injury [4], detection of diabetic retinopathy [5], mining of electronic health records [6] to brain-machine-interfaces [7,8]. While Kaggle has pioneered the crowdsourcing of machine learning challenges to incentivize data scientists from around the world to advance algorithm and model design, the increasing complexity of healthcare domain problems demands interdisciplinary teams with expertise in data science, the problem domain, and competent software engineers with access to large compute resources. Teams or people who meet these criteria are few and far between, leading to a small pool of possible participants and a loss of experts dedicating their time to solving important problems. Participation is even further restricted in the context of any challenge run on confidential use cases or with sensitive data.
In order to protect such sensitive and proprietary data, while at the same time enabling a crowdsourced challenge, we have recently introduced a challenge ecosystem that utilizes the so-called modelto-data paradigm pioneered by the DREAM (Dialogue for Reverse Engineering Assessments and Methods) Challenges [9,10]. This approach allows the solver community to submit their models to the platform which will then autonomously organize model training and testing in a secure cloud environment and provide feedback on model performance to participants. Solvers can then use the model performance to improve their algorithms. In this scheme, the participants cannot download or directly access the challenge data at any point but have the full suite of crowdsourced challenge tools at their disposal. This challenge concept opens the door to running crowdsourced challenges and to enabling broad public benchmarking against proprietary or sensitive datasets which cannot be made publicly available [11]. Using this idea, we recently designed and ran the Deep Learning Epilepsy Detection Challenge to crowdsource the development of an automated labelling system for brain recordings, aiming to advance epilepsy research.
Epilepsy is a neurological disease that affects over 1% of the world population [12]. Patients suffer from sudden and unexpected seizures which impact their physical health and mental wellbeing [13]. Being a highly individualized condition, its expression changes from patient to patient. Even a specific patient's pathology can vary over time. This makes adequate diagnosis, treatment, and disease management extremely challenging: one third of all epilepsy patients suffer from refractory epilepsy. Two-thirds of patients respond to medication in some way at some point in their journey, but oftentimes the little understood evolving nature of the disease leads to fading or transient therapeutic control [12].
The most common method of tackling this challenge is to monitor patients continuously and log disease episodes of relevance in disease diaries [14]. These longitudinal data repositories can then be used to investigate and adjust the effect of medication in quasi-real-time, and to study the correlation between treatment regimens and disease progression. While this data-driven approach to treatment management and in-situ care optimization is seen as key to fundamentally changing the success of treatment and efficiency of clinical trials [15] until recently, real-world implementations of disease diaries have been entirely manual and thus highly inefficient. Manually created disease diaries are only approximately 50% accurate [16]. This is not rooted in sloppy reporting techniques. It is the individualized and incapacitating nature of the disease itself that leaves patients unable to recognize, remember, or keep track of their own seizures. That makes it impossible for untrained observers to recognize and describe seizure episodes in clinically actionable ways [15]. In order to overcome this challenge, and to leverage a plethora of wearable and mobile sensing platforms, the field has turned to exploring the use of machine learning techniques for the development of automatic patient monitoring systems [17].
Amongst a broad spectrum of sensor modalities ranging from video cameras to smart watches [18], the electroencephalogram (EEG), which uses scalp electrodes, is considered to be the gold standard for seizure monitoring in clinical as well as non-clinical environments [13]. However, while EEG monitoring systems have evolved from relying on intracranial implanted electrodes to use of non-invasive clinical and non-clinical wearable devices, automatic annotation of EEG data remains a challenging machine learning problem. Primary reasons for this include low signal to noise ratio, movement artefacts, poor electrical conduction and nonlinearly distorted

Research in context
Evidence before this study Epilepsy is a highly individualized neurological condition with disease expressions changing over time. It thus cannot be diagnosed, treated and managed in a uniform and equally efficient way across patients. The ability to monitor patients individually and continuously and to log seizure episodes in disease diaries is key to gaining a patient-specific understanding of the disease. This can empower doctors to optimize and adjust medication response and pharmaceutical companies to design more efficient clinical trials. Until recently such disease diaries were created manually making data review a highly cost-and time-intensive process and rendering records inaccurate when populated by patients through self-reporting outside the clinic.

Added values of this study
New deep learning-based techniques allow automatic detection of seizures in brain activity data monitored by electroencephalography (EEG) sensors. Harnessing the power of crowdsourced artificial intelligence algorithm development and one of the largest EEG datasets in existence we have demonstrated an automatic seizure detection model which can assist human reviewers in substantially reducing the amount of raw EEG data to be annotated manually.

Implications of all the available evidence
Our system uses deep learning technology and custom data preparation techniques to automatically learn patient-specific seizure signatures. It then filters seizure segments out of raw EEG data for verification by an expert neurologist. Our system can be tuned by the human reviewer to achieve detection sensitivities in excess of 90% or raw data reduction factors of up to 142x. crosstalk between spatially adjacent sensors. Disease-specific intricacies such as the highly individualized profiles of seizure patterns make generalizability of detection models across patients challenging. As a result, in today's practice, EEGs are still interpreted manually, or 'read' by trained neurologists. The associated time and cost burdens are substantial and account for approximately 5% of the total hospital charges for epilepsy patients admitted to Intensive Care Units (ICUs) in the US [13]. Furthermore, doctors responsible for this highly repetitive and time-consuming process find themselves caught between the equally undesirable options of either having to limit the time they can devote to attend to their patients [19], reducing the duration of monitoring sessions or reducing the amount of data to be manually reviewed [13].
A variety of machine learning (ML)-based automatic EEG annotation systems have been proposed [13,20] to reduce this burden. Of special interest are deep-learning models as they can learn to automatically recognise different seizure patterns for individual patients which allows to calibrate these detection algorithms to patient-specific disease expressions. Some have been deployed and tested in clinical scenarios [21,22]. Clinical acceptance of this technology has been slow [23]. The lack of commonly adopted performance metrics to evaluate performance and compare to human expert reviewers [24] has inhibited broad adoption of these systems in critical care settings. Generalisability of performance across datasets collected at different institutions has been problematic also [13].
Using one of the world's largest EEG datasets, the TUH Seizure Corpus [25,26], the Deep Learning Epilepsy Detection Challenge tasked participants to develop deep learning models for automatic annotation of epileptic seizure signals in raw EEG data with maximum sensitivity and minimum false alarm rates. Using the Time-Aligned Event Scoring (TAES) metric, an evaluation framework custom-designed to score high-resolution automatic EEG annotation algorithms [24], we assessed the potential of these annotation models for use by clinical neurologists as assistive labelling systems for raw EEG monitoring data.
In the following sections we describe the architecture and functionality of our custom-developed crowdsourcing challenge platform, with a special focus on its model-to-data feature, the design and execution of the Deep Learning Epilepsy Detection Challenge, as well as the scientific outcomes and validation results of the best performing participant models.

Methods
With a goal to run a challenge that mobilizes the largest possible pool of participants globally across IBM, we designed a crowdsourced challenge called the Deep Learning Epilepsy Detection Challenge. Participants were asked to develop an automatic labelling system to reduce the time a clinician would need to diagnose patients with epilepsy. Labelled data for the challenge were provided by Temple University Hospital (TUH) [22,26]. We partitioned this data to create training, validation and blind test sets which participants could work with only through our platform.
To provide an experience with a low barrier of entry, and to demonstrate that following the model-to-data paradigm a crowdsourced challenge can run efficiently without participants ever having to directly access or download the challenge data, we designed a generalizable challenge platform based on the following principles: (1) eliminate the need of in-depth knowledge of the specific domain. (i.e. no participant should need to be a neuroscientist or epileptologist); (2) eliminate the need of more than basic programming knowledge (i.e. no participant should need to learn how to process fringe data formats and stream data efficiently), (3) eliminate the need for participants to provide their own computing resources, and (4) eliminate the need for participants to download or directly access the challenge data in any way.
The platform guided participants through the entire process from sign-up to model submission, facilitated collaboration, and provided instant feedback to the participants through data visualization and intermediate online leaderboards. The competitive phase of the Deep Learning Epilepsy Detection Challenge ran for 6 months. Twenty-five teams, with a total number of 87 data scientists and software engineers from 14 global IBM locations participated. Seven teams submitted final solutions five of which were valid final submissions as per the challenge rules. The architecture of the platform that was designed and developed as well as data and model flow through it during the challenge are shown in Fig. 1. The entire system consists of a number of interacting components: (1) A web portal serves as the entry point to challenge participation, providing challenge information, such as timelines and challenge rules, scientific background information and a description of the data used for this challenge. The portal also facilitated the formation of teams and provided participants with an intermediate leaderboard of submitted results and a final leaderboard at the end of the challenge. A screenshot of the starting page of the web portal can be found in the supplemental information (Fig. S13). (2) IBM Watson Studio [27] is the umbrella term for a number of services offered by IBM and accessible to participants. Upon creation of a user account through the web portal, an IBM Watson Studio account was automatically created for each participant that gave users access to the (3) IBM Data Science Experience (DSX) platform which hosted a user interface and starter kit and formed the main component for designing and testing models during the challenge. DSX allows for real-time collaboration on shared notebooks between team members. A starter kit in the form of Jupyter notebooks [28], supporting the popular deep learning libraries TensorFLow [29] and PyTorch [30], was provided to all teams to guide them through the challenge process. Upon instantiation, the starter kit loaded the necessary python libraries and custom functions for the invisible integration with (4) IBM Cloud Object Storage (COS) [31] and the analytics engine (5) Watson Machine Learning (WML). In dedicated notebook cells, participants could develop custom pre-processing code (including custom montages), machine learning models, and post-processing algorithms. The starter kit provided instant feedback about participants' custom routines through data visualizations. Using the notebook only, teams were able to run their code on WML, making use of a compute cluster of IBM's resources. The starter kit also enabled submission of the final code to a data storage to which only the challenge team had access. WML provided access to shared compute resources (Graphics Processing Units, GPUs). Code was bundled automatically into the starter kit and deployed on WML. WML in turn had access to shared storage from which it requested recorded data and to which it stored the participant's code and trained models. The data for this challenge resided in COS. Note that using the starter kit, participants submitted their model code to the platform which autonomously organized model training and validation on the raw data and provided back model performance results to participants. The participants could then investigate this feedback in order to better design custom algorithms. This approach is called a model-todata paradigm which unlike in Kaggle-style challenge scenarios keeps data shielded from the solver community while at the same time allowing a crowdsourced approach to model development. (6) Utility Functions were loaded into the starter kit at instantiation. This set of functions included code to pre-and post-process data into a more common format while preserving all seizure related information, to optimize streaming through the use of the NutsFlow and NutsML libraries [32], and to provide seamless access to all services used. (7) Final code scoring after completion of the challenge was conducted in an automated way as soon as code was submitted through the starter kit.

Data sources and preparation
All data used in this study is available as open source data at the web site: https://www.isip.piconepress.com/projects/tuh_eeg/html/ downloads.shtml. This data was collected at Temple University Hospital, a research and teaching hospital. The study has been conducted under the approval of the Temple University Institutional Review Board (IRB) under IRB No. 20,774, which supports the release of data once it has been properly anonymized. The IRB has been in existence since 2012 and has been renewed on an annual basis (currently through 2021). Patients consent to the use of their data for research and teaching through a written consent as part of their admission record at Temple Hospital. The data has been carefully anonymized before being released from Temple Hospital so that a patient's identity cannot be reconstructed from the data. Files mapping the anonymized data to identifiable data are maintained with Temple Hospital's Health Insurance Portability and Accountability Act (HIPAA)-protected network and never leave the hospital. The study has been conducted in compliance with this ethical approval.
The TUH EEG Seizure Corpus v1.2.0 [22] which contains scalp EEG records of 315 patients with annotated seizure times was split into training and validation datasets for the challenge ( Table 1). The dataset is composed of 822 monitoring sessions with 280 sessions containing a total of 2012 seizures. Annotation protocols including explanations of data collection and split processes are provided on the TUH Open Source EEG Resources platform [25]. The validation dataset was used to determine team rankings on the intermediate leaderboard during the competitive phase (Fig. S1, supplemental information). Another dataset containing annotated data from 50 patients following the same format as v1.2.0 was used as a blind held-out test dataset (Table 1) for final team rankings on the final leaderboard at the end of the challenge (Fig. S1, supplemental information). After completion of the challenge this blind test dataset was merged with v1.2.0 and made publicly available as version v1.2.1 of the TUH seizure corpus thus allowing reproducibility of and continuous benchmarking against the results published in this paper.
The size of training, validation, and blind test sets are shown in Table 1. Training and validation datasets were composed to reflect a balanced demographic profile (49.5% of patients in the training dataset are male, 44% of patients in the validation dataset are male, further demographic distributions for the datasets are provided in [22]). Training and validation sets were used during the competitive phase of the challenge following the model-to-data paradigm described above not allowing participants any direct access to or downloading of any of the data. The blind, held-out test set was not accessible to participants' models at any time during the challenge and was only used once by the challenge organising team to evaluate the submitted models during the scoring phase after the completion of the competitive phase (see Fig. S1 supplemental information for challenge timeline).
The TUH EEG Seizure Corpus consists of EEG sessions recorded according to the 10/20 electrode configuration [33] and utilizing the European Data Format (EDF) [26]. We converted the recorded EEG signal into a set of montages, or differentials, of electrode signals based on guidelines proposed by the American Clinical Neurophysiology Society [34]. In this challenge, we used the transverse central parietal (TCP) montage system for accentuating spike activity which has been shown to improve performance in EEG classification tasks [35].

Evaluation procedure
The evaluation of machine learning algorithms for seizure detection lacks standardization. Typically, two different types of methods are used: epoch-based and term-based. Epoch-based methods compute a summary score decision per unit of time. Term-based methods score on an event basis and do not count individual frames.
Both methods have disadvantages. While epoch-based scoring generally weighs duration of events more heavily, term-based methods are a permissive way of scoring and can result in artificially high sensitivities. In this challenge, we use a method called Time-Aligned Event Scoring (TAES) that utilizes concepts of both epoch-based and term-based methods. It considers percentage overlap between reference and hypothesis and weighs errors accordingly. The TAES metric is described in detail in [24]. Note that since TAES weighs both the number and duration of identified seizures, the sensitivity vs. false positive profile is not the same as for standard methods where sensitivity typically increases with an increasing false positive rate. In TAES the sensitivity is penalized at both low and high false positives. For low false alarms the sensitivity is low since enough seizures are not being discovered by the classifier. At high false alarm rates, since most samples are marked as seizures, although the total duration of identified seizures is high, the number of unique seizures identified is low and thus TAES again penalizes the sensitivity value.
Evaluation Metric: The two qualities of an automatic seizure detection system should be high sensitivity and low false alarm rate. For the purpose of this challenge, we use the following metric to combine these two parameters into an evaluation metric E with E = (FA / S) -ɛ * S where FA is False Alarm per 24 h, S is Sensitivity, and ɛ is a positive constant. The best solution will have the smallest E. Note that E has two contributing terms. The first term FA/S ensures that systems with lower FA and higher S are preferred. The second term ensures that higher S solutions are preferred if for two systems the (FA/S) ratio is same. This formula constitutes the pre-defined objective function for measuring success and remained unchanged during the course of this challenge.
Scoring: During the competitive phase of the challenge scoring happened instantaneously: Once a model had been trained, it was evaluated using a validation data set and the score was submitted, displayed and ranked against other participants' models in the leaderboard section of the challenge portal. During the evaluation phase (i.e. after completion of the competitive phase) we gave participants a 2-week time window to submit their final trained model. We extracted the pre-processing model and post-processing code from each submission and ran these models on a held-out blind test dataset (to which participants had not had access to at any point during the challenge). This was the final submission evaluation similar to the "private leaderboard" in Kaggle. In Kaggle, this "private leaderboard" is also immediate since one submits only the predictions. For our challenge, we ran the participants' final submitted code on the blind test dataset, which took 3 weeks to complete for all final submissions. The reason for deviating from conventional Kaggle-style protocol by submitting only predictions is that unlike Kaggle we keep raw data confidential and do not provide it to participants at any point.

Role of funding source
The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Results
At the completion of the challenge, 7 teams submitted their final algorithms which were evaluated against the blind test set. Upon review of all final submissions, we found that 5 out of the 7 teams had made valid submissions as per the challenge rules. These 5 teams were named Ids_cpmp, Otameshi, AI4MH, Team SG, and EpiInsights. They were considered in the final evaluation stage. Several measures of performance obtained using the validation dataset (leaderboard) and the blind test set are provided in Fig. 2: evaluation metric (E), sensitivity (S), false alarm rate (FA/ 24 h) as well as a sensitivity vs. FA/24 h plot for all 5 submissions. It can be seen that the performance of the 5 submissions is similar in both validation and test sets, indicating that there is no evidence of overfitting. Any automatic seizure detection system, be it a retrospective assistive labelling system or a real-time alert system, needs to be at least as sensitive as a human observer for it to be clinically relevant. This sensitivity goal for an automated system is 75% [24,36]. For false alarm rates equal to or lower than those of human observers the detection system could replace monitoring clinicians. For false alarm rates higher than those of human observers the system is not suitable to replace them, but for low enough false alarm rates such a system can be used as data reduction tool which decreases the amount of raw EEG data a human annotator needs to review. While unassisted human annotators will review the entirety of all raw EEG data, use of an assistive labelling system allows review of only those EEG segments which the system detects: both, correctly in terms of true positives (actual ictal segments) and incorrectly in form of false positives (false alarms, actual non-ictal segments). We call the total amount of raw EEG data composed by all accumulated false positive segments the annotation overhead, and the total duration of raw EEG data defined by all true positives the annotation ground truth. In the following section we show that 4 out of the 5 automatic seizure detection systems developed in this challenge could be used to reduce the annotation overhead by up to several orders of magnitude thus substantially decreasing the labelling time burden for human annotators.
At their lowest false alarm rate levels none of the 5 final submission models reached 75% detection sensitivity thus rendering the developed algorithms unsuitable as real-time alert systems (Fig. 2  (b)). However, as part of their final solution, team Otameshi and Ids_cpmp introduced an engineering post-processing step which added synthetic false alarms (details provided in supplemental information). This step introduced a hyperparameter which allowed for the tuning of sensitivity and FA rate of the developed models. We removed this engineering step for producing the results shown in Fig. 2 to be able to assess detection performance at the lowest achievable false alarm rates for all submissions. We then added the engineering step back into the final submissions of all 5 teams which allowed an increase in sensitivity to above the 75% level threshold for all submissions except for the one from team AI4MH which is therefore excluded from the following analyses (detailed reproducible individual descriptions of all solutions including neural network architectures, hyperparameter selection procedures, training and scoring methods, data pre-and post-processing techniques and visual solution flow charts are provided in section III of the supplemental information). The total false alarm numbers per 24 h obtained by each of these four submissions at 75% sensitivity are shown in Fig. 4a and yield the shortest achievable annotation overheads for each automatic seizure detection system as depicted in Fig. 3.
Teams Otameshi, EpiInsights and Team SG all achieve minimum annotation overheads of 7 min. In good approximation it can be assumed that on average~0.2% or~3 min of a continuous 24h-long raw EEG recording describe ictal segments while 98.8% or 1437 min of the raw data are correlated with non-ictal episodes [22]. For unassisted human labelling of 24 h of raw EEG data this means that the seizure ground truth is 3 min and the annotation overhead is 1437 min. Using the automatic labelling systems reduces the annotation overhead to 7 min thus reducing the amount of total raw EEG data that needs to be reviewed by a human expert from 24 h to 10 min.
Note that we do not claim this time to be the time that it would take a human annotator to label the data. Actual human annotation times are determined by annotation procedures, review protocols as well as the degree of expertise and practice of the reviewers. Regardless of these factors the assistive detection systems described above reduce the overall amount of data that needs to be reviewed by up to two orders of magnitude with a maximum achievable reduction factor of 142x (Fig. 4b) and thus lead to a substantial decrease of the time and cost burden for all human annotation scenarios.
Further investigating the effect of the engineering step introduced by teams Otameshi and Ids_cpmp, we found that as more false alarms are included the sensitivity reaches a maximum and then decreases again. This effect can be attributed to the impact of the TAES evaluation metric which penalizes both low and high false alarms as explained above. Fig. 5 plots the path from 75% sensitivity to maximum achievable sensitivity against false alarm rates for all 4 submissions. With increasing false alarm rates, the respective data reduction factors decrease (Fig. 6). Exploiting this effect allows the development of a tunable assistive labelling system: annotation sensitivities beyond 90% can be achieved but come at the cost of lower data reduction factors, i.e. the price for higher labelling sensitivity is longer data review time. This tunability allows clinical experts to cater the quality of their annotation services to healthcare provider and insurer specific frameworks: depending on the amount of billable time for data review and the amount of data to be reviewed, a custom data reduction factor can be calculated that compresses the total raw data to the exact size that can be reviewed during the billable time while at the same time optimizing annotation sensitivity.
Note that three systems (Otameshi, EpiInsights and Team SG) allow for maximum detection sensitivities of 90.63%, 91.60%, and Fig. 3. In order to label 24 h of EEG recordings an unassisted human annotator has to review all 24 h of raw EEG data (top). Using the systems developed in this challenge, the amount of data needing review is the sum of the seizure ground truth (correctly detected true positive actual seizure segments) plus the annotation overhead (incorrectly detected false positive segments). All 4 automatic systems operate at 75% detection sensitivity. A conservative upper bound approximation for the total seizure ground truth duration in a 24 h raw EEG data recording is~0.2% [22] or~3 min. The best models achieve a minimum annotation overhead of 7 min which therefore allows to reduce the total amount of raw EEG data to be reviewed by a human annotator from 24 h down to 10 min or less. Note that the duration of seizure ground truth may fluctuate across patients, i.e. a patient might experience longer or more frequent seizure episodes on certain days which impacts the total duration of raw EEG data to be reviewed for that day. The annotation overhead however remains unaffected and will stay at the levels shown in the figure for all patients at all times. 91.57%, respectively (Fig. 7a) with data reduction factors of 28, 24 and 22 respectively. This reduces 24 h of raw data to a 51.4min-long raw data segment to be reviewed by the human annotator (Fig. 7b). Note that seizure ground truths will fluctuate across patients and over time which in turn causes fluctuating EEG data reduction factors. Hence, the developed assistive labelling systems will have the strongest annotation time saving impact for situations in which seizures are rare (short seizure ground truth) and normal brain activity is prevalent (large annotation overhead). Table 2 provides a summary of the performance parameters for the final valid submissions of all teams.
Throughout various crowdsourcing challenges, it has been observed that aggregating predictions from multiple algorithms improves over the best individual algorithm [37,38], a technique known as ensemble learning in the ML literature. The success of ensembles depends on various factors including the diversity and performance of individual algorithms [39]. We constructed several ensembles such as majority vote and the recent SUMMA algorithm [39] to evaluate all valid final submissions and compared their performance with the individual submissions. However, none of the ensembles performed better than the best individual submission in the ensemble. We mainly attribute this to the number of algorithms used for ensemble learning (5 algorithms) and the lack of sufficient diversity between these algorithms which is partly due to the fact that all the teams used the same training data.

Discussion
We developed and tested a novel cloud-based platform for running crowdsourced artificial intelligence challenges. The platform uses a model-to-data technique to prevent the solver community from downloading or directly accessing the challenge data while at the same time offering a notebook framework for developing models and a suite of machine learning and data pre-and post-processing tools.
Running the crowdsourced Deep Learning Epilepsy Detection Challenge in collaboration with Temple University, we enlisted a total of 87 scientists and software engineers from 14 research centres around the world to build deep learning models for automatically Fig. 4. An engineering step introducing a hyperparameter which allowed a trade-off between sensitivity and FA rate was included in the submissions of teams Otameshi and Ids_comp. This engineering step was applied to all 5 final submissions 4 of which thereby reached sensitivities of 75% or higher. (a) shows false alarm rates at the 75% detection sensitivity mark for those 4 models. (b) shows the reduction factors of raw EEG data that has to be reviewed by human annotators for each system. Team EpiInsights achieves the highest reduction factor of 142x.   6. Reduction factors of raw EEG data to be reviewed by a human annotator vs. detection sensitivity going from 75% to maximum achievable sensitivity values for each system. The models from teams Otameshi, EpiInsights and Team SG achieve maximum detection sensitivities of 90.63%, 91.60%, and 91.57%, respectively and two-order of magnitude data reduction factors. detecting seizures in the largest existing corpus of electroencephalography (EEG) data. The best performing models demonstrated the feasibility of an assistive EEG annotation tool that could reduce the amount of raw EEG data to be reviewed by human experts by a factor of 142x thus promising to substantially decrease the time and cost burden to keep digital disease diaries.
In this section we discuss the two core aspects of this study: (i) the performance of the model-to-data crowdsourcing AI platform, and (ii) the assistive automatic EEG annotation system which was produced as part of the crowdsourced Deep Learning Epilepsy Detection Challenge.
Investments by enterprises, medical institutions and academic organizations operating in the healthcare and life sciences sector regularly result in the generation of datasets which carry substantial information content and therefore have substantial monetary and strategic value. These datasets are often large, unstructured, and noisy which makes them uniquely primed for analysis through artificial intelligence technology. However, the abundance of data is not matched by an equally strong supply of data science resources capable of developing and applying AI to drive insights from the data. Crowdsourcing the analysis of the data can solve this resourcing problem and at the same time accelerates speed, innovation and broad reproducibility of AI solutions, a benchmarking feature which the medical AI field is in dire need of [11].
As data owners intend to protect the value of their data, they are not willing to share it with open communities of solvers, ruling out the use of conventional 'Kaggle-style' AI crowdsourcing ecosystems which make challenge data directly available to the solver community. In the absence of an alternative collaborative infrastructure, many such datasets remain proprietary and unavailable for crowdsourced analysis and public benchmarking. In an effort to circumvent the need to publicly share their data and still be able to use conventional crowdsourcing platforms, some data owners have resorted to using redacted data for enabling external crowdsourced challenges [40] which generally compromises the quality of the model solutions.
In other scenarios companies may use conventional crowdsourcing platforms internally [41] but in these cases, they exclusively rely on internal data scientist resources which limits size and efficiency of the solver community substantially and inhibits transparency and external verifiability of results.
Our model-to-data crowdsourcing challenge platform overcomes these limitations by allowing participants to publicly build, test, evaluate and validate AI models on proprietary data while at the same time avoiding the need to grant them access to the data itself. The novelty of our platform lies in the fact that all steps and resources required from learning about the scientific use case and challenge design to performing data pre-processing, AI model development, testing, optimisation, and submission are fully integrated in one coherent workflow, eliminating all infrastructural and procedural overhead that is not related to developing AI models. The most important platform capability is the IBM Watson Studio ecosystem which automatically provisions all compute resources through the IBM Watson Machine Learning service and all data management resources through the IBM Cloud Object Storage service. Watson Studio also leverages the Jupyter notebook framework which provides a ready-made AI coding infrastructure for data scientists. This layer of automated operational management which allows challenge participants to exclusively focus on model development and relieves them of any other operational tasks is a key advantage and novelty of our platform over conventional Kaggle-style platforms.
The platform enables collaboration between data scientists whilst keeping proprietary or sensitive data secure and protected. Our platform accomplishes this by employing a model-to-data approach in which the challenge datasets are never directly accessed by the participants who instead create models compliant with the formatting of the data based on a small sample data provided by the data Fig. 7. (a) Applying the engineering step introduced by teams Otameshi and Ids_cpmp raises the maximum detection sensitivities to 90.63%, 91.60% and 91.57%, respectively. This comes at the cost of increased false alarm rates and decreased data reduction factors which are shown in (b). Note that even at maximum sensitivity level the lowest data reduction factor (22, Team SG) still allows to compress 24 h of raw EEG data down to a~1h-short segment of raw EEG data to be reviewed by a human annotator. Table 2 Overview of performance parameters achieved by the final models against the blind held-out test dataset after applying the engineering step introduced by team Otameshi. The far-right column lists the minimum achievable net amount of false positive data segments (annotation overhead) which each model produces at 75% detection sensitivity and which need to be reviewed by human experts together with the correctly detected true positives (seizure ground truth) for AI-assisted manual EEG labelling. owners. They then submit their sample models to a repository, residing within a secure cloud environment which is inaccessible to participants. There, and shielded from participants, the submitted models are trained and evaluated on the hidden data. Model performance is determined based on a pre-defined evaluation metric and the results are handed back to the respective participants. Following this scheme, the model-to-data challenge platform keeps the data shielded behind a firewall at all times while facilitating model ingestion into the model evaluator and extraction of model performances out of it. We have demonstrated and tested the first working instance of our model-to-data platform with the Deep Learning Epilepsy Detection Challenge. Further work will focus on platform upgrades through additional features for increased data safeguarding and HIPAA compliance. We plan to opensource the platform and run regular crowdsourced deep learning challenges. The annotation models developed as part of the Deep Learning Epilepsy Detection Challenge by teams Otameshi, EpiInsights, Ids_cpmp and Team SG are capable of automatically filtering ictal segments out of raw EEG data with sensitivities that are comparable to human experts. Reaching this sensitivity regimen comes at the cost of a higher false alarm rate which, since it is substantially higher than the number of true positive samples, requires human experts to manually review all samples which the models detects for final annotation. Using these AI models as assistive filtering tools allows human data reviewers to cut down the amount of raw data that needs to be reviewed by up to two orders of magnitude. Only the collaborative combination of an automatic AI model and a human expert decision maker allows improvement of the efficiency of the EEG review and labelling process. This is a common example of how AI technology enters the realm of real-world applications: AI does not replace the human expert but rather serves as an assistive tool that enables faster and more efficient decision making.
Note that neither one of the four top performing models nor ensembling versions of the models outperform all others. For example, the model of team EpiInsights yields the highest overall achievable sensitivity of 91.60% and largest data reduction factor of 142x at 75% sensitivity but it is the model of team Ids_cpmp that produces the highest data reduction factor of 44x at maximum sensitivity. The tunability of the system is key to its deployment configuration: the choice of analytical models depends on the target sensitivity level of the overall review and the amount of time which the human reviewer is willing to invest in the final review step.
It is also important to note that we do not derive a quantitative statement on how much time exactly human reviewers will save using the developed automatic detection models. Data review processes, protocols and routines differ across institutions as do the experience and labelling performance levels of human reviewers. Furthermore, seizure frequencies per 24 h vary across patients and over the course of monitoring time windows, and the more ictal samples a 24 h raw data segment contains, the less room for raw data compression there is. These factors all affect the impact of using the automatic filtering system on the net time savings of human reviewers. Therefore, for this study we chose the net amount of raw EEG data that has to be reviewed by human annotators as a common parameter to assess the workload reduction which our system offers. Future work will test the applicability and benchmark the performance and generalisability of our automatic detection system across a variety of real-world clinical settings.
Besides integrating our models into clinical processes as assistive annotation tools of historic data, future work will also focus on further reducing false positive rates while maintaining the sensitivity levels reported in this study. If false positive rates can be reduced to human levels of 1FA/24 h [36] then the model could be used as a real-time seizure alert system.
There exists a plethora of metric frameworks for assessing the seizure detection performance of machine learning models which, although they often use similar terminology, do not allow direct performance comparison of the respective algorithms. An analysis of all popular performance metrics is beyond the scope of this paper and has been done elsewhere [24] (pre-print). However, we provide a simple example to illustrate this point: in a first scenario a deep learning model is used to detect the occurrence of a seizure event which is defined by seizure onset and end times. In a second scenario a deep learning model is used to detect seizure durations in the very same dataset. Both scenarios will describe the employed algorithms as seizure detection models and might even use the same statistical parameters to report on their performance. However, in the first scenario the algorithm will only have to detect one single ictal data sample within a seizure segment to claim success. In the second scenario the success of the algorithm will depend on how many ictal samples it detects correctly within a seizure segment. Awareness of this context and the underlying use case is crucial for being able to meaningfully compare the performance of machine learning models and to choose an appropriate validation metric for an experiment in the first place.
In this study we applied the Time-Aligned Event Scoring (TAES) metric which has been custom developed to assess the performance of detection algorithms in scenarios where both, detecting the number of events and their duration are equally important. Therefore, and in order to allow meaningful benchmarking, we stayed within the TAES framework whenever comparing the performance of models described in this study against state-of-the-art technology.
Several Kaggle or Kaggle-style AI challenges on detecting [42] and forecasting [43] epileptic seizures using EEG data have been held in the past. While these challenges also followed the crowdsourcing approach, they differ substantially from the challenge we report on in this paper with respect to management and type of challenge data as well as the obtained performances of winning models. To facilitate AI model development and data processing experiments the challenge organizers made all challenge data directly available to participants for both challenges. For the Kaggle challenge, combined intracranial EEG datasets from humans and dogs were used as challenge data whereas we used exclusively non-invasive human scalp EEG data in our challenge. The Kaggle-style Neureka challenge employed a scalp EEG dataset and the TAES scoring metric but tasked participants to develop AI models for forecasting seizures (i.e. unlike detection, predicting them before they occur) and to at the same time minimize the number of EEG channels. The winning model showed a human-level FA rate of 1.44/24 h but also a sensitivity of 12.37% which prevents the model from being suitable for real-life clinical applications [44].
Future work will focus on assessing the suitability of our assistive EEG annotation system in real-world clinical settings and on upgrading and open sourcing our model-to-data crowdsourcing AI challenge platform based on the insights we gained from running the Deep Learning Epilepsy Detection Challenge.
All authors read and approved the final version of the manuscript. The corresponding authors Stefan Harrer and Gustavo Stolovitzky declare that they had full access to all the data in the study and that they had final responsibility for the decision to submit for publication.

Declaration of Competing Interest
SR, IKK and SH are inventors on issued patent US 10,596,377. HY is an inventor on pending patent US 16/670,177. All other authors do report no conflicts of interest.


Detection of Berezinskii-Kosterlitz-Thouless transition via Generative Adversarial Networks

The detection of phase transitions in quantum many-body systems with lowest possible prior knowledge of their details is among the most rousing goals of the flourishing application of machine-learning techniques to physical questions. Here, we train a Generative Adversarial Network (GAN) with the Entanglement Spectrum of a system bipartition, as extracted by means of Matrix Product States ans\"atze. We are able to identify gapless-to-gapped phase transitions in different one-dimensional models by looking at the machine inability to reconstruct outsider data with respect to the training set. We foresee that GAN-based methods will become instrumental in anomaly detection schemes applied to the determination of phase-diagrams.

Introduction
Recently we have experienced an explosion in the application of machine learning related techniques to face physical problems and in particular in quantum many-body and statistical physics [1,2]. The relevance of these techniques is grounded in the ability of the machines to process extremely complex data in an efficient way for the sake of pursuing classification tasks, pattern recognition and even generation of brand new data responding to some constraints or development of decision processes. In spite of their impressive achievements, the research is still in constant and hectic evolution and a great benefit is represented by the application of these techniques to fields beforehand unexplored. Physics is an extremely fruitful discipline in this sense. The common aim of manipulating and studying complex data allows a sharing of expertise that leads on the one hand to reveal new physics and enhance the power of analysis and on the other hand it helps the machine learning community improve the techniques and understand the intriguing mechanisms of artificial learning.
One of the specific tasks in physics that is undergoing to a huge effort in machine learning application is the phases recognition and classification. There is plenty of very recent works concerning the study of phase transitions both with supervised and unsupervised algorithms [3][4][5][6][7][8][9][10][11][12]. Indeed, the exploration of phase diagrams has always been a relevant physical problem both from a numerical/analytical and experimental point of view. The detection of a phase transition of a system normally consists in the measure of specific quantities which reflect some changes in the physics through the transition from one thermodynamic state to another. We believe that the most relevant strengths of machine learning algorithms in generalization and discovering of hidden patterns can perfectly suit the scope. In particular, protocols like anomaly detection have lately demonstrated to be a powerful tool for the mapping of physical systems' phase diagrams in a completely unsupervised manner and with very little or even no prior physical information about the kind of phases under consideration.
In this work, we develop a new architecture able to map the phase diagram of physical systems through an anomaly detection scheme. Our method is based on Generative Adversarial Networks (GANs) and improves the networks adopted by the previous works of [10,13] because of the different training procedure that inherits the improved modern standards from the machine learning community. We test our method with the detection of the Berezinskii-Kosterlitz-Thouless (BKT) phase transition in three different quantum systems at zero temperature. The very peculiar aspects of such a transition have been object of intense theoretical and experimental studies since its first prediction for the classical two-dimensional XY model [14,15]. The absence of an explicit symmetry breaking makes its characterization beyond the Landau-Ginzburg criterion. Moreover, its gappless-to-gapped nature implies a very slow convergence with the system's size and a smooth behaviour of the thermodynamic quantities. For these reasons, the determination of the critical parameters has always been a challenging task if compared with other kinds of phase transitions. Recently, some explicit attempts to address this task by means of machine learning algorithms were made [10,11,[16][17][18][19][20].
The choice of the input features for the network is crucially linked with the success of the learning outcome together with the skeleton of the architecture. We choose to employ the entanglement spectrum (to be defined below), a very general quantity related to the degree of quantum correlations among the sub-portions of the system. Its use was originally proposed in [5] as input data on which the unsupervised detection of patterns leads to the identification of phase transitions. The study of entanglement properties had indeed a huge reception in the physics community for the characterization of quantum many-body systems and has driven the escalation of the quantum information theories. We provide a detailed justification of this choice as well as some physical insights on what is happening to the features the network is fed with during the transition. Our approach is able to detect the BKT transitions in the XXZ spin chain, the Bose Hubbard model and its generalization for the two species case. Despite the microscopic differences, such models share a phase whose low-energy physics is described by Tomanaga-Luttinger liquids. Following the intuition of [21] we know that when such a description is valid, the entanglement spectrum exhibits very peculiar properties that are lost once the system undergoes the BKT phase transition. Therefore we train the GAN architecture by passing the (lower part of) the entanglement spectrum, and show that it is able to properly signal the emergence of anomalies in the spectrum when one crosses the BKT critical point and enters the gapped phases. It is worth stressing that such a task is performed without any a priori knowledge (order parameter, relevant correlation functions...) about the phases which break the Tomonaga-Luttinger liquid description. All the physical information is contained in the entanglement spectrum without further manipulation.
The method is not thought to compete with the most traditional methods for the detection of BKT transitions concerning the numerical precision. It is rather a way to identify with little prior knowledge the presence of different phases -as we show -even for small system sizes, and for subtle transition as the here studied BKT.

Entanglement Spectrum as dataset
In this work we deal with different quantum many-body systems at zero temperature. Because of the huge number of degrees of freedom, their characterization always requires a selection or a compression of the information in order to study their properties. We want to take into account a universal quantity that does not need the combination with any additional knowledge of the concrete system. For this purpose, we choose the degree of quantum correlation -or entanglement -among the internal components of the many-body systems that has been proven to contain a wealth of physical information.
There exist many measures of quantum entanglement and a big subset of the possibilities is based on the quantification of the correlations among sub-portions of the system. While still lots of questions are open concerning the multipartite case, the situation is now very clear for the bipartite entanglement of pure states. Specifically, the system is divided in two complementary sub-systems A (the new system) and B (the environment) and a measure for the degree of entanglement between them can be introduced. The most famous one is the Von Neumann entropy, that is exactly related to the Shannon entropy in information theory. It can be computed as S V N = −Tr A ρ A log ρ A (ρ A ≡ Tr B ρ being the reduced density matrix for the A subsystem). For the properties of the trace, a simpler expression for the entropy is obtained via diagonalization of the reduced density matrix ρ A in terms of its eigenvalues p i which are probability values. The flatness or steepness of their distribution is accounted for in the Von Neumann entropy for the sake of measuring the degree of entanglement. For example, a completely flat distribution witnesses a maximally entangled state, while a fully peaked one is associated to a product state, i.e. the quantum correlations between the two partitions are respectively maximum and minimum. Their distribution can be interpreted as a Boltzmann distribution of an effective entanglement Hamiltonian, in other words the transformed eigenvalues ξ i = − log(p i ) constitute the energy spectrum of the entanglement Hamiltonian that takes the name of entanglement spectrum (ES) [22,23]. From now on, for simplicity we will refer both to the entanglement Hamiltonian eigenvalues ξ i and the reduced density matrix eigenvalues p i as ES since they are related through a map.
Besides the trivial cases, it has been highlighted in several contexts that the ES exhibits relations and degeneracies which reflect very peculiar aspects of the physics and the symmetries of the system [24]. For this reason, the study of its structures already revealed itself as a powerful tool for the detection of changes in the physics of a system along a quantum critical point. The advantages are manifold: the ES is not directly related to any local order parameter nor correlator and it is not necessary to know the topological character of the phases -the topological order actually emerges automatically from the ES -in order to deal with it. The absence of prior knowledge on the system makes this choice very general and model-independent for all the cases in which the access to the ES is available, in contrast to the more traditional methods for the phases detection. In this work, we want to bring the study of the ES structures to another level: their interpretation is not human but machine driven. In particular, we want to perform the detection of a phase transition by means of machine learning algorithms in an unsupervised fashion. We develop a GAN architecture that takes as input feature the ES and is able to detect the changes in its structures when the system undergoes a phase transition.

GAN for unsupervised anomaly detection
The idea behind the method is based on the same principles of Refs. [10,13]: the detection of the phase transition relies on the ability of reconstruction of the ES. The principal ingredient for this purpose is the deep neural network auto encoder (AE) which maps every vector of real positive eigenvalues of the ES included in the [0, 1] interval (that we call x) to another vector (z) in a low-dimensional latent space through a parametrized function f θ and maps back z to the original space with another parametrized function g φ : The parameters φ and θ are optimized during the training procedure on samples that belong to a chosen region of the phase diagram (normal samples) meaning that the AE learns to reconstruct the original input in that region. The ability of reproducing the normal samples is lost when the AE faces an abnormal vector. In order to quantify this ability, we look at the reconstruction (or reproduction) loss between the input x = (λ 1 , ..., λ N ) and the output x = (λ 1 , ...,λ N ): Figure 1: Sketch of the GAN architecture we used for the phase transition detection. The AE takes in input a vector with the eigenvalues of the ES (x) and operates an encoding and a decoding to its reconstructed versionx. The discriminator takes in input x orx and gives a real numberŷ.
In this way we choose the anomaly score that indicates the presence of an abnormality in a sample if it is greater than a certain threshold at test-time. In other words, if the AE is correctly trained, it has learnt a probability distribution from which the training examples are sampled and can detect when an example does not belong to the same distribution by quantifying its being non-conforming to it. In the exploration of a phase diagram, this protocol has several advantages because, for instance, it allows to choose the region of the normal data where their production is numerically favourable or where the physics of the system is well understood. Nevertheless, it allows for a scan of the phase diagram in an unsupervised way because no prior knowledge of the possible phase labelling is required. The usual training procedure involves the minimization of the reconstruction loss but it has been demonstrated that the adversarial training usually improves the optimization of the parameters and entails a better reconstruction. We decide to provide the AE with a discriminator that takes x orx as input and outputs a single real numberŷ. The composite structure is a GAN [25] and is depicted in Fig. 1. The role of the discriminator is to distinguish if the vector in input is a real example (one ES from the training set) or a fake one (a reconstructed ES from the output of the AE). For real examples the output of the discriminator should reachŷ = 1 whereas for fake examplesŷ = 0. The two networks are trained together in a competitive way following this scheme for every batch of training samples: • the discriminator is optimized alone in order to be able to predict that N batch /2 samples from the dataset are real and that their reconstructions by the AE are fake • the AE -that is a full-fledged generator of samples -is optimized in order to fool the discriminator, hence making it predict that the remaining N batch /2 reconstructed samples are real.
During the training, the following adversarial loss is minimized, jointly with the the reconstruction loss: where y is the target value of the discriminator. As in common practise for actualizing the adversarial competition, we separately train the sub-networks by freezing the respective parameters' optimization. Since in general the reconstruction loss and the adversarial loss range within intervals of different magnitudes, we combine them adding some weights in the overall loss: The hyperparameters λ and are tuned in order to adjust the influence of the individual losses as well as the learning rates of the two subnetworks' optimizers. At test-time the discriminator is not necessary anymore: the AE -that have benefited from the adversarial training -faces the whole phase diagram and the anomaly score is computed. We employ a GAN architecture implemented with PyTorch [26,27]. Among all the possible structures for the layers, we tried to adopt a Convolutional Neural Network (CNN) whose principle is to convolve the layers with some filters but we find that the Dense structure over-performs the CNN for our task. The reason must probably be ascribed to the full-connectivity meaning that every neuron of each layer is connected with all the neurons of the previous and the following ones. In the AE, the MaxPool filter is added for down-sampling the features in the encoding process (by considering the biggest values and neglecting the smallest ones in a fixed size sliding window) and for the decoding part the UpSample is used for going back to the original space (it fills the holes created by the up-sampling to a bigger dimension through customized interpolations). Moreover, a skip connection between two latent layers is added to improve the model performance. It consists in the transfer of an encoding layer's information, i.e. for example x E , to the same-size layer in the decoder x D before the activation function. In Fig. 1 this is represented with a red arrow. Skip connections were first introduced to prevent gradient vanishing in deep networks and then in residual networks in order to overcome the so-called degradation problem (see [28,29] and references therein). Supposing that x E is transformed toward the decoder with a function F (x E ) and the symmetric layer in the decoder x D = σ(F (x E )) is obtained by applying an additional non-linearity σ, the skip connection: does not introduce neither extra-parameters nor computational complexity. Nevertheless, the optimization of the new F (i.e. the residual mapping) is easier thanks to the information of x E and, more specifically, was found to be responsible of the learning of good abstract representations in the bottleneck, rather than local details of the data [30]. All the manipulations on the data are made through standard built-in methods of the library.
To set out even more clearly and in mathematical terms, the f θ function of Eq. (1) can be written in its expanded form combining different fully connected modules of the form h(W γ · x + b γ ) where W γ is a parametric weight matrix, b γ is a bias vector and h is a nonlinear activation function. So z = f θ (x) is written as: with i the index of the layers. The h non-linear functions used are ReLu and Tanh as indicated in Fig. 1.
To our knowledge, this is the first time that an adversarial training of the AE is implemented via a GAN architecture in order to address the problem of the phase transition detection.

Description of the models
We are now set to discuss the application of the techniques to the BKT phase transition. Its elusive and usually numerically demanding nature derives from the exponentially slow gap opening and the smooth behaviour of the thermodynamic quantities, which call for quite sophisticate scaling analysis and often for calculations with twisted boundary conditions [31].
We decide to apply our method to three different one-dimensional (1D) quantum systems on a lattice at zero temperature. The bipartite entanglement properties for 1D systems that are described by a 1+1 conformal field theory (CFT) are now well known [32]: on one side it has been highlighted that the entanglement entropy scaling with the size of the subsystem depends on the central charge [33]; on the other side, in different contexts it has been demonstrated that the structures in the full ES contains even more: it can be used as a fingerprint of topological order [22,23], the difference between the two largest non trivially degenerated eigenvalues (Schmidt gap) changes along a quantum critical point [34,35] and the lower part of real-space ES is organized in interesting CFT structures [21] (see also Fig.2). In light of the fact that the transition we want to observe occurs between a Tomonaga-Luttinger liquid (described by a CFT as above) [36] and a gapped state, we know from [21] that the mentioned ES structures should disappear after the transition. This fact fits with the idea of using an anomaly detection protocol to locate the phase transition: the machine learns the structures of the ES in one phase and when it faces anomalies in these structures -i.e., when the physics of the system changes -it reports that there has been a deviation from the training dataset's normal behavior. The variation of these features normally manifests clearly in the thermodynamic limit. Even though the small changes in the ES for the accessible sizes of the physical systems would make the discernment between two phases a delicate task for the human eye, the machine succeeds in carrying out the distinction.

XXZ model
The first model we investigate is the XXZ model, which is exactly solvable. The boundaries of the phase diagram are therefore precisely known (for very good reviews see [37,38]) and it can be used as a benchmark. The Hamiltonian for the spin 1/2 chain of L sites is: where S α = 1 2 σ α (σ α being the Pauli matrices). By changing the magnitude of the anysotropy ∆, the model has a ferromagnetic ground state (∆ > 1), a paramagnetic behaviour that is described by a Luttinger liquid (−1 < ∆ < 1) and an anti-ferromagnetic ground state for ∆ < −1. We are interested in the transition between the Luttinger liquid phase and the antiferromagnetic one because it belongs to the BKT universality class. Instead, the transition from the paramagnet to the ferromagnet is first-order and therefore much less difficult to identify.

Bose Hubbard model
The second model we study is the Bose-Hubbard model (BH). The phase diagram of this nonintegrable model has been thoroughly analized in literature and, hence, it represents a good benchmark for our unsupervised phase recognition method. Detailed information on both the numerical and experimental aspects about this method can be found in reviews [39,40]. The Hamiltonian for a L-sites system is: with b † j (b j ) the bosonic creation (annihilation) operator at site j and n j = b † j b j the number operator. For fixed unitary filling ν = n j = 1, the ground state behaves like a 1D superfluid below U/J 3.39 [41] whose low-energy spectrum is effectively described by a Luttinger liquid. The transition from the superfluid phase to the Mott insulator phase at this filling is a BKT transition too.

Two-component Bose Hubbard model
The last model we address is the generalization of the BH for two-species (A and B), i.e., two Bose-Hubbard models coupled via a local density-density interaction term. The Hamiltonian reads: The phase diagram of the two-species Bose Hubbard model (BH2S) is very rich and it includes pair-superfluidity, counterflow superfluidity, phase separation, charge density wave quasi order and supersolidity [42,43], whose boundaries in are still not well determined. We focus on a small region of the phase space by choosing a Z 2 symmetric mixture with hopping parameters t α = 1, fixed intra-species on-site repulsions U α = U = 10 and total unitary filling ν A = ν B = 0.5. For U AB < 0 the ground state is either composed of two superfluids (2SF) or of a single superfluid of couples -namely pair-superfluidity (PSF) -for inter-species attractions U AB small enough to prevent collapse [44]. In the 2SF phase, the low energy spectrum is described by two Luttinger liquids corresponding to gapless modes in the density (in-phase) and spin (out-of-phase) channels. Through the transition to PSF, the spin channel acquires a gap and undergoes to a BKT phase transition whereas the density channel remains gapless. It is worth to mention that the position of the quantum critical point is not only still unknown but it is not completely clear [45] if the transition occurs just below U AB < 0 or for a finite value of U AB in the thermodynamic limit. In this work we provide numerical evidence that is occurs for a finite U AB for every size we simulate, in support to out previous work [44] with a completely different method.

Results
We deal with the many-body problem for finite sizes of the systems through a Matrix Product States (MPS) ansatz. It is always possible to extract the eigenvalues of the reduced density shows the same data of (a) obtained by subtraction of the lowest parabola ξ k=0 and by setting the distance between ξ k=1 δN =0 and ξ k=0 δN =0 equal to one in order to better appreciate the equally spaced structure and the degeneracies. In (d) and (e) the equivalent procedure as in (b) is applied to the diagonal sections of the paraboloids which correspond to the density (δN A = δN B ) and the spin channel (δN A = −δN B ) respectively. After the phase transition to PSF, only the eigenvalues of the density channel (d) maintain the conformal behaviour while the spin channel acquires a gap as shown below in Fig. 3. matrix p i from the tensor network by means of contractions of the tensors and singular value decompositions (SVDs) without altering the physical content of the stored information [46,47]. This is possible for every physical bipartition and specifically we choose a symmetric bipartition that cuts the system in the middle.

Features of the dataset
We show in Fig. 2 the ES for the BH and the BH2S in the Luttinger liquid regime. We display the logarithm with base 10 of the reduced density matrix eigenvalues ξ = − log 10 (λ i ), which are related to the eigenvalues of the entanglement Hamiltonian, as function of the particle number sector (for the XXZ model being the number of spinless interacting fermions resulting from the mapping of the Hamiltonian under Jordan-Wigner transformations [38]). The choice of the base 10 is just related to the more convenience in the order of magnitude readout. Since there are multiple eigenvalues for each sector, they can be ordered in a decreasing way: we label every eigenvalue with the (shifted) sector number, δN = N − L, and with the sorting index, k.
For the XXZ and the BH model, the structures of the ES at fixed k exhibit a parabolic envelope and the parabolas with different ks have the same curvature. Besides, the parabolas are translated by multiples of the difference between the first and the second eigenvalues of the δN = 0 sector: As a consequence, by subtracting every parabola by the first one (k = 0) and by rescaling with ∆ξ 0 , the manipulated ES displays some equally spaced structures with integer distances whose degeneracies are explained through a parallelism with energy spectra of boundary CFTs [21]. These conformal structures are then destroyed after the phase transition as can be seen by comparing Fig. 2 and Fig. 3.
Despite the apparent ease of description, the extraction of such parabolas (and their degeneracy counting) from a finite size system is far from being an easy task, since the convergence to thermodynamic limit is, once more, extremely slow. For this reason, the ability of generalization typical of a machine driven approach is instrumental in carrying out the pattern recognition for pursuing the distinction of the different regimes.
More interesting properties appear in the generalization to the BH2S model where the situation is slightly more complicated. The symmetries associated to the A and B number of particles' conservations provide two labels for the sectors δN A and δN B . The parabolas become equally spaced paraboloids in the region where the two Luttinger liquids coexist. As already anticipated, once the system undergoes the phase transition to PSF the gap opens only in the spin channel. Indeed, as clear from Fig. 3 after the opening of the gap, the arrangement of the ES eigenvalues on the density channel slice of the paraboloids (δN A = δN B ) still behaves according to the above-mentioned conformal structures while it loses these fascinating properties in the spin channel (δN A = −δN B ). It is furthermore relevant that in the PSF the parabolic envelopes observed in the density channel are the only preserved while the ES structures do not exhibit parabolas for any other direction on the δN A /δN B plane. We did not find any analysis about this generalization in the literature. we plot the result for the BH model. In this case the loss starts to rise after the training points but remains under the 10%, instead in the region of the transition the behaviour of the curves changes according to the size of the system. Even by changing the training window, the splitting of the curves remains very closed to the transition. In (c) the BH2S model for which the predicted value for the transition is still uncertain. The plot shows a trend very similar to the single species BH model. We hypothesize that the transition should happen in the −0.25 < U AB /U < −0.2 interval where the curves begin to separate in analogy to the previous case. This result is in good agreement with the prediction in [44].

BKT detection
In order to carry out the BKT detection, we consider the biggest eigenvalues of the ES in magnitude for various system's sizes for all the three different models. Their sorting is done in a decreasing way for the first ES corresponding to the origin of the phase diagram. Then all the spectra are sorted along the same sequence of symmetry sectors, without however passing the charge value to the GAN. Therefore it has no knowledge about the underlying parabolic structures and the whole procedure cannot be ascribed to a simple fitting of them. We divide a portion of the data in the Luttinger liquid regime in two disjoint datasets: training set and validation set. The training was always performed with samples belonging to the regions where the system behaves like Luttinger liquids, therefore where the input features show the above-described peculiar properties. The validation points are instead taken from an interval of the parameter space after the training toward the transition. With this splitting, we set a threshold both for the training loss (L tr rec ) and the validation loss (L val rec ) at training time (see the App. A for numerical details). The idea behind this procedure is that if we pick the validation points sufficiently far apart from the phase transition, their loss will remain low even if the network does not account for them for the optimization. In this way the training and validation data are always coherent between each other for different system's sizes with respect to a random reshuffle and the criterion of setting a threshold makes sense. Otherwise it could happen that for some special random splitting the losses are particularly low or high depending on their arrangement on the phase space. Moreover, by looking for a low loss in the validation set we check that the network reaches an optimization good enough to generalize well just outside the training dataset. For the XXZ, the training samples for each system's size are taken from the region −0.65 ≤ ∆/J ≤ 0 (about 550 points) and the validation ones from −0.8 ≤ ∆/J < −0.65. The BH training consists in about 500 points from the interval 0 ≤ U/J ≤ 2.5 and the validation from 2.5 < U/J ≤ 3 while for the BH2S model there are about 200 samples from the region −0.1 ≤ U AB /U ≤ 0 as training points and the validation ones are from −0.15 ≤ U AB /U < −0.1. In order to check the stability of the algorithm, we verified that the results do not show substantial differences when the intervals are rescaled (see App. A for the details). Moreover, we found that the results are similar even if only one AE per model is trained on a single system's size and then faces the test sets for the other sizes. This procedure is applicable and could be useful when the computational production of the training dataset is particularly costly and a rough estimation of the phase diagram boundaries is enough in a first approach.
The final reconstruction loss curve for each size is finally obtained by subtracting the loss computed on the full dataset by the trained network with the average training loss in order to purge the curves of the noise. The results are represented in Fig. 4. We have proven that the predictions and the final AEs really take advantage of the discriminator if compared to AEs that are not trained via adversarial competition. We are able to achieve training losses of the 10 −3 order within 250 epochs. It is fundamental to reach good values of precision for pursuing an accurate reconstruction of the ES's structures since the magnitude of the eigenvalues drops rapidly after the first leading ones. This level of precision is sufficient to faithfully reproduce at least the first two parabolas of the ES in the Luttinger regimes whereas the AEs become unable to reconstruct the structures after the phase transition.
All the results show that the loss experiences a rise in the region compatible with the phase transition. Besides, the increase depends on the size of the system and exhibits greater and greater steep going towards the thermodynamic limit. As a difference with respect to the previous works involving the same scheme [10,13] in which the authors exploit tensor network techniques for the simulation of the thermodynamic limit, we cannot expect a sudden change in the ES structures both because of the finite size and the BKT nature of the transition. Anyway, by looking at their results for the first and second order phase transitions we suspect that the loss increase in those circumstances would be much more sharp. In any case, it is worth to point out that this method allows for an automate scan of the phase diagram that cannot be a substitute to usual numerical methods for the location of the transition (for the BKT see for example [41,48]), therefore getting a comparable precise value for the critical parameters is beyond the scope.

Conclusion
In this work we show that the entanglement spectrum represents a reliable quantity to look at in order to perform a machine driven detection of the elusive BKT transition for onedimensional quantum systems by injecting almost no prior knowledge about them. We face such a task by using a GAN that benefits from the adversarial training in an anomaly detection scheme. We find that such architecture is a valid candidate for the purpose of discerning the change in the physics associated to the BKT transition and capable to remarkably improve the performances with respect to the previously adopted networks.
With this work we make a step forward in the exchange of technology between the physics and the machine learning communities. A step that gives rise to new interesting applications that we leave for a future investigation, e.g. the analysis of the features and correlations that the machine recognises as relevant for the detection of a phase transition; the tackling of the same problem by using different input features for the machine; a study on how far the machine can go in the understanding of the phase transition nature and universality in order to guide the entrance of more sophisticated quantities to solve the problem (like e.g. the stiffness specifically for our case); the implementation of an hopefully more precise procedure for the location of the transition by combining the losses obtained from the training in all the different phases separately. While writing this manuscript we became aware that even better performances could be achieved by taking inspiration from the most modern techniques adopted by the machine learning experts such as GANomaly architecture, AnoGAN, etc. [49][50][51]. Let us conclude by remarking once again, that this approach is not supposed to become a substitute to the traditional methods for the phase transitions detection but allows to obtain a qualitative map of a phase diagram with very little knowledge about the nature and arrangement of the phases in an automatic fashion.
Note added: during the submission process of this manuscript we found out [52] that provides a slightly different approach to the detection of phase transitions in spin systems using a GAN architecture.

A Numerical details and training windows
The training was performed by setting λ = 0.1 and = 10 in Eq. (4). The typical threshold for the losses in the training and validation region for the XXZ model areL tr rec = 0.00005 and L val rec = 0.0001, for the BH model areL tr rec = 0.005 andL val rec = 0.02 and for the BH2S model areL tr rec = 0.005 andL val rec = 0.05. The parameters of the network were optimized through ADAM optimizer with learning rates of the generator lr G = 0.01 and of the discriminator lr D = 0.0001. For both optimizers, a CosineAnnealingLR scheduler was employed in order to adjust the learning rate at training-time and improve the convergence.
The stability of the method was checked by comparing the loss curves for different training regions, we report in Fig. 5 the results obtained by varying the width of the training window in the parameter space for the XXZ and the BH model. The loss computed from the GAN reconstruction is also compared with the Kullback-Leibler divergence (KL) defined as follows: which is a measure of the relative entropy between two probability distributions and therefore is a proper measure for the distance between two ES. The reference distribution was taken to be the ES at the origin of the phase diagram and the distance from it was computed for every ES labelled with the control variable. It is represented in Fig. 5 as a black dashed line whose values run over the rightmost y-axis. It clearly appears that for the XXZ case the KL has a delayed and slow change after the transition while for the BH model the change occurs in the right region but certainly less sharp than the loss performed with the GAN.