How is the system constructed to be linear in the size of the narrative input and the terminology?
The system scans the text word-by-word once and performs a voting task for each word. It also keeps track of the position of the previous words.
The collection of narrative spontaneous reports is an irreplaceable source for the prompt detection of suspected adverse drug reactions (ADRs): qualified domain experts manually revise a huge amount of narrative descriptions and then encode texts according to MedDRA standard terminology. The manual annotation of narrative documents with medical terminology is a subtle and expensive task, since the number of reports is growing up day-by-day. MagiCoder, a Natural Language Processing algorithm, is proposed for the automatic encoding of free-text descriptions into MedDRA terms. MagiCoder procedure is efficient in terms of computational complexity (in particular, it is linear in the size of the narrative input and the terminology). We tested it on a large dataset of about 4500 manually revised reports, by performing an automated comparison between human and MagiCoder revisions. For the current base version of MagiCoder, we measured: on short descriptions, an average recall of $86\%$ and an average precision of $88\%$; on medium-long descriptions (up to 255 characters), an average recall of $64\%$ and an average precision of $63\%$. From a practical point of view, MagiCoder reduces the time required for encoding ADR reports. Pharmacologists have simply to review and validate the MagiCoder terms proposed by the application, instead of choosing the right terms among the 70K low level terms of MedDRA. Such improvement in the efficiency of pharmacologists' work has a relevant impact also on the quality of the subsequent data analysis. We developed MagiCoder for the Italian pharmacovigilance language. However, our proposal is based on a general approach, not depending on the considered language nor the term dictionary.
Pharmacovigilance includes all activities aimed to systematically study risks and benefits related to the correct use of marketed drugs. The development of a new drug, which begins with the production and ends with the commercialization of a pharmaceutical product, considers both pre-clinical studies (usually tests on animals) and clinical studies (tests on patients). After these phases, a pharmaceutical company can require the authorization for the commercialization of the new drug. Notwithstanding, whereas at this stage drug benefits are well-know, results about drug safety are not conclusive BIBREF0 . The pre-marketing tests cited above have some limitations: they involve a small number of patients; they exclude relevant subgroups of population such as children and elders; the experimentation period is relatively short, less than two years; the experimentation does not deal with possibly concomitant pathologies, or with the concurrent use of other drugs. For all these reasons, non-common Adverse Drug Reactions (ADRs), such as slowly-developing pathologies (e.g., carcinogenesis) or pathologies related to specific groups of patients, are hardly discovered before the commercialization. It may happen that drugs are withdrawn from the market after the detection of unexpected collateral effects. Thus, it stands to reason that the post-marketing control of ADRs is a necessity, considering the mass production of drugs. As a consequence, pharmacovigilance plays a crucial role in human healthcare improvement BIBREF0 . Spontaneous reporting is the main method pharmacovigilance adopts in order to identify adverse drug reactions. Through spontaneous reporting, health care professionals, patients, and pharmaceutical companies can voluntarily send information about suspected ADRs to the national regulatory authority. The spontaneous reporting is an important activity. It provides pharmacologists and regulatory authorities with early alerts, by considering every drug on the market and every patient category. The Italian system of pharmacovigilance requires that in each local healthcare structure (about 320 in Italy) there is a qualified person responsible for pharmacovigilance. Her/his assignment is to collect reports of suspected ADRs and to send them to the National Network of Pharmacovigilance (RNF, in Italian) within seven days since they have been received. Once reports have been notified and sent to RNF they are analysed by both local pharmacovigilance centres and by the Drug Italian Agency (AIFA). Subsequently, they are sent to Eudravigilance BIBREF1 and to VigiBase BIBREF2 (the European and the worldwide pharmacovigilance network RNF is part of, respectively). In general, spontaneous ADR reports are filled out by health care professionals (e.g., medical specialists, general practitioners, nurses), but also by citizens. In last years, the number of ADR reports in Italy has grown rapidly, going from approximately ten thousand in 2006 to around sixty thousand in 2014 BIBREF3 , as shown in Figure FIGREF3 . Since the post-marketing surveillance of drugs is of paramount importance, such an increase is certainly positive. At the same time, the manual review of the reports became difficult and often unbearable both by people responsible for pharmacovigilance and by regional centres. Indeed, each report must be checked, in order to control its quality; it is consequently encoded and transferred to RNF via “copy by hand” (actually, a printed copy). Recently, to increase the efficiency in collecting and managing ADR reports, a web application, called VigiFarmaco, has been designed and implemented for the Italian pharmacovigilance. Through VigiFarmaco, a spontaneous report can be filled out online by both healthcare professionals and citizens (through different user-friendly forms), as anonymous or registered users. The user is guided in compiling the report, since it has to be filled step-by-step (each phase corresponds to a different report section, i.e., “Patient”, “Adverse Drug Reaction”, “Drug Treatments”, and “Reporter”, respectively). At each step, data are validated and only when all of them have been correctly inserted the report can be successfully submitted. Once ADR reports are submitted, they need to be validated by a pharmacovigilance supervisor. VigiFarmaco provides support also in this phase and is useful also for pharmacovigilance supervisors. Indeed, VigiFarmaco reports are high-quality documents, since they are automatically validated (the presence, the format, and the consistency of data are validated at the filling time). As a consequence, they are easier to review (especially with respect to printed reports). Moreover, thanks to VigiFarmaco, pharmacologists can send reports (actually, XML files BIBREF4 ) to RNF by simply clicking a button, after reviewing it. Online reports have grown up to become the 30% of the total number of Italian reports. As expected, it has been possible to observe that the average time between the dispatch of online reports and the insertion into RNF is sensibly shorter with respect to the insertion from printed reports. Notwithstanding, there is an operation which still requires the manual intervention of responsibles for pharmacovigilance also for online report revisions: the encoding in MedDRA terminology of the free text, through which the reporter describes one or more adverse drug reactions. MedDRA (Medical Dictionary for Regulatory Activities) is a medical terminology introduced with the purpose to standardize and facilitate the sharing of information about medicinal products in particular with respect to regulatory activities BIBREF5 . The description of a suspected ADR through narrative text could seem redundant/useless. Indeed, one could reasonably imagine sound solutions based either on an autocompletion form or on a menu with MedDRA terms. In these solutions, the description of ADRs would be directly encoded by the reporter and no expert work for MedDRA terminology extraction would be required. However, such solutions are not completely suited for the pharmacovigilance domain and the narrative description of ADRs remains a desirable feature, for at least two reasons. First, the description of an ADR by means of one of the seventy thousand MedDRA terms is a complex task. In most cases, the reporter who points out the adverse reaction is not an expert in MedDRA terminology. This holds in particular for citizens, but it is still valid for several professionals. Thus, describing ADRs by means of natural language sentences is simpler. Second, the choice of the suitable term(s) from a given list or from an autocompletion field can influence the reporter and limit her/his expressiveness. As a consequence, the quality of the description would be also in this case undermined. Therefore, VigiFarmaco offers a free-text field for specifying the ADR with all the possible details, without any restriction about the content or strict limits to the length of the written text. Consequently, MedDRA encoding has then to be manually implemented by qualified people responsible for pharmacovigilance, before the transmission to RNF. As this work is expensive in terms of time and attention required, a problem about the accuracy of the encoding may occur given the continuous growing of the number of reports. According to the described scenario, in this paper we propose INLINEFORM0 , an original Natural Language Processing (NLP) BIBREF6 algorithm and related software tool, which automatically assigns one or more terms from a dictionary to a narrative text. A preliminary version of INLINEFORM1 has been proposed in BIBREF7 . MagiCoder has been first developed for supporting pharmacovigilance supervisors in using VigiFarmaco, providing them with an initial automatic MedDRA encoding of the ADR descriptions in the online reports collected by VigiFarmaco, that the supervisors check and may correct or accept as it is. In this way, the encoding task, previously completely manual, becomes semi-automatic, reducing errors and the required time for accomplishing it. In spite of its first goal, MagiCoder has now evolved in an autonomous algorithm and software usable in all contexts where terms from a dictionary have to be recognized in a free narrative text. With respect to other solutions already available in literature and market, MagiCoder has been designed to be efficient and less computationally expensive, unsupervised, and with no need of training. MagiCoder uses stemming to be independent from singular/plural and masculine/feminine forms. Moreover, it uses string distance and other techniques to find best matching terms, discarding similar and non optimal terms. With respect to the first version BIBREF7 , we extended our proposal following several directions. First of all, we refined the procedure: MagiCoder has been equipped with some heuristic criteria and we started to address the problem of including auxiliary dictionaries (e.g., in order to deal with synonyms). MagiCoder computational complexity has been carefully studied and we will show that it is linear in the size of the dictionary (in this case, the number of LLTs in MedDRA) and the text description. We performed an accurate test of MagiCoder performances: by means of well-known statistical measures, we collected a significant set of quantitative information about the effective behavior of the procedure. We largely discuss some crucial key-points we met in the development of this version of MagiCoder, proposing short-time solutions we are addressing as work in progress, such as changes in stemming algorithm, considering synonyms, term filtering heuristics. The paper is organized as follows. In Section SECREF2 we provide some background notions and we discuss related work. In Section SECREF3 we present the algorithm MagiCoder, by providing both a qualitative description and the pseudocode. In Section SECREF4 we spend some words about the user interface of the related software tool. In Section SECREF5 we explain the benchmark we developed to test INLINEFORM0 performances and its results. Section SECREF6 is devoted to some discussions. Finally, in Section SECREF7 we summarize the main features of our work and sketch some future research lines.