radiology reports are the primary work product of radiologists , and they contain a wealth of information that could be mined for teaching , research , administrative , and quality assurance purposes .
radiologists need to search radiology reports to locate specific findings and diagnoses , and there is a growing body of work to create methods to search radiology reports and to extract information from them.15 a good report search engine could enable for radiology what google has done for the world wide web  rapid and relevant retrieval of documents that users seek . several search engines for radiology reports have been described that retrieve reports based on word content.1,3,6 these search engines look for keywords that match terms in the user s search .
for example , search engines can look through collections of radiology reports and return a list of cases containing the word  appendicitis .
 a challenge limiting the success of current radiology search engines is their inability to differentiate positive findings from those mentioned in context of uncertainty and negation .
for example , three reports with impressions  no evidence of appendicitis ,   can not completely exclude appendicitis ,  and  acute appendicitis  would be retrieved in a user search for  appendicitis  even though only one of them actually refers to a case of appendicitis .
thus , only one of these three retrieved reports would be relevant to the radiologist s search ; the other two types of reports are false - positive documents .
these false positives clutter the search results and reduce efficiency of search . to circumvent this challenge , radiology report search engines would need to detect the context of findings in reports  positive , negated , or uncertain findings .
considerable work has been done on assertion classification,710 some of which used subsets of radiology reports to test their systems,2,7 but few have focused specifically on negations in unstructured radiology reports.11,12 in addition , little work has been done to date in applying such methods to improve the precision of searches of radiology reports . our goal is to develop a method to improve the precision of searches of radiology reports by determining the context of terms in the text .
specifically , our goal is to develop a system that detects negated and uncertain findings and uses this information to exclude the corresponding reports from the search results .
we describe radreportminer , a context - aware search engine , and compare its performance with a currently available keyword - based method of search .
following institutional review board approval , 238,952 radiology reports for clinical exams performed between january 1 , 2006 and december 31 , 2006 were acquired from the radiology information system and imported into a mysql database ( mysql version 14.12 distribution 5.0.45 for win32 , running on windows xp and apache http web server 2.2.6 ) . the front - end user interface was created using php 5.2.5 ( hypertext preprocessor ) . to create radreportminer , we built php scripts that applied a modified version of negex,13 an algorithm developed to detect negation in medical ( nonradiology ) reports
. negex is a rule - based system for detecting negation , adopting regular expressions to detect signals in the texts that commonly indicate negation .
we expanded negex s functionality by : ( 1 ) adding to its rule base a set of negation phrases specific to the radiology domain , such as  no evidence of ,  
we also implemented a module to detect uncertainty associated with findings by adding a separate category of  uncertainty  terms to negex , such as  can not exclude  and  not ruled out 
radreportminer parses free - text radiology reports into their individual sections , recognizing the main headings such as history , findings , and impression .
demarcating the report sections is important since radreportminer gives the user the opportunity to exclude partiuclar report sections that often contain search terms that are confounded by negation or uncertainty , particularly the history section . for this study , every report was processed in its entirety without excluding any section .
radreportminer identifies the start of the history section by looking for any of the phrases  history ,  
the history section ends when  findings  or a synonym of  impression  is encountered .
the radreportminer algorithm uses regular expressions to search sentences for negations preceding and following the keyword , up to six words apart ( inclusive ) , as with negex.13 radreportminer includes regular expressions to detect statements of uncertainty in a similar fashion as to detecting negation . when a match is found , radreportminer computes a relevance score using the rules listed in table  1 .
the relevance score is used to classify each report into one of two categories : positive or negative , indicating whether the report contains the search term in a positive context ( the term is stated to be present ) or a negative context ( term stated to be absent or uncertainty is expressed about the term ) . when radreportminer returns multiple search results , it sorts them by decreasing relevance to the radiologist . 

table  1relevance scoring and classification in radreportminerthis table lists the heuristic clues in text used to detect positive , negative , and uncertainty phrases in radiology reports and the score assigned to each when the corresponding clue is found to be associated with a finding .
the scores are summed for each finding and the total score is used to classify the report as positive or negative according the score range shown in the table relevance scoring and classification in radreportminer this table lists the heuristic clues in text used to detect positive , negative , and uncertainty phrases in radiology reports and the score assigned to each when the corresponding clue is found to be associated with a finding .
the scores are summed for each finding and the total score is used to classify the report as positive or negative according the score range shown in the table the scoring system classifies a report as negated when the search term is associated with a negation phrase in radreportminer s regular expression database . when a search term is found to be associated with an uncertainty phrase ,
the report is classified as uncertain . both negated and uncertain reports are considered negative or not containing the finding of interest .
reports containing the search term without any associated negation or uncertainty phrases are classified as positive . if the history section of a report can not be separated reliably or if the search term is found only in the history section , the report is classified as negative .
we created a set of gold standard report classifications by asking a radiologist to perform five searches of single terms in our radiology reports database .
the reports were retrieved using an exact - match keyword - based search engine , which has 100% precision and recall in identifying reports containing the keyword / key phrase of interest .
 optic neuritis ,   pneumonia ,   hydronephrosis ,  and  fracture . 
after each search term was entered , the radiologist manually categorized the first 100 results ( or fewer , if fewer were found ) as  positive  or 
negative  for the finding or diagnosis in question , thus establishing the ground truth .
if more than 100 reports contained the keyword of interest , only the first 100 reports were used .
a positive report was defined as one that gave the radiologist reasonable certainty that the desired finding or diagnosis would be present on the image(s ) being described .
all other reports were classified as negative . to evaluate radreportminer , the same set of reports for which the radiologist provided the gold standard classification
the reports were also written to individual files for processing by google desktop , a common word - based indexing engine , to compare its search results with those obtained using radreportminer .
after google desktop completed indexing the reports , the same five searches used to evaluate the radreportminer system were performed with google desktop .
link at the top right corner of the google desktop results was selected to sort the results using google desktop s internal algorithm .
precision and recall were calculated for radreportminer and google desktop , and the results were compared against the radiologist s ground truth determination for each report .
precision , the fraction of relevant documents retrieved by the search engine , was calculated by dividing the number of true - positive reports ( reports marked positive by the search engine that were also marked positive by the radiologist ) by the total number of reports retrieved ( classified as positive ) by the search engine .
recall , the fraction of all existing relevant documents retrieved , was calculated by dividing the number of true - positive reports by the total number of reports marked positive by the radiologist within the 100-reports - or - fewer collection .
mean precision and recall were computed in two different ways : per - term and per - document . per - term
means were calculated by weighing each search term equally , regardless of the number of reports retrieved ( denominator is the number of search terms ) . per - document
means were calculated by weighing each retrieved report equally ( denominator is the number of reports retrieved ) .
fisher s exact test was used to determine statistical significance of the differences between the two search methods for each search term .
wilcoxon signed ranks test was used to detect differences in overall precision and recall between the two search methods .
a total of 464 reports were returned for the five search terms ( four terms each yielded the maximum 100 reports and one yielded 64 reports ) .
the radiologist classified 119 ( 26% ) of these as positive and 345 ( 74% ) as negative .
of these , 86 ( 81% ) were true positives ( marked as positive by the radiologist ) ; thus , radreportminer achieved an overall per - document precision of 81% ( 86 of 106 reports ) and a recall of 72% ( 86 of 119 reports ) .
google desktop classified 385 documents as positive , of which 104 ( 27% ) were true positives .
thus , goolgle desktop achieved a precision of 27% ( 104 of 385 reports ) and a recall of 87% ( 104 of 119 reports ) . per - term and per - document calculations produced the same numbers except for radreportminer s recall statistics , which were 76% per - term and 72% per-document.the overall precision for radreportminerwas significantly higher than that of google desktop whether weighing each report equally ( per - document ; p  <  0.0001 ) or weighing each search term equally ( per - term ; p  =  0.042 ; median difference of 48% with 95% confidence interval of 3475% ) .
google desktop generally achieved higher recall than radreportminer ; the difference was marginally significant per - document ( p  =  0.0057 ) but not statistically significant on a mean basis per - term ( p  =  0.273 ) ; for two individual terms , the differences were statistically significant .
table  2search performance comparison between radreportminer and google desktopthe top third of the table lists the terms searched and the number of results retrieved by each search engine .
the middle third shows the precision values associated with each search term , calculated by dividing the number of true - positive reports by the total number of reports retrieved , with the percentage to the side .
the bottom third shows recall values , calculated by dividing the number of true - positive reports by the total number of positive reports .
the right - most column lists the p values search performance comparison between radreportminer and google desktop the top third of the table lists the terms searched and the number of results retrieved by each search engine .
the middle third shows the precision values associated with each search term , calculated by dividing the number of true - positive reports by the total number of reports retrieved , with the percentage to the side .
the bottom third shows recall values , calculated by dividing the number of true - positive reports by the total number of positive reports .
the right - most column lists the p values as shown in table  2 , a different number of reports were retrieved by radreportminer and google desktop for the search terms  fracture  and  pneumonia ,  for which google desktop returned 83 and 38 reports , compared with 100 and 100 reports for radreportminer , respectively . for the three other search terms , both search engines returned the same number of reports .
the majority of false positives from radreportminer were due to unrecognized uncertainties and word distance ( the number of words from the negation phrase to the search term , including both ) greater than six words .
for example , with  no  as the negation term and  appendicitis  as the search term , the following phrase which has a word distance of seven would not be recognized by radreportminer as a negated search term :  noperiappendiceal fat stranding to suggestappendicitis . 
the majority of false negatives from radreportminer were due to absence of keyword in the findings / impression sections and negation of some but not all instances of the keyword .
table  3radreportminer false positivesthe majority of radreportminer s false positives ( e.g. , nonfracture classified as fracture ) were due to unrecognized uncertainties and word distance greater than six words .
word distance is defined as the number of words from the negation phrase to the search term , including both .
for example , with  no  as the negation term and  appendicitis  as the search term , the following phrase has a word distance of seven :  noperiappendiceal fat stranding to suggestappendicitistable  4radreportminer false negativesthe majority of radreportminer s false negatives ( e.g. , true fracture classified as negative ) were due to an absence of the keyword in the findings / impression sections .
another 20% were due to negation of certain instances of the keyword but not others .
the algorithm classifies a report as negative once a single instance of negation of that keyword is found radreportminer false positives the majority of radreportminer s false positives ( e.g. , nonfracture classified as fracture ) were due to unrecognized uncertainties and word distance greater than six words .
word distance is defined as the number of words from the negation phrase to the search term , including both .
for example , with  no  as the negation term and  appendicitis  as the search term , the following phrase has a word distance of seven :  noperiappendiceal fat stranding to suggestappendicitis  radreportminer false negatives the majority of radreportminer s false negatives ( e.g. , true fracture classified as negative ) were due to an absence of the keyword in the findings / impression sections .
another 20% were due to negation of certain instances of the keyword but not others .
the algorithm classifies a report as negative once a single instance of negation of that keyword is found
the ideal radiology search engine is one that has high  recall  ( retrieving all the relevant reports pertinent to a user s interest from the entire database of reports ) and precision ( retrieving mostly relevant reports among all reports retrieved ) . for searching radiology reports
, precision is often more important than recall  when radiologists use a search engine to find reports containing a specific finding or diagnosis , they want high - precision searches to minimize the number of false - positive reports ; irrelevant reports require the radiologist to read and manually exclude them , wasting precious time and lowering efficiency .
high recall is important in situations where the radiologist is interested in searching for rare findings and diagnoses ; one would not want to miss any reports of such rare cases , and the number of false positives would be small since the search term is rare .
our interest is to improve search in circumstances of search for common diseases and diagnoses  to reduce the number of false - postive search results .
the current word - based search engines perform superbly in terms of recall , but often lack precision . while high recall is certainly beneficial when the radiologist is searching for uncommon entities , it can be highly problematic when searching for common diseases and findings .
consider , for example , a search engine with a precision of 27% , as we found with google desktop . a search using this engine that returns 200 reports contains only 54 that are true positives
 the radiologist would need to wade through and discard 146 false - positive reports to discover the positives . on the other hand , the radreportminer system , having a precision of 81% , would return only 38 false - positive hits .
given the high precision of radreportminer , it is not surprising that it had a lower recall than google desktop , though the difference was only marginally statistically significant .
our results demonstrate improved precision of our system over the word - based google desktop search engine .
we consider our results beneficial to radiologists wanting efficient methods to search reports with higher precision than current keyword - based methods provide .
in fact , based on our results , for each search term , one would need to go through an average of 36.8 more results returned by google desktop to find the same number of positive reports returned by radreportminer , amounting to reading 137% additional reports .
first , there were a small number of terms selected for the searches used in our evaluation .
we are currently undertaking a study on a larger sample of reports with more search terms .
another limitation of our work is that it does not account for multiple keywords and synonyms .
for example , we are unable to search for  pneumonia  or  infiltrate  near  infectious .
 we are also unable to search for both the singular and plural forms of  fracture .
a module to recognize multiple forms of a word and to handle multiple keywords would address several of these limitations .
all the software used in this project , with the exception of the operating system ( windows xp ) , is in the public domain and , thus , is affordable to any institution wishing to implement such a search engine .
we based our algorithm on a previously researched and published method of negation detection , which shortens the development time and reduces coding complexity .
we ultimately aim to deploy our methods in radiology report search systems to help radiologists to find information in vast report archives more effectively .
adding negation and uncertainty identification to a word - based radiology report search engine improves the precision of search results over a search engine that does not take this information into account .
our approach may be useful to adopt into current report retrieval systems to help radiologists to more accurately search for radiology reports .
additional negation phrases incorporated into the radreportminer algorithm : 
 clear ofhealedno evidence ofno evidence forresolution ofno longer seenno longer presentno longer appreciatedno longer visualizedhas resolvedhave resolvedhas healedhave healed no longer appreciated phrases 15 were added to the prenegation list .
uncertainty terms utilized in the radreportminer algorithm : 
 can not rule outcan not rule outcannot completely rule outcan not completely rule outcannot absolutely rule outcan not absolutely rule outcannot excludecan not excludecould representmay representequivocalcannot be ruled outcan not be ruled outcannot be excludedcan not be excludednot ruled outnot excludedcannot be completely excludednot completely ruled outshould also be considered can not completely rule out can not completely rule out can not absolutely rule out can not absolutely rule out can not be completely excluded not completely ruled out should also be considered phrases 111 are preuncertainties that occur before the search term .
 radiology reports contain information that can be mined using a search engine for teaching , research , and quality assurance purposes . 
 current search engines look for exact matches to the search term , but they do not differentiate between reports in which the search term appears in a positive context ( i.e. , being present ) from those in which the search term appears in the context of negation and uncertainty . 
 we describe radreportminer , a context - aware search engine , and compare its retrieval performance with a generic search engine , google desktop . 
 we created a corpus of 464 radiology reports which described at least one of five findings ( appendicitis , hydronephrosis , fracture , optic neuritis , and pneumonia ) . 
 each report was classified by a radiologist as positive ( finding described to be present ) or negative ( finding described to be absent or uncertain ) . 
 the same reports were then classified by radreportminer and google desktop . 
 radreportminer achieved a higher precision ( 81% ) , compared with google desktop ( 27% ; p  <  0.0001 ) . 
 radreportminer had a lower recall ( 72% ) compared with google desktop ( 87% ; p  =  0.006 ) . 
 we conclude that adding negation and uncertainty identification to a word - based radiology report search engine improves the precision of search results over a search engine that does not take this information into account . 
 our approach may be useful to adopt into current report retrieval systems to help radiologists to more accurately search for radiology reports . 
Introduction
Methods
Results
Discussion
Conclusion
Appendix A
Appendix B