Introduction
============

Data are the lifeblood of the knowledge economy and health care. Google, Microsoft, and other modern companies compete with innovative solutions that support the generation, analysis, management, and exchange of data. Health care and public health similarly focus efforts on generating, collecting, analyzing, and sharing data about individual patients and populations, respectively. Data are vital to joint activities including surveillance of chronic and communicable disease, population health assessments, and health care policy. Effective practice requires access to representative, complete, and timely data from multiple sources ([@b1-ojphi-03-23], [@b2-ojphi-03-23]).

Unfortunately the quality of the data stored in information systems across industries and organizations is often poor ([@b3-ojphi-03-23]). Typical data quality issues encountered include: inaccurate data, inconsistencies across data sources, and incomplete (or unavailable) data necessary for operations or decisions ([@b4-ojphi-03-23]). For example, a large bank found that data in its credit-risk management database were only 60 percent complete, which necessitated additional scrutiny by anyone using its data ([@b5-ojphi-03-23]). In health care, the completeness of data in electronic health record (EHR) systems has been found to vary from 30.7 to 100 percent ([@b6-ojphi-03-23]).

Evidence from the information management literature on the impacts of these issues is sparse, but estimates of impacts include: increased costs ranging from 8--12 percent of organizational revenue, and up to 40--60 percent of a service organization's expenses consumed due to poor data; poorer decisions that take longer to make; lower data consumer satisfaction with information systems; and increased difficulty in reengineering work and information flows to improve service delivery ([@b4-ojphi-03-23]). Impacts on health care include ill-informed decisions when humans or machines use poor quality data inputs from EHR systems ([@b7-ojphi-03-23], [@b8-ojphi-03-23]).

The public health literature suggests additional impacts of poor data quality issues for public health agencies and processes. For example, spontaneous reporting rates for infectious diseases range from 9 to 99 percent and have remained relatively unchanged from 1970 -- 2000 ([@b9-ojphi-03-23]). While some conditions, such as sexually transmitted infections, are reported approximately 80 percent of the time, many conditions (e.g. pertussis, shigellosis) are reported less than half of the time. Timeliness, another attribute of data quality ([@b3-ojphi-03-23]), has also been found to be a challenge in public health reporting ([@b10-ojphi-03-23]). Delays in the receipt of notifiable disease data (timeliness) and the lack of a complete set of reports (completeness) impact public health agency surveillance processes, including but not limited to the ability of agencies to respond to emerging disease threats.

Although the available evidence suggests that data quality issues are real and can have significant impact on operational processes, personnel, and budgets, too often public health practitioners assume that the increasing flow of electronic data from various information systems used in modern practice are of equal quality. This can lead to suboptimal outcomes beyond those described above, including failed implementations of public health information systems and inefficiencies in data collection and analyses. To ensure the information and knowledge generated from electronic observational clinical data can support effective public health practice, public health agencies must develop effective strategies to accurately characterize electronic clinical data sources both during the initial deployment phase and routinely reassess data characteristics of the operational system on a regular basis. In this paper, we outline the practical needs regarding careful analysis and continuous re-examination of data and their sources. We then describe how our U.S. Centers for Disease Control and Prevention (CDC) Center of Excellence in Public Health Informatics is working with public health informatics stakeholders to develop a framework that will support the operationalization of analyzing data quality continuously within an organization. We conclude with thoughts on the implications for such a framework and our next steps in the development and validation of the framework.

The Importance of Understanding Data Needs in Public Health
===========================================================

Developing methods and operational practices for assessing data quality requires an understanding of the various public health business processes, as well as the context in which those processes occur. Business processes are sometimes referred to as "use-cases." To understand a particular business process or use-case, one must ask questions about what, where, when, why, and how data are collected, stored, shared, and used to support the activities performed by public health and health care professionals involved in the business process. An example of a public health business process is syndromic surveillance.

Syndromic surveillance detects initial manifestations of disease before diagnoses (clinical or laboratory) are established ([@b11-ojphi-03-23]--[@b13-ojphi-03-23]). Data and information in syndromic surveillance systems come from a variety of sources, including hospital emergency department visits, ambulatory clinic visits, school absenteeism, poison control centers, and over-the-counter medication sales ([@b2-ojphi-03-23]). In syndromic surveillance, agencies have an inherent need to understand the quality of the data from emergency departments, schools, and a variety of other sources to assess whether the data are able to accurately describe emerging public health threats. Poor quality data will lead epidemiologists down dead end pathways including false-alerts or missed alerts, which will waste scarce public health resources.

Initially syndromic surveillance focused solely on patients' chief complaints, the primary reason for their visit to a health care provider, as they presented in the emergency department (ED). Chief complaints were found to be generally representative, complete, sensitive, specific, and reliable enough to detect emerging outbreaks including the start of the influenza season and bioterrorism events ([@b2-ojphi-03-23]). Subsequent research demonstrated that grouping chief complaints into syndromes and adding discharge diagnosis information into syndromic surveillance analysis improves validity and reliability ([@b14-ojphi-03-23]--[@b16-ojphi-03-23]). The current effort to improve the meaningful use (MU) of EHR systems incentivizes ED and ambulatory providers to submit chief complaint data to public health ([@b17-ojphi-03-23], [@b18-ojphi-03-23]). Before leveraging chief complaint data from all EHR source systems, the quality of new sources must be assessed for completeness, validity, and reliability. For example, chief complaints reported by specialty providers, like thoracic surgeons, may not be as useful for detecting emergent threats as those from the ED or full-spectrum primary care clinics. Data from specialty providers likely are predominantly grouped into a single syndromic category and therefore likely to lack specificity and sensitivity, an issue under investigation now by the Indiana State Department of Health and other health departments across the U.S.

In addition to syndromic surveillance, MU is driving health care providers to increasingly submit electronic laboratory reporting (ELR) and immunization data to public health agencies. This will dramatically increase the number of data sources providing electronic data to public health agencies over the next 5 to 10 years. For example, in Indiana where we have steadily increased the number of providers utilizing ELR to report communicable disease cases to the state health agency over the last 10 years, we now have over 200 discrete data feeds from laboratory information systems, and expect that number to increase substantially. Continuously analyzing data quality across those feeds is a necessary but challenging task. ELR monitoring is especially challenging given the frequency with which laboratories update and append new tests to their catalogues ([@b19-ojphi-03-23], [@b20-ojphi-03-23]).

Beyond meaningful use, public health agencies must collect, manage, and utilize data and information across a number of program areas. Each program area may impose unique requirements on data captured through a general or shared collection method. Recently the state health agency in Indiana sent a letter to several hospital laboratories highlighting the lack of guarantor information (e.g., name and contact information for the person financially responsible for health care expenses) when submitting ELR information to the state's communicable disease program. The letter originated not from the communicable disease program director but a different program within the agency responsible for following up on case reports for minors. Some public health officials perceived the traditional patient contact information in the ELR messages as insufficient and decided to instead use guarantor information which typically contains parental or legal guardian's contact information. However, these decisions were not shared with the data providers nor laboratory directors until after the state had performed an analysis to determine which hospitals were sending the guarantor information and which ones were not. A lack of communication about data needs and uses across program areas within the agency resulted in confused laboratory directors contacting vendors and data partners about the letters from the state. Furthermore, it's important to note that the laboratory information systems (LIS), which are commonly used to process lab results, do not readily capture patient guarantor information. Instead, this is often managed by enterprise billing software. Consequently, supplying guarantor information likely requires non-traditional integration of two information systems, which may require new processes outside of the typical ELR information flow.

Furthermore many public health professionals, like data consumers in other health care segments and industries, believe that data is easily and uniformly captured and stored across the spectrum of health care services. However, data are captured for a specific purpose, and the collection of additional data elements is costly. Additional data elements require staff to ask for and then record the information, which translates into additional time and labor. Software systems often must be modified to accommodate the new data fields and new workflow, also a costly process. Therefore data consumers must understand the impact of the cost of data collection on the characteristics of data captured in various environments, like their completeness, when making decisions about secondary use. Public health officials, for example, might benefit from understanding that elements like the provider's phone number and address have little clinical relevance to the physician receiving the results of a lab test. These fields are poorly populated by laboratory information systems. Although these fields can be required according to state and federal regulations, it does not guarantee that they will be complete and available for public health surveillance processes. For example, provider addresses and phone numbers are missing in more than 95% of the electronic laboratory messages sent directly from lab systems to the Indiana Network for Patient Care (INPC), a health information exchange repository that receives lab data from over 70 hospitals and their send out facilities ([@b21-ojphi-03-23]). Thus policies to require additional data elements are unlikely to impact data collection processes unless laboratories and hospitals are incentivized to capture the additional data elements needed for public health surveillance processes.

The above set of issues illustrates a clear need for public health agencies to assess and document their data needs and sources. Without clear expectations regarding public health's desired needs and uses of data, information systems will likely receive, analyze, manage, and generate poor quality data, information, and knowledge about population health. One pressing role for public health informatics professionals is clear: they must support agencies' assessment of data needs and sources. This means asking questions about the what, how, and why of existing business processes as well as the data, and their sources, involved in those processes. Informatics professionals should further capture and share information with public health stakeholders regarding what can and cannot be feasibly collected from clinical information systems given existing information and work flows. However, there is a paucity of practical tools and resources for informatics professionals to utilize. In the rest of this article, we describe our approach to developing a framework to support better assessment of needs and sources to support more efficient use of information systems and resources.

What Our CDC Center of Excellence is Doing to Better Understand Data Needs
==========================================================================

The Indiana Center of Excellence in Public Health Informatics (ICEPHI) resolved to develop methods and operational practices for assessing data quality from a myriad of clinical data sources. Our initial attempt involved scheduling meetings with a broad range of stakeholders within the Indiana public health community. We invited public health researchers, state health officials, local health officials, and informatics professionals in both health care and public health settings. While the meetings confirmed the real-world challenges previously described, they were suboptimal with respect to outlining the necessary methods and practices for performing data quality assessment. Given the demands of public health personnel, not surprisingly meetings often only contained a subset of the full stakeholders involved in ICEPHI, so it was not possible to have every stakeholder in attendance at any given meeting. Furthermore, participating stakeholders often failed to provide full details about their information needs. This was due to a variety of reasons, including individuals lacking full details for a particular workflow, individuals who are uncomfortable speaking at meetings, and individuals who primarily work with reports but do not fully understand the provenance of the data that comprise those reports. We transitioned from the meeting strategy when it became clear that this approach would not capture data quality perceptions and needs in a systematic fashion that would meaningfully inform standard methods and practices.

To systematically explore data quality perceptions and needs from a diverse group of stakeholders in public health and informatics, we selected a modified Delphi technique ([@b22-ojphi-03-23]). The Delphi technique was chosen because this method can 1) structure group communication and decision making processes and 2) foster consensus from very diverse groups of individuals ([@b23-ojphi-03-23]). Delphi studies have proven useful in a range of industries and settings, including medicine, nursing, and informatics ([@b23-ojphi-03-23]--[@b25-ojphi-03-23]).

The Delphi study is being conducted as the third of three phases in our systematic approach to examining data quality perceptions and needs. In phase one, we convened a diverse set of purposefully selected stakeholders from across academia, state health agencies, and local health departments to define and prioritize a limited set of public health work processes that involve data collection from health care providers and information systems. The majority of our stakeholders were selected from Indiana. However, we further invited select individuals from other states who had previously collaborated with our Center of Excellence.

In phase two, informatics researchers within ICEPHI compiled a list of data elements required under Indiana state law to be reported to the health department for the various work processes under discussion. The stakeholders were invited to review this list and propose additional data elements they felt would be useful to have in those work processes. The additional data elements were appended to the respective work process lists. These final lists represent an initial, stakeholder-driven view of the data necessary to optimally perform common work processes in public health practice. These processes included: aggregate surveillance for influenza across counties and the state; individual case reporting for meningococcal disease; and ongoing monitoring of diabetes levels at the county and state level.

The third and final phase of our design involves administering the modified Delphi study. We used the work processes, including their full descriptions and list of data elements, from the prior phases to create an online survey instrument to record the opinions and perceptions of a much wider group of public health stakeholders. Our goal is to recruit 20--30 stakeholders with expertise in various aspects of public health practice, those which align with our work processes, and achieve consensus on the need for and use of the data elements specified in phase two. Individuals in the larger stakeholder group will be able to suggest additional data elements each round, and participants will be able to amend their responses based on a review of the groups collective thinking to date (this is a major component of Delphi studies). In addition to helping ICEPHI more robustly specify the work processes and necessary data elements, stakeholders will also be asked to indicate their perceptions of the quality of data they currently receive from existing sources. We will, for example, ask about the timeliness and completeness of patient contact information provided in existing case reports. Here, too, we hope to achieve consensus on the perceived need for specific data elements to be timely or complete since various data sources will possess different timeliness and completeness characteristics.

To date we have completed the first two phases of our project as well as the survey instrument we will use to conduct the third phase. In the coming months, ICEPHI will recruit a sizable group of public health stakeholders and conduct the Delphi study with this group.

Future Directions and Work
==========================

The primary goal of our work is to utilize the product of the study to develop and evaluate systems that can improve public health practice in Indiana. We aim to take the results of the Delphi study and begin applying it to our work within the INPC. We will focus on the prioritized work processes and data elements to ensure the INPC is addressing these adequately for the local and state agencies engaged with data exchange activities. Where deficiencies are found, we will work with ICEPHI stakeholders to improve data capture, analysis, and reporting. We will further examine and document the challenges with meeting the consensus-based priorities of public health stakeholders given the state of clinical information systems, MU, and the capacity of the INPC. Our findings and lessons will be shared with the public health informatics community to inform future public health and informatics research, practice and policy in Indiana and beyond.

In addition to local applications, we intend to develop a workbook or other product to provide guidance on data quality issues to other public health entities. The workbook would consist of both a general framework for addressing data quality issues in a setting where electronic data is being received from numerous information systems and templates that agency personnel could complete to identify and prioritize agency business processes and needs. The framework will be largely informed by our Delphi study methods. However, we recognize that agencies will likely lack the ability to replicate a full Delphi study. Recognizing that simple stakeholder meetings are insufficient, the framework will identify strategies for bringing stakeholders together meaningfully to discuss and prioritize items developed initially by agency staff. The templates will guide staff members through the complex issues surrounding data quality and help them generate documents that stakeholders can review and use to establish priorities, policies, and directives. The goal for the workbook is to be practical, and its design and usage will be modeled after other practical informatics tools such as the U.S. Agency for Healthcare Research and Quality (AHRQ) "Health Information Technology Evaluation Toolkit" ([@b26-ojphi-03-23]) and the Public Health Informatics Institute (PHII) "Collaborative Requirements Development Methodology" ([@b27-ojphi-03-23]).

Finally, our aims include a comparison of measured data quality perceptions and needs with measured real-world data quality from clinical information systems. Our current project will collect perceived quality of existing ELR, syndromic surveillance, and other public health information system data as well as the perceived needs of those in public health working with ELR, syndromic, and other information system data. ICEPHI also includes an operational health information exchange (the INPC) with hundreds of interfaces to real-world clinical information systems. We aim to measure the quality of the data (e.g., completeness, timeliness) flowing into the INPC. We can then compare the real-world quality of the data with the perceived quality and needs of public health stakeholders. Such evidence will provide insight on where we are in developing public health informatics systems in relation to where public health practitioners need them to be.

Conclusion
==========

The quality of data from information systems varies over time and across systems, and it is often poor. Using poor data from clinical information systems can lead to poor decisions regarding patient care and public health policy. Despite an abundance of poor data flowing through electronic systems, current policies and practices, knowingly or unintended, assume equal quality across all sources therefore pushing for the adoption and use of more electronic data feeds to public health. Public health officials, epidemiologists, informaticians, and other professionals who recognize the unequal quality of data sources lack practical tools and resources to adequately assess quality to improve the utilization of electronic data sources that feed public health information systems.

Our CDC funded Center of Excellence in Public Health Informatics will continue to examine the quality of data and their sources. We aim to develop novel, practical approaches for continuously assessing data quality to improve the decisions and work processes involved in modern public health practice. Our findings and lessons learned should contribute to future research and policies that will improve data quality, utilization of available data sources, and the effectiveness of public health information systems. In other words, ICEPHI will ask questions about data and information sources that will lead to improved population health processes and outcomes. This is an important role for all public health informatics centers and professionals to perform within their agencies and communities.

> "The art and science of asking questions is the source of all knowledge."
>
> Thomas Berger

The work of the Indiana Center of Excellence in Public Health Informatics reported here is supported by a grant (1P01HK000077-01) from the U.S. Centers for Disease Control and Prevention. The contents of this paper are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention.

**Conflict of Interest**

The authors declare that they have no real or apparent conflicts of interest.
