Software Hadoop, Hive, R. Unix-based. Big Data Data Source Clinical data from more than 1,100 discrete logical, Characteristics (distributed/centralized) operational healthcare sources in the Indiana Network for Patient Care (INPC) the nation's largest and health information exchange. Volume (size) More than 12 million patients, more than 4 billion discrete clinical observations. > 20 TB raw data. Velocity Between 500,000 and 1.5 million new real-time clinical (e.g. real time) transactions added per day. Variety We integrate a broad variety of clinical datasets from (multiple datasets, multiple sources: free text provider notes; inpatient, mashup) outpatient, laboratory, and emergency department encounters; chromosome and molecular pathology; chemistry studies; cardiology studies; hematology studies; microbiology studies; neurology studies; provider notes; referral labs; serology studies; surgical pathology and cytology, blood bank, and toxicology studies. Variability (rate of Data from clinical systems evolve over time because the change) clinical and biological concept space is constantly evolving: new scientific discoveries lead to new disease entities, new diagnostic modalities, and new disease management approaches. These in turn lead to new clinical concepts, which drive the evolution of health concept ontologies, encoded in highly variable fashion. Big Data Science Veracity (Robustness Data from each clinical source are commonly gathered (collection, curation, Issues, semantics) using different methods and representations, yielding analysis, substantial heterogeneity. This leads to systematic errors action) and bias requiring robust methods for creating semantic interoperability. Visualization Inbound data volume, accuracy, and completeness must be monitored on a routine basis using focus visualization methods. Intrinsic informational characteristics of data sources must be visualized to identify unexpected trends. Data Quality (syntax) A central barrier to leveraging electronic medical record data is the highly variable and unique local names and codes for the same clinical test or measurement performed at different institutions. When integrating many data sources, mapping local terms to a common standardized concept using a combination of probabilistic and heuristic classification methods is necessary. Data Types Wide variety of clinical data types including numeric, structured numeric, free-text, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video).
Software Hadoop, Hive, R. Unix-based. Big Data Characteristics Data Source (distributed/centralized) Clinical data from more than 1,100 discrete logical, operational healthcare sources in the Indiana Network for Patient Care (INPC) the nation's largest and health information exchange. Volume (size) More than 12 million patients, more than 4 billion discrete clinical observations. > 20 TB raw data. Velocity (e.g. real time) Between 500,000 and 1.5 million new real-time clinical transactions added per day. Variety (multiple datasets, mashup) We integrate a broad variety of clinical datasets from multiple sources: free text provider notes; inpatient, outpatient, laboratory, and emergency department encounters; chromosome and molecular pathology; chemistry studies; cardiology studies; hematology studies; microbiology studies; neurology studies; provider notes; referral labs; serology studies; surgical pathology and cytology, blood bank, and toxicology studies. Variability (rate of change) Data from clinical systems evolve over time because the clinical and biological concept space is constantly evolving: new scientific discoveries lead to new disease entities, new diagnostic modalities, and new disease management approaches. These in turn lead to new clinical concepts, which drive the evolution of health concept ontologies, encoded in highly variable fashion. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Data from each clinical source are commonly gathered using different methods and representations, yielding substantial heterogeneity. This leads to systematic errors and bias requiring robust methods for creating semantic interoperability. Visualization Inbound data volume, accuracy, and completeness must be monitored on a routine basis using focus visualization methods. Intrinsic informational characteristics of data sources must be visualized to identify unexpected trends. Data Quality (syntax) A central barrier to leveraging electronic medical record data is the highly variable and unique local names and codes for the same clinical test or measurement performed at different institutions. When integrating many data sources, mapping local terms to a common standardized concept using a combination of probabilistic and heuristic classification methods is necessary. Data Types Wide variety of clinical data types including numeric, structured numeric, free-text, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video).