Volume (size) The Mayo Clinic EHR dataset is a very large dataset containing over 5 million patients with thousands of properties each and many more that are derived from primary values. Velocity not real-time but updated periodically (e.g. real time) Variety Structured data, a patient has controlled vocabulary (multiple datasets, (CV) property values (demographics, diagnostic mashup) codes, medications, procedures, etc.) and continuous property values (lab tests, medication amounts, vitals, etc.). The number of property values could range from less than 100 (new patient) to more than 100,000 (long term patient) with typical patients composed of 100 CV values and 1000 continuous values. Most values are time based, i.e. a timestamp is recorded with the value at the time of observation. Variability (rate of Data will be updated or added during each patient change) visit. Big Data Science Veracity (Robustness Data are annotated based on domain ontologies or (collection, Issues) taxonomies. Semantics of data can vary from labs to curation, labs. analysis, Visualization no visualization action) Data Quality Provenance is important to trace the origins of the data and data quality Data Types text, and Continuous Numerical values Data Analytics Integrating data into semantic graph, using graph traverse to replace SQL join. Developing semantic graph mining algorithms to identify graph patterns, index graph, and search graph. Indexed Hbase. Custom code to develop new patient properties from stored data. Big Data Specific For individualized cohort, we will effectively be building a datamart for Challenges (Gaps) each patient since the critical properties and indices will be specific to each patient. Due to the number of patients, this becomes an impractical approach. Fundamentally, the paradigm changes from relational lookup to semantic graph traversal. Big Data Specific Physicians and patient may need access to this data on mobile platforms Challenges in Mobility Security & Privacy Health records or clinical research databases must be kept secure/private. Requirements Highlight issues for Data integration: continuous values, ontological annotation, taxonomy generalizing this use Graph Search: indexing and searching graph case (e.g. for ref. Validation: Statistical validation architecture) More Information (URLs) Note: <additional comments>
Volume (size) The Mayo Clinic EHR dataset is a very large dataset containing over 5 million patients with thousands of properties each and many more that are derived from primary values. Velocity (e.g. real time) not real-time but updated periodically Variety (multiple datasets, mashup) Structured data, a patient has controlled vocabulary (CV) property values (demographics, diagnostic codes, medications, procedures, etc.) and continuous property values (lab tests, medication amounts, vitals, etc.). The number of property values could range from less than 100 (new patient) to more than 100,000 (long term patient) with typical patients composed of 100 CV values and 1000 continuous values. Most values are time based, i.e. a timestamp is recorded with the value at the time of observation. Variability (rate of change) Data will be updated or added during each patient visit. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Data are annotated based on domain ontologies or taxonomies. Semantics of data can vary from labs to labs. Visualization no visualization Data Quality Provenance is important to trace the origins of the data and data quality Data Types text, and Continuous Numerical values Data Analytics Integrating data into semantic graph, using graph traverse to replace SQL join. Developing semantic graph mining algorithms to identify graph patterns, index graph, and search graph. Indexed Hbase. Custom code to develop new patient properties from stored data. Big Data Specific Challenges (Gaps) For individualized cohort, we will effectively be building a datamart for each patient since the critical properties and indices will be specific to each patient. Due to the number of patients, this becomes an impractical approach. Fundamentally, the paradigm changes from relational lookup to semantic graph traversal. Big Data Specific Challenges in Mobility Physicians and patient may need access to this data on mobile platforms Security & Privacy Requirements Health records or clinical research databases must be kept secure/private. Highlight issues for generalizing this use case (e.g. for ref. architecture) Data integration: continuous values, ontological annotation, taxonomy Graph Search: indexing and searching graph Validation: Statistical validation More Information (URLs) Note: <additional comments>