program), proprietary systems (Granta, etc.) 12 100 TB (current), 500 Regular data Varied data and simulation MongoDB, GPFS, MapReduce and search Simulation-Driven TB within five years, added from results PyMatGen, FireWorks, that join simulation and Materials scalable key-value simulations VASP, ABINIT, experimental data Genomics and object store NWChem, databases needed BerkeleyGW, varied community codes 13 Imagery – hundreds Vectors Imagery, vector (various Geospatially enabled Closest point of approach, Large-Scale of terabytes; vector transmitted in formats such as shape files, RDBMS, Esri deviation from route, Geospatial data – tens of near real time KML, text streams) and many ArcServer, Geoserver point density over time, Analysis and gigabytes but billions object structures PCA and ICA Visualization of points 14 FMV – 30–60 frames Real time A few standard imagery or Custom software and Visualization as overlays Object per second at video formats tools including on a GIS, basic object Identification and 1080P traditional RDBMS and detection analytics and Tracking resolution; WALF – display tools integration with 1–10 frames per sophisticated situation second at 10,000 x awareness tools with data 10,000 full-color fusion resolution 15 Tens of terabytes to Much real-time, Text files, raw media, Hadoop, Accumulo Near real-time alerts Intelligence Data hundreds of imagery imagery, video, audio, (BigTable), Solr, NLP, based on patterns and Processing and petabytes, individual intelligence electronic data, Puppet (for baseline changes, link Analysis warfighters (first devices that data deployment and analysis, geospatial responders) would gather a petabyte security) and Storm; analysis, text analytics have at most one to of data in a few GIS (sentiment, entity hundreds of hours extraction, etc.) gigabytes 16 12 million patients, 0.5 – 1.5 million Broad variety of data from Teradata, PostgreSQL, Information retrieval Electronic more than 4 billion new real-time doctors, nurses, laboratories MongoDB, Hadoop, methods (tf-idf), NLP, Medical Record discrete clinical clinical and instruments Hive, R maximum likelihood Data observations, > 20 transactions estimators, Bayesian TB raw data added per day networks
program), proprietary systems (Granta, etc.) 12 Simulation-Driven Materials Genomics 100 TB (current), 500 TB within five years, scalable key-value and object store databases needed Regular data added from simulations Varied data and simulation results MongoDB, GPFS, PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes MapReduce and search that join simulation and experimental data 13 Large-Scale Geospatial Analysis and Visualization Imagery – hundreds of terabytes; vector data – tens of gigabytes but billions of points Vectors transmitted in near real time Imagery, vector (various formats such as shape files, KML, text streams) and many object structures Geospatially enabled RDBMS, Esri ArcServer, Geoserver Closest point of approach, deviation from route, point density over time, PCA and ICA 14 Object Identification and Tracking FMV – 30–60 frames per second at 1080P resolution; WALF – 1–10 frames per second at 10,000 x 10,000 full-color resolution Real time A few standard imagery or video formats Custom software and tools including traditional RDBMS and display tools Visualization as overlays on a GIS, basic object detection analytics and integration with sophisticated situation awareness tools with data fusion 15 Intelligence Data Processing and Analysis Tens of terabytes to hundreds of petabytes, individual warfighters (first responders) would have at most one to hundreds of gigabytes Much real-time, imagery intelligence devices that gather a petabyte of data in a few hours Text files, raw media, imagery, video, audio, electronic data, data Hadoop, Accumulo (BigTable), Solr, NLP, Puppet (for deployment and security) and Storm; GIS Near real-time alerts based on patterns and baseline changes, link analysis, geospatial analysis, text analytics (sentiment, entity extraction, etc.) 16 Electronic Medical Record Data 12 million patients, more than 4 billion discrete clinical observations, > 20 TB raw data 0.5 – 1.5 million new real-time clinical transactions added per day Broad variety of data from doctors, nurses, laboratories and instruments Teradata, PostgreSQL, MongoDB, Hadoop, Hive, R Information retrieval methods (tf-idf), NLP, maximum likelihood estimators, Bayesian networks