47 200 TB (current), 500 Data analyzed Re-analysis datasets are MapReduce or the Data mining customized Atmospheric TB within 5 years incrementally inconsistent in format, like, SciDB or other for specific event types Turbulence resolution, semantics, and scientific database metadata; interpretation/analysis of each of these input streams into a common product 48 Up to 30 PB/year 42 GB/second Variety across simulation National Center for Need analytics next to Climate Studies from 15 end-to-end from simulations groups and between Atmospheric Research data storage simulations at observation and simulation (NCAR) PIO library and NERSC, more at utilities NCL and NCO, other HPC centers parallel NetCDF 49 — — From omics of the microbes PFLOWTran, postgres, Data mining, data quality DOE-BER in the soil to watershed HDF5, Akuna, NEWT, assessment, Subsurface hydro-biogeochemistry, etc. across Biogeochemistry from observation to datasets, reduced model simulation development, statistics, quality assessment, data fusion 50 — Streaming data Flux data merged with EddyPro, custom Data mining, data quality DOE-BER from ~150 towers biological, disturbance, and analysis software, R, assessment, AmeriFlux and in AmeriFlux and other ancillary data Python, neural correlation across FLUXNET over 500 towers networks, Matlab datasets, data Networks distributed assimilation, data globally collecting interpolation, statistics, flux quality assessment, data measurements fusion 51 4 TB/year for a city Streaming data Tuple-based: timeseries, R/Matlab, Weka, Forecasting models, Consumption with 1.4 million from millions of database rows; graph-based: Hadoop; GIS-based machine learning models, forecasting in sensors, such as Los sensors network topology, customer visualization time series analysis, Smart Grids Angeles connectivity; some semantic clustering, motif data for normalization detection, complex event processing, visual network analysis
47 Atmospheric Turbulence 200 TB (current), 500 TB within 5 years Data analyzed incrementally Re-analysis datasets are inconsistent in format, resolution, semantics, and metadata; interpretation/analysis of each of these input streams into a common product MapReduce or the like, SciDB or other scientific database Data mining customized for specific event types 48 Climate Studies Up to 30 PB/year from 15 end-to-end simulations at NERSC, more at other HPC centers 42 GB/second from simulations Variety across simulation groups and between observation and simulation National Center for Atmospheric Research (NCAR) PIO library and utilities NCL and NCO, parallel NetCDF Need analytics next to data storage 49 DOE-BER Subsurface Biogeochemistry — — From omics of the microbes in the soil to watershed hydro-biogeochemistry, from observation to simulation PFLOWTran, postgres, HDF5, Akuna, NEWT, etc. Data mining, data quality assessment, across datasets, reduced model development, statistics, quality assessment, data fusion 50 DOE-BER AmeriFlux and FLUXNET Networks — Streaming data from ~150 towers in AmeriFlux and over 500 towers distributed globally collecting flux measurements Flux data merged with biological, disturbance, and other ancillary data EddyPro, custom analysis software, R, Python, neural networks, Matlab Data mining, data quality assessment, across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion 51 Consumption forecasting in Smart Grids 4 TB/year for a city with 1.4 million sensors, such as Los Angeles Streaming data from millions of sensors Tuple-based: timeseries, database rows; graph-based: network topology, customer connectivity; some semantic data for normalization R/Matlab, Weka, Hadoop; GIS-based visualization Forecasting models, machine learning models, time series analysis, clustering, motif detection, complex event processing, visual network analysis correlation