images, hundreds of simulations of person recognition and translation; video thousands of analytic tracking, document analysis, person/object detection partially challenges from human-computer dialogue, and tracking; event video clips, multiple data multimedia detection; terabytes of smaller streams search/extraction imagery/document fully ground-truthed matching; novelty test collections detection; structural semantic temporal analytics 32 Petabytes, hundreds Real time and Rich iRODS Supports general analysis DataNet (iRODS) of millions of files batch workflows 33 Small as metadata to Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, -- The Discinnet Big Data MySQL Process 34 A few terabytes Evolving in time Rich Database Data graph processing Semantic 35 50–400 GB per day, Continuous stream Images Octopus for Volume reconstruction, Light Source total ~400 TB of data, but Tomographic feature identification, etc. Beamlines analysis need not Reconstruction, Avizo be real time (http://vsg3d.com) and FIJI (a distribution of ImageJ) 36 ~100 TB total Nightly update Images, spectra, time series, Custom data processing Detection of rare events Catalina increasing by 0.1 TB runs processes in catalogs pipeline and data and relation to existing Transient a night accessing PBs real time analysis software diverse data Survey of base astronomy data, 30 TB a night from successor LSST in 2020s 37 Several petabytes Analysis done in Image and simulation data MPI, FFTW, viz New analytics needed to DOE Extreme from Dark Energy batch mode with packages, numpy, Boost, analyze simulation results Data from Survey and Zwicky data from OpenMP, ScaLAPCK, Cosmological Sky Transient Factory, observations and PSQL and MySQL Survey simulations > 10 PB simulations databases, Eigen, cfitsio, updated daily astrometry.net, and
images, hundreds of thousands of partially video clips, terabytes of smaller fully ground-truthed test collections simulations of analytic challenges from multiple data streams person recognition and tracking, document analysis, human-computer dialogue, multimedia search/extraction translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50–400 GB per day, total ~400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo (http://vsg3d.com) and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ~100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night from successor LSST in 2020s Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data 37 DOE Extreme Data from Cosmological Sky Survey Several petabytes from Dark Energy Survey and Zwicky Transient Factory, simulations > 10 PB Analysis done in batch mode with data from observations and simulations updated daily Image and simulation data MPI, FFTW, viz packages, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and New analytics needed to analyze simulation results