biometric focused on biometrics, object and person applications understanding; machine images, hundreds of simulations of recognition and tracking, translation; video thousands of partially analytic document analysis, person/object detection ground-truthed video challenges from dialogue, and tracking; event clips, terabytes of multiple data multimedia search/extraction detection; smaller fully streams imagery/document test matching; novelty collections detection; structural semantic temporal analytics 32 Petabytes, hundreds Real time and Rich iRODS Supports general analysis DataNet (iRODS) of millions of files batch workflows 33 Small as metadata to Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, -- The Discinnet Big Data MySQL Process 34 A few terabytes Evolving in time Rich Database Data graph processing Semantic 35 50 to 400 GB per day, Continuous stream Images Octopus for Volume reconstruction, Light Source total ≈400 TB of data, but Tomographic feature identification, etc. Beamlines analysis need not Reconstruction, Avizo be real time () and FIJI (a distribution of ImageJ) 36 ≈100 TB total Nightly update Images, spectra, time series, Custom data Detection of rare events Catalina increasing by 0.1 TB a runs processes in catalogs processing pipeline and relation to existing Time Transient night accessing PBs of real time and data analysis diverse data Survey base astronomy data, software 30 TB a night from successor LSST in 2020s 37 Several petabytes Analysis done in Image and simulation data MPI, FFTW, viz New analytics needed to DOE Extreme from Dark Energy batch mode with packages, numpy, analyze simulation results Data from Survey and Zwicky data from Boost, OpenMP, Cosmological Sky Transient Factory, observations and ScaLAPCK, PSQL and Survey simulations > 10 PB simulations MySQL databases, updated daily Eigen, cfitsio,
biometric images, hundreds of thousands of partially ground-truthed video clips, terabytes of smaller fully test collections focused on simulations of analytic challenges from multiple data streams biometrics, object and person recognition and tracking, document analysis, dialogue, multimedia search/extraction applications understanding; machine translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50 to 400 GB per day, total ≈400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo () and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ≈100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night from successor LSST in 2020s Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data 37 DOE Extreme Data from Cosmological Sky Survey Several petabytes from Dark Energy Survey and Zwicky Transient Factory, simulations > 10 PB Analysis done in batch mode with data from observations and simulations updated daily Image and simulation data MPI, FFTW, viz packages, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, New analytics needed to analyze simulation results Time