Software MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 Big Data Data Source Observational data will be generated by the Dark Characteristics (distributed/centralized) Energy Survey (DES) and the Zwicky Transient Factory in 2015 and by the Large Synoptic Sky Survey starting in 2019. Simulated data will generated at DOE supercomputing centers. Volume (size) DES: 4 PB, ZTF 1 PB/year, LSST 7 PB/year, Simulations > 10 PB in 2017 Velocity LSST: 20 TB/day (e.g. real time) Variety 1) Raw Data from sky surveys 2) Processed Image (multiple datasets, data 3) Simulation data mashup) Variability (rate of Observations are taken nightly; supporting change) simulations are run throughout the year, but data can be produced sporadically depending on access to resources Big Data Science Veracity (Robustness (collection, Issues) curation, Visualization and Interpretation of results from detailed simulations analysis, Analytics requires advanced analysis and visualization action) techniques and capabilities. Supercomputer I/O subsystem limitations are forcing researchers to analysis to replace post-processing explore “in-situ” methods. Data Quality Data Types Image data from observations must be reduced and compared with physical quantities derived from simulations. Simulated sky maps must be produced to match observational formats. Big Data Specific Storage, sharing, and analysis of 10s of PBs of observational and Challenges (Gaps) simulated data. Big Data Specific LSST will produce 20 TB of data per day. This must be archived and Challenges in Mobility made available to researchers world-wide. Security & Privacy Requirements Highlight issues for generalizing this use case (e.g. for ref. architecture) More Information (URLs) Note: <additional comments>
Software MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 Big Data Characteristics Data Source (distributed/centralized) Observational data will be generated by the Dark Energy Survey (DES) and the Zwicky Transient Factory in 2015 and by the Large Synoptic Sky Survey starting in 2019. Simulated data will generated at DOE supercomputing centers. Volume (size) DES: 4 PB, ZTF 1 PB/year, LSST 7 PB/year, Simulations > 10 PB in 2017 Velocity (e.g. real time) LSST: 20 TB/day Variety (multiple datasets, mashup) 1) Raw Data from sky surveys 2) Processed Image data 3) Simulation data Variability (rate of change) Observations are taken nightly; supporting simulations are run throughout the year, but data can be produced sporadically depending on access to resources Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization and Analytics Interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities. Supercomputer I/O subsystem limitations are forcing researchers to explore “in-situ” analysis to replace post-processing methods. Data Quality Data Types Image data from observations must be reduced and compared with physical quantities derived from simulations. Simulated sky maps must be produced to match observational formats. Big Data Specific Challenges (Gaps) Storage, sharing, and analysis of 10s of PBs of observational and simulated data. Big Data Specific Challenges in Mobility LSST will produce 20 TB of data per day. This must be archived and made available to researchers world-wide. Security & Privacy Requirements Highlight issues for generalizing this use case (e.g. for ref. architecture) More Information (URLs) Note: <additional comments>