learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow up resources 1. ~1 PB per 1. interpretation 1. MPI, 1. interpretation -- -- -- year becoming of results from OpenMP, C, of results using DOE 7PB a year detailed C++, F90, advanced Extreme observational simulations FFTW, viz visualization Data from datal requires packages, techniques and advanced python, FFTW, capabilities Sky Survey analysis and numpy, Boost, visualization OpenMP, techniques and ScaLAPCK, 37 capabilities PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. supercomputer I/O subsystem limitations must be addressed 1. 20TB data 1. analysis on 1. standard -- -- 1. links between -- per day both the astrophysics remote telescopes Large simulation and reduction and central analysis Survey Data observational software as well sites for data as Perl/Python Cosmology simultaneously wrapper scripts 2. techniques for 2. Oracle 38 handling RDBMS, Cholesky Postgres psql, as decompostion for well as GPFS thousands of and Lustre file simulations with systems and tape matricies of archives order 1M on a 3. parallel image side storage 1. real time 1. experimental 1. legacy 1. histograms and 1. data protection 1. data quality on -- data from data from computing model fits complex apparatus Particle Accelerator ALICE, ATLAS, infrastructure (visual) Physics at and Analysis CMS, LHb (computing 39 LHC instruments 2. histograms, nodes) 2. scatter-plots with 2. distributed model fits cached files data 3. Monte-Carlo (storage) collection computations 3. object
learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow up resources 37 DOE Extreme Data from Sky Survey 1. ~1 PB per year becoming 7PB a year observational datal 1. interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. supercomputer I/O subsystem limitations must be addressed 1. interpretation of results using advanced visualization techniques and capabilities -- -- -- 38 Large Survey Data for Cosmology 1. 20TB data per day 1. analysis on both the simulation and observational data simultaneously 2. techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side 1. standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, as well as GPFS and Lustre file systems and tape archives 3. parallel image storage -- -- 1. links between remote telescopes and central analysis sites -- 39 Particle Physics at LHC 1. real time data from Accelerator and Analysis instruments 2. data collection 1. experimental data from ALICE, ATLAS, CMS, LHb 2. histograms, scatter-plots with model fits 3. Monte-Carlo computations 1. legacy computing infrastructure (computing nodes) 2. distributed cached files (storage) 3. object 1. histograms and model fits (visual) 1. data protection 1. data quality on complex apparatus --