astrometry.net, and Minuit2 38 Petabytes of data 400 images of 1 GB Images Linux cluster, Oracle Machine learning to find Large Survey Data from Dark Energy in size per night RDBMS server, optical transients, Cholesky for Cosmology Survey Postgres PSQL, large decomposition for memory machines, thousands of simulations standard Linux with matrices of order 1 interactive hosts, million on a side and GPFS; for simulations, parallel image storage HPC resources; standard astrophysics reduction software as well as Perl/Python wrapper scripts 39 15 PB of data Data updated Different format for each Grid-based Sophisticated specialized Particle Physics at (experiment and continuously with stage in analysis but data environment with over data analysis code followed LHC Monte Carlo sophisticated uniform within each stage 350,000 cores running by basic exploratory combined) per year selection and simultaneously statistics (histogram) with test analysis but all complex detector analyzed efficiency corrections "properly" offline 40 Eventually 120 PB of Data updated Different format for each DIRAC Grid software Sophisticated specialized Belle II High Monte Carlo and continuously with stage in analysis but data data analysis code followed Energy Physics observational data sophisticated uniform within each stage by basic exploratory Experiment selection and statistics (histogram) with test analysis but all complex detector analyzed efficiency corrections "properly" offline 41 Terabytes/year Data updated Big data uniform Custom analysis based Pattern recognition, EISCAT 3D (current), 40 PB/year continuously with on flat file data storage demanding correlation incoherent scatter starting ≈2022 real-time test routines, high-level radar system analysis and batch parameter extraction full analysis 42 Low volume (apart Mainly real-time Six separate projects with R and Python Data assimilation, ENVRI from EISCAT 3D given data streams common architecture for (Matplotlib) for (statistical) analysis, data Environmental above), one system infrastructure, data very visualization, custom mining, data extraction,
astrometry.net, and Minuit2 38 Large Survey Data for Cosmology Petabytes of data from Dark Energy Survey 400 images of 1 GB in size per night Images Linux cluster, Oracle RDBMS server, Postgres PSQL, large memory machines, standard Linux interactive hosts, GPFS; for simulations, HPC resources; standard astrophysics reduction software as well as Perl/Python wrapper scripts Machine learning to find optical transients, Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side and parallel image storage 39 Particle Physics at LHC 15 PB of data (experiment and Monte Carlo combined) per year Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage Grid-based environment with over 350,000 cores running simultaneously Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 40 Belle II High Energy Physics Experiment Eventually 120 PB of Monte Carlo and observational data Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage DIRAC Grid software Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 41 EISCAT 3D incoherent scatter radar system Terabytes/year (current), 40 PB/year starting ≈2022 Data updated continuously with real-time test analysis and batch full analysis Big data uniform Custom analysis based on flat file data storage Pattern recognition, demanding correlation routines, high-level parameter extraction 42 ENVRI Environmental Low volume (apart from EISCAT 3D given above), one system Mainly real-time data streams Six separate projects with common architecture for infrastructure, data very R and Python (Matplotlib) for visualization, custom Data assimilation, (statistical) analysis, data mining, data extraction,