the science for supernovae. Variety While the imaging data is similar, the analysis for the (multiple datasets, 4 different types of cosmological measurements and mashup) comparisons to simulation data is quite different. Variability (rate of Weather and sky conditions can radically change change) both the quality and quantity of data. Big Data Science Veracity (Robustness Astrophysical data is a as the statistician’s nightmare (collection, Issues) both the uncertainties in a given measurement curation, change from night-to-night in addition to the cadence analysis, being highly unpredictable. Also, most all of the action) cosmological measurements are systematically limited, and thus understanding these as best possible is the highest priority for a given survey. Visualization Interactive speed of web UI on very large data sets is an ongoing challenge. Basic querying and browsing of data to find new transients as well as monitoring the quality of the survey is a must. Ability to download large amounts of data for offline analysis is another requirement of the system. Ability to combine both simulation and observational data is also necessary. Data Quality Understanding the systematic uncertainties in the observational data is a prerequisite to a successful cosmological measurement. Beating down the uncertainties in the simulation data to under this level is a huge challenge for future surveys. Data Types Cf. above on “Variety” Data Analytics Big Data Specific New statistical techniques for understanding the limitations in simulation Challenges (Gaps) data would be beneficial. Often it is the case where there is not enough computing time to generate all the simulations one wants and thus there is a reliance on emulators to bridge the gaps. Techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side. Big Data Specific Performing analysis on both the simulation and observational data Challenges in Mobility simultaneously. Security & Privacy No special challenges. Data is either public or requires standard login Requirements with password. Highlight issues for Parallel databases which could handle imaging data would be an generalizing this use interesting avenue for future research. case (e.g. for ref. architecture) More Information (URLs) , http://desi.lbl.gov, & http://www.darkenergysurvey.org Note: <additional comments> Astronomy and Physics: Analysis of LHC (Large Hadron Collider) Data
the science for supernovae. Variety (multiple datasets, mashup) While the imaging data is similar, the analysis for the 4 different types of cosmological measurements and comparisons to simulation data is quite different. Variability (rate of change) Weather and sky conditions can radically change both the quality and quantity of data. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Astrophysical data is a statistician’s nightmare as the both the uncertainties in a given measurement change from night-to-night in addition to the cadence being highly unpredictable. Also, most all of the cosmological measurements are systematically limited, and thus understanding these as best possible is the highest priority for a given survey. Visualization Interactive speed of web UI on very large data sets is an ongoing challenge. Basic querying and browsing of data to find new transients as well as monitoring the quality of the survey is a must. Ability to download large amounts of data for offline analysis is another requirement of the system. Ability to combine both simulation and observational data is also necessary. Data Quality Understanding the systematic uncertainties in the observational data is a prerequisite to a successful cosmological measurement. Beating down the uncertainties in the simulation data to under this level is a huge challenge for future surveys. Data Types Cf. above on “Variety” Data Analytics Big Data Specific Challenges (Gaps) New statistical techniques for understanding the limitations in simulation data would be beneficial. Often it is the case where there is not enough computing time to generate all the simulations one wants and thus there is a reliance on emulators to bridge the gaps. Techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side. Big Data Specific Challenges in Mobility Performing analysis on both the simulation and observational data simultaneously. Security & Privacy Requirements No special challenges. Data is either public or requires standard login with password. Highlight issues for generalizing this use case (e.g. for ref. architecture) Parallel databases which could handle imaging data would be an interesting avenue for future research. More Information (URLs) , http://desi.lbl.gov, & http://www.darkenergysurvey.org Note: <additional comments> Astronomy and Physics: Analysis of LHC (Large Hadron Collider) Data