Software Custom data processing pipeline and data analysis software, operating under Linux. Some archives on Windows machines, running a MS SQL server databases. Big Data Data Source Distributed: Characteristics (distributed/centralized) Survey data from 3 (soon more?) telescopes Archival data from a variety of resources connected through the VO framework Follow-up observations from separate telescopes Volume (size) The survey generates up to ~ 0.1 TB per clear night; ~ 100 TB in current data holdings. observational data amount to no more than a few % of that. Archival data in external (VO-connected) archives are in PBs, but only a minor fraction is used. Velocity Up to ~ 0.1 TB / night of the raw survey data. (e.g. real time) Variety The primary survey data in the form of images, (multiple datasets, processed to catalogs of sources (db tables), and mashup) time series for individual objects (light curves). Follow-up observations consist of images and spectra. Archival data from the VO data grid include all of the above, from a wide variety of sources and different wavelengths. Variability (rate of Daily data traffic fluctuates from ~ 0.01 to ~ 0.1 change) TB / day, not including major data transfers between the principal archives (Caltech, UA, and IUCAA). Big Data Science Veracity (Robustness A variety of automated and human inspection (collection, curation, Issues, semantics) quality control mechanisms is implemented at all analysis, stages of the process. action) Visualization Standard image display and data plotting packages are used. We are exploring visualization mechanisms for highly dimensional data parameter spaces. Data Quality (syntax) It varies, depending on the observing conditions, and it is evaluated automatically: error bars are estimated for all relevant quantities. Data Types Images, spectra, time series, catalogs. Data Analytics A wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself. Big Data Specific Development of machine learning tools for data exploration, and in Challenges (Gaps) particular for an automated, real-time classification of transient events, given the data sparsity and heterogeneity. Effective visualization of hyper-dimensional parameter spaces is a major challenge for all of us. Big Data Specific Not a significant limitation at this time. Challenges in Mobility Security & Privacy None.
Software Custom data processing pipeline and data analysis software, operating under Linux. Some archives on Windows machines, running a MS SQL server databases. Big Data Characteristics Data Source (distributed/centralized) Distributed: Survey data from 3 (soon more?) telescopes Archival data from a variety of resources connected through the VO framework Follow-up observations from separate telescopes Volume (size) The survey generates up to ~ 0.1 TB per clear night; ~ 100 TB in current data holdings. observational data amount to no more than a few % of that. Archival data in external (VO-connected) archives are in PBs, but only a minor fraction is used. Velocity (e.g. real time) Up to ~ 0.1 TB / night of the raw survey data. Variety (multiple datasets, mashup) The primary survey data in the form of images, processed to catalogs of sources (db tables), and time series for individual objects (light curves). Follow-up observations consist of images and spectra. Archival data from the VO data grid include all of the above, from a wide variety of sources and different wavelengths. Variability (rate of change) Daily data traffic fluctuates from ~ 0.01 to ~ 0.1 TB / day, not including major data transfers between the principal archives (Caltech, UA, and IUCAA). Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) A variety of automated and human inspection quality control mechanisms is implemented at all stages of the process. Visualization Standard image display and data plotting packages are used. We are exploring visualization mechanisms for highly dimensional data parameter spaces. Data Quality (syntax) It varies, depending on the observing conditions, and it is evaluated automatically: error bars are estimated for all relevant quantities. Data Types Images, spectra, time series, catalogs. Data Analytics A wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself. Big Data Specific Challenges (Gaps) Development of machine learning tools for data exploration, and in particular for an automated, real-time classification of transient events, given the data sparsity and heterogeneity. Effective visualization of hyper-dimensional parameter spaces is a major challenge for all of us. Big Data Specific Challenges in Mobility Not a significant limitation at this time. Security & Privacy None.