workflow 43 Around 1 PB (current) Data taken in Raw data, images with final Matlab for custom raw Custom signal CReSIS Remote increasing by 50 to missions layer data used for science data processing, custom processing to produce Sensing 100 TB per mission, including test image processing radar images that are future expedition ≈1 analysis and then software, GIS as user analyzed by image PB each later batch interface processing to find layers processing 44 110 TB raw data and Data come from Image and annotation files ROI_PAC, GeoServer, Process raw data to get UAVSAR Data 40 TB processed, plus aircraft and so GDAL, images that are run Processing smaller samples incrementally tools; through image added, data moving to clouds processing tools and occasionally get accessed from GIS reprocessed: new processing methods or parameters 45 MERRA collection Periodic updates Many applications to SGE Univa Grid Engine Federation software NASA LARC/GSFC (below) represents every six months combine MERRA reanalysis Version 8.1, iRODS iRODS most of total data, data with other reanalyses Version 3.2 and/or 3.3, other smaller and observational data such IBM GPFS Version 3.4, collections as CERES Cloudera Version 46 480 TB from MERRA Increases at ≈1 Applications to combine Cloudera, iRODS, CAaaS MERRA Analytic TB/month MERRA reanalysis data with Amazon AWS Services other re-analyses and observational data 47 200 TB (current), 500 Data analyzed Re-analysis datasets are Map/Reduce or the like, Data mining customized Atmospheric TB within 5 years incrementally inconsistent in format, SciDB or other scientific for specific event types Turbulence resolution, semantics, and database metadata; interpretation/analysis of each of these input streams into a common product 48 Up to 30 PB/year 42 GB/second from Variety across simulation National Center for Need analytics next to Climate Studies from 15 end-to-end simulations groups and between Atmospheric Research data storage simulations at NERSC, observation and simulation (NCAR) PIO library and more at other HPC utilities NCL and NCO,
workflow 43 CReSIS Remote Sensing Around 1 PB (current) increasing by 50 to 100 TB per mission, future expedition ≈1 PB each Data taken in missions including test analysis and then later batch processing Raw data, images with final layer data used for science Matlab for custom raw data processing, custom image processing software, GIS as user interface Custom signal processing to produce radar images that are analyzed by image processing to find layers 44 UAVSAR Data Processing 110 TB raw data and 40 TB processed, plus smaller samples Data come from aircraft and so incrementally added, data occasionally get reprocessed: new processing methods or parameters Image and annotation files ROI_PAC, GeoServer, GDAL, tools; moving to clouds Process raw data to get images that are run through image processing tools and accessed from GIS 45 NASA LARC/GSFC iRODS MERRA collection (below) represents most of total data, other smaller collections Periodic updates every six months Many applications to combine MERRA reanalysis data with other reanalyses and observational data such as CERES SGE Univa Grid Engine Version 8.1, iRODS Version 3.2 and/or 3.3, IBM GPFS Version 3.4, Cloudera Version Federation software 46 MERRA Analytic Services 480 TB from MERRA Increases at ≈1 TB/month Applications to combine MERRA reanalysis data with other re-analyses and observational data Cloudera, iRODS, Amazon AWS CAaaS 47 Atmospheric Turbulence 200 TB (current), 500 TB within 5 years Data analyzed incrementally Re-analysis datasets are inconsistent in format, resolution, semantics, and metadata; interpretation/analysis of each of these input streams into a common product Map/Reduce or the like, SciDB or other scientific database Data mining customized for specific event types 48 Climate Studies Up to 30 PB/year from 15 end-to-end simulations at NERSC, more at other HPC 42 GB/second from simulations Variety across simulation groups and between observation and simulation National Center for Atmospheric Research (NCAR) PIO library and utilities NCL and NCO, Need analytics next to data storage