networks and (Simfrastructure, methods databases, semantic web tools) 31 >900 million web Legacy evaluations Wide variety of data types PERL, Python, C/C++, Information extraction, NIST Information pages occupying 30 mostly focused on including textual Matlab, R filtering, search, and Access Division TB of storage, 100 retrospective search/extraction, machine development tools; summarization; image and million tweets, 100 analytics, newer translation, speech create ground-up test voice biometrics; speech million evaluations recognition, image and voice and measurement recognition and biometric focused on biometrics, object and applications understanding; machine images, hundreds of simulations of person recognition and translation; video thousands of real-time analytic tracking, document analysis, person/object detection partially challenges from human-computer dialogue, and tracking; event video clips, multiple data multimedia detection; terabytes of smaller streams search/extraction imagery/document fully ground-truthed matching; novelty test collections detection; structural semantic temporal analytics 32 Petabytes, hundreds Real time and Rich iRODS Supports general analysis DataNet (iRODS) of millions of files batch workflows 33 Small as metadata to Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, -- The Discinnet Big Data MySQL Process 34 A few terabytes Evolving in time Rich Database Data graph processing Semantic 35 50–400 GB per day, Continuous stream Images Octopus for Volume reconstruction, Light Source total ~400 TB of data, but Tomographic feature identification, etc. Beamlines analysis need not Reconstruction, Avizo be real time () and FIJI (a distribution of ImageJ) 36 ~100 TB total Nightly update Images, spectra, time series, Custom data Detection of rare events Catalina increasing by 0.1 TB runs processes in catalogs processing pipeline and relation to existing Transient a night accessing PBs real time and data analysis diverse data Survey of base astronomy software data, 30 TB a night
networks and methods (Simfrastructure, databases, semantic web tools) 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million biometric images, hundreds of thousands of partially video clips, terabytes of smaller fully ground-truthed test collections Legacy evaluations mostly focused on retrospective analytics, newer evaluations focused on simulations of real-time analytic challenges from multiple data streams Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice biometrics, object and person recognition and tracking, document analysis, human-computer dialogue, multimedia search/extraction PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement applications Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and understanding; machine translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50–400 GB per day, total ~400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo () and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ~100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data