of high-resolution (six megapixels or more) images per second 27 500+ billion photos Over 500 million Images and metadata Hadoop MapReduce, Robust non-linear least Organizing on Facebook, 5+ images uploaded including EXIF (Exchangeable simple hand-written squares optimization billion photos on to Facebook each Image File) tags (focal multi-threaded tools problem, SVM Unstructured Flickr day distance, camera type, etc.) (Secure Shell [SSH] and Collections of sockets for Consumer Photos communication) 28 30 TB/year Near real-time Schema provided by social Hadoop IndexedHBase Anomaly detection, stream Truthy Twitter compressed data data storage, media data source; currently and HDFS; Hadoop, clustering, signal Data querying and using Twitter only; plans to Hive, Redis for data classification, online analysis expand, incorporating management; Python: learning; information Google+ and Facebook SciPy NumPy and MPI diffusion, clustering, for data analysis dynamic network visualization 29 Gigabytes (text, Data continuously So far mostly homogeneous XML technology, Pattern recognition (e.g., Crowd Sourcing in surveys, experiment updated and small data sets; expected traditional relational speech recognition, Humanities values) to hundreds analyzed large distributed databases automatic audio-visual of terabytes incrementally heterogeneous datasets analysis, cultural patterns), (multimedia) identification of structures (lexical units, linguistic rules, etc.) 30 Can be hundreds of Dynamic networks, Many types of networks Graph libraries (Galib, Network visualization CINET for gigabytes for a single network collection NetworkX); distributed Network Science network, growing workflow management networks and (Simfrastructure, methods databases, semantic web tools) 31 >900 million web Legacy evaluations Wide variety of data types PERL, Python, C/C++, Information extraction, NIST Information pages occupying 30 mostly focused on including textual Matlab, R development filtering, search, and Access Division TB of storage, 100 retrospective search/extraction, machine tools; create ground-up summarization; image and million tweets, 100 analytics, newer translation, speech test and measurement voice biometrics; speech million evaluations recognition, image and voice applications recognition and biometric focused on biometrics, object and understanding; machine
of high-resolution (six megapixels or more) images per second 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop MapReduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities Gigabytes (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of gigabytes for a single network, networks and methods Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management (Simfrastructure, databases, semantic web tools) Network visualization 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million biometric Legacy evaluations mostly focused on retrospective analytics, newer evaluations focused on Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice biometrics, object and PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement applications Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and understanding; machine