process thousands of high-resolution (six megapixels or more) images per second 27 500+ billion photos Over 500 million Images and metadata Hadoop Map/Reduce, Robust non-linear least Organizing on Facebook, 5+ images uploaded including EXIF (Exchangeable simple hand-written squares optimization billion photos on to Facebook each Image File) tags (focal multi-threaded tools problem, SVM Unstructured Flickr day distance, camera type, etc.) (Secure Shell [SSH] and Collections of sockets for Consumer Photos communication) 28 30 TB/year Near real-time Schema provided by social Hadoop IndexedHBase Anomaly detection, stream Truthy Twitter compressed data data storage, media data source; currently and HDFS; Hadoop, clustering, signal Data querying and using Twitter only; plans to Hive, Redis for data classification, online analysis expand, incorporating management; Python: learning; information Google+ and Facebook SciPy NumPy and MPI diffusion, clustering, for data analysis dynamic network visualization 29 GBs (text, surveys, Data continuously So far mostly homogeneous XML technology, Pattern recognition (e.g., Crowd Sourcing in experiment values) to updated and small data sets; expected traditional relational speech recognition, Humanities hundreds of analyzed large distributed databases automatic audio-visual terabytes incrementally heterogeneous datasets analysis, cultural patterns), (multimedia) identification of structures (lexical units, linguistic rules, etc.) 30 Can be hundreds of Dynamic networks, Many types of networks Graph libraries (Galib, Network visualization CINET for GBs for a single network collection NetworkX); distributed Network Science network, 1,000 to growing workflow management 5,000 networks and (Simfrastructure, methods databases, semantic web tools) 31 >900 million web Legacy evaluations Wide variety of data types PERL, Python, C/C++, Information extraction, NIST Information pages occupying 30 mostly focused on including textual Matlab, R filtering, search, and Access Division TB of storage, 100 retrospective search/extraction, machine development tools; summarization; image and million tweets, 100 analytics, newer translation, speech create ground-up test voice biometrics; speech million evaluations recognition, image and voice and measurement recognition and
process thousands of high-resolution (six megapixels or more) images per second 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop Map/Reduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities GBs (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of GBs for a single network, 1,000 to 5,000 networks and methods Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management (Simfrastructure, databases, semantic web tools) Network visualization 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million Legacy evaluations mostly focused on retrospective analytics, newer evaluations Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and