natural or industrial disaster 26 Current datasets Much faster than Neural net very In-house GPU kernels Small degree of batch Large-Scale Deep typically 1 to 10 TB, real-time heterogeneous as it learns and MPI-based statistical pre-processing, Learning possibly 100 million processing; for many different features communication all other data analysis images to train a autonomous developed by performed by the learning self-driving car driving, need to Stanford, C++/Python algorithm itself process thousands source of high-resolution (six megapixels or more) images per second 27 500+ billion photos Over 500 million Images and metadata Hadoop MapReduce, Robust non-linear least Organizing on Facebook, 5+ images uploaded including EXIF (Exchangeable simple hand-written squares optimization billion photos on to Facebook each Image File) tags (focal multi-threaded tools problem, SVM Unstructured Flickr day distance, camera type, etc.) (Secure Shell [SSH] Collections of and sockets for Consumer Photos communication) 28 30 TB/year Near real-time Schema provided by social Hadoop IndexedHBase Anomaly detection, Truthy Twitter compressed data data storage, media data source; currently and HDFS; Hadoop, stream clustering, signal Data querying and using Twitter only; plans to Hive, Redis for data classification, online analysis expand, incorporating management; Python: learning; information Google+ and Facebook SciPy NumPy and MPI diffusion, clustering, for data analysis dynamic network visualization 29 Gigabytes (text, Data continuously So far mostly homogeneous XML technology, Pattern recognition (e.g., Crowd Sourcing in surveys, experiment updated and small data sets; expected traditional relational speech recognition, Humanities values) to hundreds analyzed large distributed databases automatic audio-visual of terabytes incrementally heterogeneous datasets analysis, cultural patterns), (multimedia) identification of structures (lexical units, linguistic rules, etc.) 30 Can be hundreds of Dynamic Many types of networks Graph libraries (Galib, Network visualization CINET for gigabytes for a single networks, network NetworkX); Network Science network, collection growing distributed workflow management
natural or industrial disaster 26 Large-Scale Deep Learning Current datasets typically 1 to 10 TB, possibly 100 million images to train a self-driving car Much faster than real-time processing; for autonomous driving, need to process thousands of high-resolution (six megapixels or more) images per second Neural net very heterogeneous as it learns many different features In-house GPU kernels and MPI-based communication developed by Stanford, C++/Python source Small degree of batch statistical pre-processing, all other data analysis performed by the learning algorithm itself 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop MapReduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities Gigabytes (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of gigabytes for a single network, Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management Network visualization