Note: <additional comments> Big Data) Requirements WG Use Case Template Aug 11 2013 NBD(NIST Use Case Title Truthy: Information diffusion research from Twitter Data Vertical (area) Scientific Research: Complex Networks and Systems research Author/Company/Email Filippo Menczer, Indiana University, ; Alessandro Flammini, Indiana University, ; Emilio Ferrara, Indiana University, ; Actors/Stakeholders and Research funded by NFS, DARPA, and McDonnel Foundation. their roles and responsibilities Goals Understanding how communication spreads on socio-technical networks. Detecting potentially harmful information spread at the early stage (e.g., deceiving messages, orchestrated campaigns, untrustworthy information, etc.) Use Case Description (1) Acquisition and storage of a large volume of continuous streaming data from Twitter (~100 million messages per day, ~500GB data/day increasing over time); (2) near real-time analysis of such data, for anomaly detection, stream clustering, signal classification and (3) data retrieval, big data visualization, data-interactive Web interfaces, public API for data querying. Current Compute(System) Current: in-house cluster hosted by Indiana Solutions University. Critical requirement: large cluster for data storage, manipulation, querying and analysis. Storage Current: Raw data stored in large compressed flat files, since August 2010. Need to move towards Hadoop/IndexedHBase & HDFS distributed storage. Redis as a in-memory database as a buffer for analysis. Networking 10GB/Infiniband required. Software Hadoop, Hive, Redis for data management. Python/SciPy/NumPy/MPI for data analysis. Big Data Data Source Distributed with replication/redundancy – Characteristics (distributed/centralized) Volume (size) ~30TB/year compressed data Velocity (e.g. real time) Near real-time data storage, querying & analysis Variety (multiple Data schema provided by social media data source. datasets, mashup) Currently using Twitter only. We plan to expand incorporating Google+, Facebook Variability (rate of Continuous real-time data-stream incoming from change) each source. Big Data Science Veracity (Robustness 99.99% uptime required for real-time data (collection, Issues, semantics) acquisition. Service outages might corrupt data curation, integrity and significance. analysis, Visualization Information diffusion, clustering, and dynamic action) network visualization capabilities already exist.
Note: <additional comments> NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Truthy: Information diffusion research from Twitter Data Vertical (area) Scientific Research: Complex Networks and Systems research Author/Company/Email Filippo Menczer, Indiana University, ; Alessandro Flammini, Indiana University, ; Emilio Ferrara, Indiana University, ; Actors/Stakeholders and their roles and responsibilities Research funded by NFS, DARPA, and McDonnel Foundation. Goals Understanding how communication spreads on socio-technical networks. Detecting potentially harmful information spread at the early stage (e.g., deceiving messages, orchestrated campaigns, untrustworthy information, etc.) Use Case Description (1) Acquisition and storage of a large volume of continuous streaming data from Twitter (~100 million messages per day, ~500GB data/day increasing over time); (2) near real-time analysis of such data, for anomaly detection, stream clustering, signal classification and (3) data retrieval, big data visualization, data-interactive Web interfaces, public API for data querying. Current Solutions Compute(System) Current: in-house cluster hosted by Indiana University. Critical requirement: large cluster for data storage, manipulation, querying and analysis. Storage Current: Raw data stored in large compressed flat files, since August 2010. Need to move towards Hadoop/IndexedHBase & HDFS distributed storage. Redis as a in-memory database as a buffer for analysis. Networking 10GB/Infiniband required. Software Hadoop, Hive, Redis for data management. Python/SciPy/NumPy/MPI for data analysis. Big Data Characteristics Data Source (distributed/centralized) Distributed – with replication/redundancy Volume (size) ~30TB/year compressed data Velocity (e.g. real time) Near real-time data storage, querying & analysis Variety (multiple datasets, mashup) Data schema provided by social media data source. Currently using Twitter only. We plan to expand incorporating Google+, Facebook Variability (rate of change) Continuous real-time data-stream incoming from each source. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) 99.99% uptime required for real-time data acquisition. Service outages might corrupt data integrity and significance. Visualization Information diffusion, clustering, and dynamic network visualization capabilities already exist.