Big Data Data Source Distributed – with replication/redundancy Characteristics (distributed/centralized) Volume (size) ~30TB/year compressed data Velocity (e.g. real time) Near real-time data storage, querying and analysis Variety (multiple Data schema provided by social media data source. datasets, mashup) Currently using Twitter only. We plan to expand incorporating Google+, Facebook Variability (rate of Continuous real-time data stream incoming from each change) source. Big Data Science Veracity (Robustness 99.99% uptime required for real-time data acquisition. (collection, curation, Issues, semantics) Service outages might corrupt data integrity and analysis, significance. action) Visualization Information diffusion, clustering, and dynamic network visualization capabilities already exist. Data Quality (syntax) Data structured in standardized formats, the overall quality is extremely high. We generate aggregated statistics; expand the features set, etc., generating derived data. Data Types Fully-structured data (JSON format) enriched with users meta-data, geo-locations, etc. Data Analytics Stream clustering: data are aggregated according to topics, meta-data and additional features, using ad hoc online clustering algorithms. Classification: using dimensional time series to generate, network features, users, geographical, content features, etc., we classify information produced on the platform. Anomaly detection: real-time identification of anomalous events (e.g., induced by exogenous factors). Online learning: applying machine learning/deep learning methods to real-time information diffusion patterns analysis, users profiling, etc. Big Data Specific Dealing with real-time analysis of large volume of data. Providing a scalable Challenges (Gaps) infrastructure to allocate resources, storage space, etc. on-demand if required by increasing data volume over time. Big Data Specific Implementing low-level data storage infrastructure features to guarantee efficient, Challenges in Mobility mobile access to data. Security and Privacy Twitter publicly releases data collected by our platform. Although, data-sources Requirements incorporate user meta-data (in general, not sufficient to uniquely identify individuals) therefore some policy for data storage security and privacy protection must be implemented. Highlight issues for Definition of high-level data schema to incorporate multiple data-sources providing generalizing this use similarly structured data. case (e.g. for ref. architecture) More Information (URLs)
Big Data Characteristics Data Source (distributed/centralized) Distributed – with replication/redundancy Volume (size) ~30TB/year compressed data Velocity (e.g. real time) Near real-time data storage, querying and analysis Variety (multiple datasets, mashup) Data schema provided by social media data source. Currently using Twitter only. We plan to expand incorporating Google+, Facebook Variability (rate of change) Continuous real-time data stream incoming from each source. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) 99.99% uptime required for real-time data acquisition. Service outages might corrupt data integrity and significance. Visualization Information diffusion, clustering, and dynamic network visualization capabilities already exist. Data Quality (syntax) Data structured in standardized formats, the overall quality is extremely high. We generate aggregated statistics; expand the features set, etc., generating derived data. Data Types Fully-structured data (JSON format) enriched with users meta-data, geo-locations, etc. Data Analytics Stream clustering: data are aggregated according to topics, meta-data and additional features, using ad hoc online clustering algorithms. Classification: using time series to generate, network features, users, geographical, content features, etc., we classify information produced on the platform. Anomaly detection: real-time identification of anomalous events (e.g., induced by exogenous factors). Online learning: applying machine learning/deep learning methods to real-time information diffusion patterns analysis, users profiling, etc. Big Data Specific Challenges (Gaps) Dealing with real-time analysis of large volume of data. Providing a scalable infrastructure to allocate resources, storage space, etc. on-demand if required by increasing data volume over time. Big Data Specific Challenges in Mobility Implementing low-level data storage infrastructure features to guarantee efficient, mobile access to data. Security and Privacy Requirements Twitter publicly releases data collected by our platform. Although, data-sources incorporate user meta-data (in general, not sufficient to uniquely identify individuals) therefore some policy for data storage security and privacy protection must be implemented. Highlight issues for generalizing this use case (e.g. for ref. architecture) Definition of high-level data schema to incorporate multiple data-sources providing similarly structured data. More Information (URLs) dimensional