Solutions Storage SAN and Direct Storage Networking Fiber, 10 gigabit Ethernet, Infiniband 40 gigabit. Software Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Big Data Data Source Survey data, other government administrative data, Characteristics (distributed/centralized) web scrapped data, wireless data, e-transaction data, potentially social media data and positioning data from various sources. Volume (size) TBD Velocity TBD (e.g. real time) Variety Textual data as well as the traditionally defined (multiple datasets, strings and numerical fields. Data can be from mashup) multiple datasets mashed together for analytical use. Variability (rate of TBD. change) Big Data Science Veracity (Robustness Data must have high veracity and systems must be (collection, Issues, semantics) very robust. The semantic integrity of conceptual curation, metadata concerning what exactly is measured and analysis, the resulting limits of inference remain a challenge action) Visualization Data visualization is useful for data review, operational activity and general analysis. It continues to evolve. Data Quality (syntax) Data quality should be high and statistically checked for accuracy and reliability throughout the collection process. Data Types Textual data, pre-defined ASCII strings and numerical data Data Analytics Analytics are required to create reliable estimates using data from traditional survey sources, government administrative data sources and sources from the digital economy. Big Data Specific Improving analytic and modeling systems that provide reliable and Challenges (Gaps) robust statistical estimated using data from multiple sources, that are scientifically transparent and while providing confidentiality safeguards that are reliable and publically auditable. Big Data Specific Mobile access is important. Challenges in Mobility Security & Privacy All data must be both confidential and secure. All processes must be Requirements auditable for security and confidentiality as required by various legal statutes. Highlight issues for Statistical estimation that provide more detail, on a more near real time generalizing this use basis for less cost. The reliability of estimated statistics from such sources still must be evaluated. “mashed up”
Solutions Storage SAN and Direct Storage Networking Fiber, 10 gigabit Ethernet, Infiniband 40 gigabit. Software Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Big Data Characteristics Data Source (distributed/centralized) Survey data, other government administrative data, web scrapped data, wireless data, e-transaction data, potentially social media data and positioning data from various sources. Volume (size) TBD Velocity (e.g. real time) TBD Variety (multiple datasets, mashup) Textual data as well as the traditionally defined strings and numerical fields. Data can be from multiple datasets mashed together for analytical use. Variability (rate of change) TBD. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Data must have high veracity and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a challenge Visualization Data visualization is useful for data review, operational activity and general analysis. It continues to evolve. Data Quality (syntax) Data quality should be high and statistically checked for accuracy and reliability throughout the collection process. Data Types Textual data, pre-defined ASCII strings and numerical data Data Analytics Analytics are required to create reliable estimates using data from traditional survey sources, government administrative data sources and sources from the digital economy. Big Data Specific Challenges (Gaps) Improving analytic and modeling systems that provide reliable and robust statistical estimated using data from multiple sources, that are scientifically transparent and while providing confidentiality safeguards that are reliable and publically auditable. Big Data Specific Challenges in Mobility Mobile access is important. Security & Privacy Requirements All data must be both confidential and secure. All processes must be auditable for security and confidentiality as required by various legal statutes. Highlight issues for generalizing this use Statistical estimation that provide more detail, on a more near real time basis for less cost. The reliability of estimated statistics from such “mashed up” sources still must be evaluated.