Current Compute(System) Linux servers Solutions Storage NetApps, Hitachi, Magnetic tapes. Networking Software Custom software, commercial search products, commercial databases. Big Data Data Source Distributed data sources from federal agencies. Characteristics (distributed/centralized) Current solution requires transfer of those data to a centralized storage. In the future, those data sources may reside in different Cloud environments. Volume (size) Hundred of Terabytes, and growing. Velocity Input rate is relatively low compared to other use (e.g. real time) cases, but the trend is bursty. That is the data can arrive in batches of size ranging from GB to hundreds of TB. Variety Variety data types, unstructured and structured (multiple datasets, data: textual documents, emails, photos, scanned mashup) documents, multimedia, social networks, web sites, databases, etc. Variety of application domains, since records come from different agencies. Data come from variety of repositories, some of which can be cloud-based in the future. Variability (rate of Rate can change especially if input sources are change) variable, some having audio, video more, some more text, and other images, etc. Big Data Science Veracity (Robustness Search results should have high relevancy and high (collection, Issues) recall. curation, Categorization of records should be highly accurate. analysis, action) Visualization TBD Data Quality Unknown. Data Types Variety data types: textual documents, emails, photos, scanned documents, multimedia, databases, etc. Data Analytics Crawl/index; search; ranking; predictive search. Data categorization (sensitive, confidential, etc.) PII data detection and flagging. Big Data Specific Perform pre-processing and manage for long-term of large and varied Challenges (Gaps) data. Search huge amount of data. Ensure high relevancy and recall. Data sources may be distributed in different clouds in future.
Current Solutions Compute(System) Linux servers Storage NetApps, Hitachi, Magnetic tapes. Networking Software Custom software, commercial search products, commercial databases. Big Data Characteristics Data Source (distributed/centralized) Distributed data sources from federal agencies. Current solution requires transfer of those data to a centralized storage. In the future, those data sources may reside in different Cloud environments. Volume (size) Hundred of Terabytes, and growing. Velocity (e.g. real time) Input rate is relatively low compared to other use cases, but the trend is bursty. That is the data can arrive in batches of size ranging from GB to hundreds of TB. Variety (multiple datasets, mashup) Variety data types, unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Variety of application domains, since records come from different agencies. Data come from variety of repositories, some of which can be cloud-based in the future. Variability (rate of change) Rate can change especially if input sources are variable, some having audio, video more, some more text, and other images, etc. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Search results should have high relevancy and high recall. Categorization of records should be highly accurate. Visualization TBD Data Quality Unknown. Data Types Variety data types: textual documents, emails, photos, scanned documents, multimedia, databases, etc. Data Analytics Crawl/index; search; ranking; predictive search. Data categorization (sensitive, confidential, etc.) PII data detection and flagging. Big Data Specific Challenges (Gaps) Perform pre-processing and manage for long-term of large and varied data. Search huge amount of data. Ensure high relevancy and recall. Data sources may be distributed in different clouds in future.