Use Case Requirements Summary Data Data Security & Lifecycle Use Case Sources Capabilities Consumer Privacy Management Others Transformation 1. large -- 1. large -- 1. Title 13 data 1. long-term -- document centralized preservation of data Census 2010 format from a storage (storage) as-is for 75 years and 2000 centralized 2. long-term storage preservation at the bit-level 3. curation process 1 including format transformation 4. access and analytics processing after 75 years 5. needs to make sure no data loss 1. distributed 1. crawl and 1. large data 1. high relevancy 1. security policy 1. pre-process for 1. mobile data sources index from storage and high recall virus scan search with NARA: 2. large data distributed data 2. various from search 2. file format similar Search, storage sources storages such as 2. high accuracy identification Retrieve, 3. bursty data 2. various NetApps, from 3. indexing from Preservation range from GB analytics Hitachi, categorization of 4. categorize records desktop to hundreds of processing Magnetic tapes records TB including 3. various 4. wide variety ranking, data storages such as of data formats categorization, NetApps, including detect PII data Hitachi, 2 unstructured 3. pre-processing Magnetic tapes and structured of data data 4. long-term 5. distributed preservation data sources in management fo different clouds large varied datasets. 5. hugh amount of data with high relevancy and recall. 1. data size 1. analytics are 1. software 1. data 1. improving 1. high veracity on 1. mobile approximately required for includes Hadoop, visualization for recommendation data and systems access Statistical one petabyte recommendation Spark, Hive, R, data review, systems that must be very robust. Survey systems, SAS, Mahout, operational reduce costs and The semantic Response continued Allegrograph, activity and improve quality integrity of monitoring and MySQL, Oracle, general analysis. while providing conceptual metadata general survey Storm, It continues to confidentiality concerning what improvement. BigMemory, evolve. safeguards that exactly is measured 3 Cassandra, Pig are reliable and and the resulting publically limits of inference auditable. remain a challenge 2. both confidential and secure all data. All processes must be auditable for security and
Use Case Requirements Summary Use Case Data Sources Transformation Capabilities Data Consumer Security & Privacy Lifecycle Management Others 1 Census 2010 and 2000 1. large document format from a centralized storage -- 1. large centralized storage (storage) -- 1. Title 13 data 1. long-term preservation of data as-is for 75 years 2. long-term preservation at the bit-level 3. curation process including format transformation 4. access and analytics processing after 75 years 5. needs to make sure no data loss -- 2 NARA: Search, Retrieve, Preservation 1. distributed data sources 2. large data storage 3. bursty data range from GB to hundreds of TB 4. wide variety of data formats including unstructured and structured data 5. distributed data sources in different clouds 1. crawl and index from distributed data sources 2. various analytics processing including ranking, data categorization, detect PII data 3. pre-processing of data 4. long-term preservation management fo large varied datasets. 5. hugh amount of data with high relevancy and recall. 1. large data storage 2. various storages such as NetApps, Hitachi, Magnetic tapes 1. high relevancy and high recall from search 2. high accuracy from categorization of records 3. various storages such as NetApps, Hitachi, Magnetic tapes 1. security policy 1. pre-process for virus scan 2. file format identification 3. indexing 4. categorize records 1. mobile search with similar from desktop 3 Statistical Survey Response 1. data size approximately one petabyte 1. analytics are required for recommendation systems, continued monitoring and general survey improvement. 1. software includes Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. data visualization for data review, operational activity and general analysis. It continues to evolve. 1. improving recommendation systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publically auditable. 2. both confidential and secure all data. All processes must be auditable for security and 1. high veracity on data and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a challenge 1. mobile access