6 15 TB presently, Currently Hadoop PDF documents and log files Hadoop, Scribe, Hive, Standard libraries for Mendeley growing about 1 TB batch jobs of social network and client Mahout, Python machine learning and per month scheduled daily, activities analytics, LDA, real-time reporting tools for recommended in aggregating readership future and social activities per document 7 Summer 2012 – 25 Media (video and Data vary from digital media Hadoop and Pig; Personalized Netflix Movie million subscribers, 4 properties) and to user rankings, user Cassandra; Teradata recommender systems Service million ratings per rankings continually profiles, and media using logistic/linear day, 3 million updated properties for content-based regression, elastic nets, searches per day, 1 recommendations matrix factorization, billion hours streamed clustering, LDA, in June 2012; Cloud association rules, storage – 2 petabytes gradient-boosted in June 2013 decision trees, and others; streaming video delivery 8 45 billion web pages Real-time updating Multiple media Map/Reduce + Bigtable; Crawling; searching, Web Search total, 500 million and real-time Dryad + Cosmos; including topic-based photos uploaded each responses to PageRank; final step searches; ranking; day, 100 hours of queries essentially a recommending video uploaded to recommender engine YouTube each minute 9 Terabytes up to Can be real time for Must work for all data Hadoop, Map/Reduce, Robust backup Business petabytes recent changes open source, and/or Continuity and vendor proprietary such Disaster as AWS, Google Cloud Recovery Within Services, and Microsoft a Cloud 10 — Needs to become Event-based — Distributed event Cargo Shipping real time, currently analysis identifying updated at events problems 11 500,000 material Ongoing increase in Many datasets with no National programs No broadly applicable Materials Data types in 1980s, much new materials standards (Japan, Korea, and analytics
6 Mendeley 15 TB presently, growing about 1 TB per month Currently Hadoop batch jobs scheduled daily, real-time recommended in future PDF documents and log files of social network and client activities Hadoop, Scribe, Hive, Mahout, Python Standard libraries for machine learning and analytics, LDA, reporting tools for aggregating readership and social activities per document 7 Netflix Movie Service Summer 2012 – 25 million subscribers, 4 million ratings per day, 3 million searches per day, 1 billion hours streamed in June 2012; Cloud storage – 2 petabytes in June 2013 Media (video and properties) and rankings continually updated Data vary from digital media to user rankings, user profiles, and media properties for content-based recommendations Hadoop and Pig; Cassandra; Teradata Personalized recommender systems using logistic/linear regression, elastic nets, matrix factorization, clustering, LDA, association rules, gradient-boosted decision trees, and others; streaming video delivery 8 Web Search 45 billion web pages total, 500 million photos uploaded each day, 100 hours of video uploaded to YouTube each minute Real-time updating and real-time responses to queries Multiple media Map/Reduce + Bigtable; Dryad + Cosmos; PageRank; final step essentially a recommender engine Crawling; searching, including topic-based searches; ranking; recommending 9 Business Continuity and Disaster Recovery Within a Cloud Terabytes up to petabytes Can be real time for recent changes Must work for all data Hadoop, Map/Reduce, open source, and/or vendor proprietary such as AWS, Google Cloud Services, and Microsoft Robust backup 10 Cargo Shipping — Needs to become real time, currently updated at events Event-based — Distributed event analysis identifying problems 11 Materials Data 500,000 material types in 1980s, much Ongoing increase in new materials Many datasets with no standards National programs (Japan, Korea, and No broadly applicable analytics