Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Analytics needs continued monitoring and improvement. Challenges (Gaps) Big Data Specific Mobile access important Challenges in Mobility Security & Privacy Need to preserve privacy for users and digital rights for media. Requirements Highlight issues for Recommender systems have features in common to e-commerce like generalizing this use Amazon. Streaming video has features in common with other content case (e.g. for ref. providing services like iTunes, Google Play, Pandora and Last.fm architecture) More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial Big Data) Requirements WG Use Case Template NBD(NIST Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and Owners of web information being searched; search engine companies; their roles and advertisers; users responsibilities Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize number of great “precision@10”; responses in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Compute(System) Large Clouds Solutions Storage Inverted Index not huge; crawled documents are petabytes of text rich media much more – Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. Final step essentially a recommender engine Big Data Data Source Distributed web sites Characteristics (distributed/centralized)
Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Mobile access important Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Recommender systems have features in common to e-commerce like Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial NBD(NIST Big Data) Requirements WG Use Case Template Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and their roles and responsibilities Owners of web information being searched; search engine companies; advertisers; users Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize “precision@10”; number of great responses in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites