generalizing this use Amazon. Streaming video has features in common with other content case (e.g. for ref. providing services like iTunes, Google Play, Pandora and Last.fm architecture) More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial: Web Search Big Data) Requirements WG Use Case Template Aug 11 2013 NBD(NIST Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and Owners of web information being searched; search engine companies; their roles and advertisers; users responsibilities Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize number of great responses “precision@10”; in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Compute(System) Large Clouds Solutions Storage Inverted Index not huge; crawled documents are petabytes of text rich media much more – Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Data Source Distributed web sites Characteristics (distributed/centralized) Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity Data continually updated (e.g. real time) Variety Rich set of functions. After processing, data (multiple datasets, similar for each page (except for media types) mashup) Variability (rate of Average page has life of a few months change) Big Data Science Veracity (Robustness Exact results not essential but important to get (collection, Issues) main hubs and authorities for search query curation, Visualization Not important although page layout critical
generalizing this use case (e.g. for ref. architecture) Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial: Web Search NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and their roles and responsibilities Owners of web information being searched; search engine companies; advertisers; users Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize “precision@10”; number of great responses in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity (e.g. real time) Data continually updated Variety (multiple datasets, mashup) Rich set of functions. After processing, data similar for each page (except for media types) Variability (rate of change) Average page has life of a few months Big Data Science (collection, curation, Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page layout critical