Use Case Description 1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Compute(System) Large Clouds Solutions Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software Map/Reduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Data Source Distributed web sites Characteristics (distributed/centralized) Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity Data continually updated (e.g. real time) Variety Rich set of functions. After processing, data similar for (multiple datasets, each page (except for media types) mashup) Variability (rate of Average page has life of a few months change) Big Data Science Veracity (Robustness Exact results not essential but important to get main hubs (collection, curation, Issues) and authorities for search query analysis, Visualization Not important although page layout critical action) Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Search of “deep web” (information behind query front ends) Challenges (Gaps) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Mobile search must have similar interfaces/results Challenges in Mobility Security and Privacy Need to be sensitive to crawling restrictions. Avoid Spam results Requirements Highlight issues for Relation to Information retrieval such as search of scholarly works. generalizing this use case (e.g. for ref. architecture) More Information (URLs)
Use Case Description 1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software Map/Reduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity (e.g. real time) Data continually updated Variety (multiple datasets, mashup) Rich set of functions. After processing, data similar for each page (except for media types) Variability (rate of change) Average page has life of a few months Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page layout critical Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Challenges (Gaps) Search of “deep web” (information behind query front ends) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security and Privacy Requirements Need to be sensitive to crawling restrictions. Avoid Spam results Highlight issues for generalizing this use case (e.g. for ref. architecture) Relation to Information retrieval such as search of scholarly works. More Information (URLs)