Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. Final step essentially a recommender engine Big Data Data Source Distributed web sites Characteristics (distributed/centralized) Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity Data continually updated (e.g. real time) Variety Rich set of functions. After processing, data (multiple datasets, similar for each page (except for media types) mashup) Variability (rate of Average page has life of a few months change) Big Data Science Veracity (Robustness Exact results not essential but important to get (collection, Issues) main hubs and authorities for search query curation, Visualization Not important although page lay out critical analysis, Data Quality A lot of duplication and spam action) Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Search of behind query front ends) “deep web” (information Challenges (Gaps) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Mobile search must have similar interfaces/results Challenges in Mobility Security & Privacy Need to be sensitive to crawling restrictions. Avoid Spam results Requirements Highlight issues for Relation to Information retrieval such as search of scholarly works. generalizing this use case (e.g. for ref. architecture) More Information (URLs) http://www.slideshare.net/kleinerperkins/kpcb-internet-trends-2013 http://www.ifis.cs.tu-bs.de/teaching/ss-11/irws http://www.worldwidewebsize.com/ Note: <additional comments> Big Data) Requirements WG Use Case Template Aug 11 2013 NBD(NIST Use Case Title Netflix Movie Service Vertical (area) Commercial Cloud Consumer Services
Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity (e.g. real time) Data continually updated Variety (multiple datasets, mashup) Rich set of functions. After processing, data similar for each page (except for media types) Variability (rate of change) Average page has life of a few months Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page lay out critical Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Challenges (Gaps) Search of “deep web” (information behind query front ends) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security & Privacy Requirements Need to be sensitive to crawling restrictions. Avoid Spam results Highlight issues for generalizing this use case (e.g. for ref. architecture) Relation to Information retrieval such as search of scholarly works. More Information (URLs) http://www.slideshare.net/kleinerperkins/kpcb-internet-trends-2013 http://www.ifis.cs.tu-bs.de/teaching/ss-11/irws http://www.worldwidewebsize.com/ Note: <additional comments> NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Netflix Movie Service Vertical (area) Commercial Cloud Consumer Services