Big Data Data Source Add movies institutionally. Collect user rankings and Characteristics (distributed/centralized) profiles in a distributed fashion Volume (size) Summer 2012. 25 million subscribers; 4 million ratings per day; 3 million searches per day; 1 billion hours streamed in June 2012. Cloud storage 2 petabytes (June 2013) Velocity Media (video and properties) and Rankings (e.g. real time) continually updated Variety Data varies from digital media to user rankings, (multiple datasets, user profiles and media properties for mashup) recommendations Variability (rate of Very competitive business. Need to aware of other change) companies and trends in both content (which Movies are hot) and technology. Need to investigate new business initiatives such as Netflix sponsored content Big Data Science Veracity (Robustness Success of business requires excellent quality of (collection, Issues) service curation, Visualization Streaming media and quality user-experience to analysis, allow choice of content action) Data Quality Rankings are intrinsically “rough” data and need robust learning algorithms Data Types Media content, user of user rankings profiles, “bag” Data Analytics Recommender systems and streaming video delivery. Recommender systems are always personalized and use logistic/linear regression, elastic nets, matrix factorization, clustering, latent Dirichlet allocation, association rules, gradient boosted decision trees and others. Winner of Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Analytics needs continued monitoring and improvement. Challenges (Gaps) Big Data Specific Mobile access important Challenges in Mobility Security & Privacy Need to preserve privacy for users and digital rights for media. Requirements Highlight issues for Recommender systems have features in common to e-commerce like generalizing this use Amazon. Streaming video has features in common with other content case (e.g. for ref. providing services like iTunes, Google Play, Pandora and Last.fm architecture) More Information (URLs) by Xavier Amatriain Note: <additional comments>
Big Data Characteristics Data Source (distributed/centralized) Add movies institutionally. Collect user rankings and profiles in a distributed fashion Volume (size) Summer 2012. 25 million subscribers; 4 million ratings per day; 3 million searches per day; 1 billion hours streamed in June 2012. Cloud storage 2 petabytes (June 2013) Velocity (e.g. real time) Media (video and properties) and Rankings continually updated Variety (multiple datasets, mashup) Data varies from digital media to user rankings, user profiles and media properties for recommendations Variability (rate of change) Very competitive business. Need to aware of other companies and trends in both content (which Movies are hot) and technology. Need to investigate new business initiatives such as Netflix sponsored content Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Success of business requires excellent quality of service Visualization Streaming media and quality user-experience to allow choice of content Data Quality Rankings are intrinsically “rough” data and need robust learning algorithms Data Types Media content, user profiles, “bag” of user rankings Data Analytics Recommender systems and streaming video delivery. Recommender systems are always personalized and use logistic/linear regression, elastic nets, matrix factorization, clustering, latent Dirichlet allocation, association rules, gradient boosted decision trees and others. Winner of Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Mobile access important Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Recommender systems have features in common to e-commerce like Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments>