Paul Hewson|Helen MacGillivray
Big data and little people
statistical circles near impossible avoid phrase big data reviewing big data years ago cukier 2010 referred sloan digital sky survey accumulated 140 terabytes data went note large synoptic survey telescope due come onstream chile year would accumulate much data every 5 days cukier 2010 also noted walmart accumulates data 1 million customer transactions every hour data scale raises whole host computational issues collection curation consider issues summarizing visualizing modelling interpreting using big data make inferences whole new statistical ball game may view opening pandoras box big data however new phenomena ayres 2007 accessible popular science account makes many social implications clear hype around big data left many people using phrase synonymously data big success stories interpretation customer transaction data left many businesses interested statistics offer handle data pentabyte scale clearly requires computational facilities software skills beyond reach many learners 919 years age range remains case core statistical literacy skills required interpret data remain true data scale decade ago ayres 2007 described way offermatica established support use randomized controlled trials website design essentially potential existing customers could randomly presented different versions website couple acquisitions company part adobe httpwwwadobecommarketingcloudtestingtargetinghtml one features offered adobe referred computer industry ab testing relation websites easily set run tests even several time determine best offers messaging content words offer carry randomized controlled trials web order find best way make sales clearly many technical issues setting service heart fundamental point statistical literacy want make assertions causality use randomly allocated experimental design understanding ab testing works much statistical concept learnt reasonably early ones education even technical skills implement industrial scale come much later netflix recommender algorithm described gomezuribe hunt 2016 one famous big data algorithms demonstrates importance mathematics developing ability interrogate modern data structures basic idea familiar many recommends purchases hires basis viewing history similarity main difference algorithms previous applications idea however cleverness algorithms main interpretative difference based observational data famous example beernappies link apparently found friday nights walmart dennis et al 2001 clear multiple testing issues see example article zheng lu 2016 risk false positive statistical significance key conceptual point observational data even genuine association may important underlying reasons association key statistical literacy concepts possibility confounding variables explain apparent association observational data need learnt early ones statistical education editors used full data investigation projects help university students learn engage aspects statistics students often choose observe everyday phenomena pique curiosity one rich source questions observations travel whether might foot cycle car public transport example collected data travel preferences bicycle walking car driving car occupancy well time day direction gender travellerdriver explore interpret data also use data 1 discuss limitations data collection process 2 compare student obtained results officially reported statistics focus back reasons numbers might differ whether differences invalidate claims made official statistics even simple critiques make clear array critical thinking needed make sense observational data perhaps range weather student snapshot narrow biassed results perhaps time day issue perhaps data collection recording needed better trained observers point though barring strong social pressures ban use big data civil libertarian grounds business governments attempt influence lives use big data jobs sectors people handle compute interpret big data first steps understanding big data data statistical literacy vital prerequisite understanding big data