'Big Data' Has Come to Mean 'Small Sampled Data'
I&#x27;d just commented on this at HN a few days ago[1], noting both that a fair amount of &quot;big data&quot; is actually surprisingly small, and that sampling-based methods are very, very, very often not only <i>good enough</i>, but tremendously reduce costs and complexity.  There are other factors, some of which favour the smaller-is-better approach, some of which favour more comprehensive approaches.<p>Background:  though my pseudonymity precludes giving specifics, I&#x27;ve worked in and around data analytics for much of three decades, across various fields.  What was &quot;big data&quot; at the onset of my career -- when multiple departments of over thirty analysts shared access to a few GB of storage distributed over multiple cluster systems -- is now something I can trivially tackle on a modest and decade-and-a-half-old desktop system.  Newer boxen, with yet more memory and SSD or hybrid drives are even more capable.<p>Under my present &#x27;nym, I&#x27;ve done analysis especially of Google+ since 2015 (largely out of personal interest and realising methods <i>were</i> available to me).  Much of that has been based on sampling methods, some has relied on more comprehensive analysis, though again, with an eye to limiting total data processing.<p>In the case of Google+, the questions of how many actively-posting users and how many active Communities exist have come up.  Sampling methods have been <i>reasonably</i> useful in assessing these.<p>Google&#x27;s sitemaps files (from <a href="https:&#x2F;&#x2F;plus.google.com&#x2F;robots.txt" rel="nofollow">https:&#x2F;&#x2F;plus.google.com&#x2F;robots.txt</a>) supplied <i>total</i> counts of profiles, communities, and several other categories.  This itself takes some work -- there are 50,000 Profile sitemap files, and 100 Communities files, containing 3.4 billion and 8.1 million entries, respectively.  Even downloading <i>just</i> the Profile sitemaps itself is roughly 37 GB of data.<p>For a <i>rough</i> estimate of active profiles, <i>it only took about 100 randomly sampled observations to come up with pretty solid value:</i>  about 9% of profiles had posted publicly ever (a ratio which remains true).  <i>Finer-level</i> detail for more frequently posting accounts takes a far larger sample <i>because those highly-active profiles are so much rarer</i>.  I&#x27;d done my first analysis based on about 50,000 records, while Stone Temple Consulting independently confirmed and extended my analysis to find an estimate of profiles posting 100+ times per month (about 50,000 across the full site).  That is, 0.0015625% of all G+ profiles extant at the time (about 3.2 billion).<p>When I first tackled the G+ Communities question, I was trying to get a sense of:<p>1. How many there were.<p>2. What typical membership was.<p>3. How many were &quot;reasonably active&quot;, based on ... somewhat contrived measures.<p>I adopted a sampling approach -- it was easy to pull a full <i>listing</i> of Communities, but web-scraping these would take time (at about 1.5s per URL).  So my first approach utilised a 12,000 record sample.  That gave a good overall view, but turned out to be thin on the very largest communities, so I ran a second pull based on 36,000 records, also randomly sampled.  (A test of sampling:  how many communities should be in both samples.  The expected and actual result matched:  53.)<p>Even then, representation from the very largest communities was thin.  Fortunately, I found someone able to sample all 8.1 million communities rapidly, and from this compiled a list of 105,000 communities, with 100+ members <i>and</i> posting activity within the 31 previous days (based on Jan 5-6, 2019).<p>Note that this highlights another aspect of media:  of <i>all 8.1 million</i> communities, only 1.3% fell into the active list.  That is <i>media and community activity tend to focus most ACTIVITY on a VANISHINGLY small subset of total members, groups, posts, or other entities.</i>  That is, attention is highly rivalrous, and is <i>exceedingly</i> unequally distributed.<p>The challenge is in finding the active entities.  Given my approach, via sitemaps, this was difficult (though a few options may have been available).<p>The upshot is that in the case of both users and communities, the <i>really active</i> and <i>interesting</i> set is, by modern standards, <i>small data</i>.  There are 50,000 to a few million active users, a few thousands (and far fewer than 100,000) truely active communities.  Even the total post volume of G+ communities is, by modern standards, strikingly small -- from January 2013 to January 2019, about 300 GB of post text was submitted.  Image content and the surrounding Web payload (800 kiB per post) inflate that considerably, but yes, <i>over half a decade</i> of a reasonably large social media&#x27;s community discussions could sit comfortably on most modern hard drives, or even much mobile storage.<p>I&#x27;ll skip over the questions of truly social analysis of such data, with connections between users and other entities, though I&#x27;ll note that <i>even for very large systems</i>, managing these at scale is difficult and the results may not be particularly usefully interpreted.  Combinatorial maths remains challenging, even with beefy iron.<p>________________________________<p>Notes:<p>1. See: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19294024" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19294024</a> and <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19294004" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=19294004</a>, on the often surprisingly <i>small</i> scale of &quot;big data&quot; and the value of sampling, respectively.