Shadow traffic: site visits that are not captured by typical analytics providers
I&#x27;m one of Parse.ly&#x27;s co-founders. This post was written by one of our product managers about a project and investigation we&#x27;ve been doing for the past few months. It first got on my team&#x27;s radar when I posted this set of tweets back in 2019:<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;amontalenti&#x2F;status&#x2F;1165262620959617025" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;amontalenti&#x2F;status&#x2F;1165262620959617025</a><p>Specifically: I noticed a huge difference between the metrics we were reporting on my blog post in Parse.ly, and the metrics being reported by my personal blog&#x27;s Cloudflare CDN (caching the content).<p>Ironically enough, this traffic was all coming from HN and the post was itself about modern JavaScript[1].<p>Since then, we&#x27;ve also been hearing from a lot of customers about various scenarios where traffic is either under-counted or mis-counted. For example, something that has been tripping us up lately is that our Twitter integration relies (partially) upon the official t.co link shortener[2], and yet, due to modern browser rules related to W3C Referrer Policy[3], the t.co link&#x27;s path segment is often not transmitted to the analytics provider, and thus the source tweet for traffic cannot be easily ascertained.<p>I firmly believe in privacy and analytics without compromise[4], so the team is trying to come up with ways to at least quantify shadow traffic at an aggregate level, and to ensure legitimate user privacy interests are honored, while making sure they don&#x27;t break legitimate privacy-safe first-party analytics use cases.<p>As a developer, something that concerned me recently was realizing that Sentry, the open source error tracking tool with a SaaS reporting frontend and a JavaScript SDK[5], gets blocked in many conservative  browser privacy setups. Though the interest to user privacy is legitimate, I think we can all agree it&#x27;d be better for site&#x2F;app operators to know when certain browsers are hitting JavaScript stack traces.<p>[1]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20785616" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20785616</a><p>[2]: <a href="https:&#x2F;&#x2F;help.twitter.com&#x2F;en&#x2F;using-twitter&#x2F;url-shortener" rel="nofollow">https:&#x2F;&#x2F;help.twitter.com&#x2F;en&#x2F;using-twitter&#x2F;url-shortener</a><p>[3]: <a href="https:&#x2F;&#x2F;www.w3.org&#x2F;TR&#x2F;referrer-policy&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.w3.org&#x2F;TR&#x2F;referrer-policy&#x2F;</a><p>[4]: <a href="https:&#x2F;&#x2F;blog.parse.ly&#x2F;post&#x2F;3394&#x2F;analytics-privacy-without-compromise&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.parse.ly&#x2F;post&#x2F;3394&#x2F;analytics-privacy-without-co...</a><p>[5]: <a href="https:&#x2F;&#x2F;sentry.io&#x2F;for&#x2F;javascript&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sentry.io&#x2F;for&#x2F;javascript&#x2F;</a>