Ask HN: BigQuery vs. Redshift vs. Athena vs. Snowflake
Disclaimer:
I&#x27;m the co-founder of a start-up that provides performance analytics for Amazon Redshift. So I&#x27;m biased towards using Redshift because otherwise I can&#x27;t sell you our service. With that cleared, some considerations.<p>One of our customers I thought made a great observation. &quot;I see a lot of people using Redshift. But I don&#x27;t see anybody using it happily&quot;. Because of all the issues that people point out in this thread. lots of operations. little visibility (&quot;black box&quot;). All very true. It&#x27;s the problem our product solves, and let me talk about how we see companies successfully using Redshift at scale. (and that customer is now a happy Redshift user, btw).<p>Here&#x27;s the architecture that we&#x27;re seeing companies on AWS moving to:<p>- all your data in S3 (I hate the term, but call it a &quot;data lake&quot;)<p>- a subset of your data in Redshift, for ongoing analysis<p>- daily batch jobs to move data in and out of Redshift from &#x2F; to S3<p>- Athena to query data that&#x27;s in S3 and not in Redshift<p>The subset of the data sitting in Redshift is determined by your needs &#x2F; use cases. For example, let&#x27;s say you have 3 years of data, but your users only query data that&#x27;s less than 6 months old. Then moving data older than 6 months to S3 makes a lot of sense. Much cheaper. For the edge cases where a users does want to query data older than 6 months, you use Athena to query data sitting in S3.<p>Why not use Athena for everything? Two reasons. For one, Athena is less mature. And then you still need a place to run your transformations, and Redshift is the better choice for that.<p>Three uses cases for Redshift:<p>1) classic BI &#x2F; reporting (&quot;the past&quot;)<p>2) log analysis (&quot;here and now&quot;)<p>3) predictive apps (&quot;the future&quot;)<p>Your use case will determine how you have to think about your data architecture. For some it&#x27;s ok if a daily report takes a few minutes to run. I&#x27;ll go get a coffee. It also means my batch jobs are running on daily or hourly cycles. But if I&#x27;m running a scoring model, e.g. for fraud prevention, I want that score to be as fresh &#x2F; real-time as possible. And make sure that the transformations leading up to that score have executed in their proper order.<p>For Redshift to work at scale, there are really only three key points you need to check the box on:<p>- setting up your WLM queues so that your important workloads are isolated from each other<p>- allocating enough concurrency and memory to each queue to enable high throughput<p>- monitoring your stack for real-time changes that can affect those WLM settings<p>That&#x27;s it. And then Redshift has a rich ecosystem and rich feature set to enable all uses cases (etl vendors, dashboard tools, etc.). Those users who have addressed the performance challenges that do come at scale for Redshift - they&#x27;re happy users and never looked back.<p>On the pay-per-TB pricing for BigQuery and Snowflake. I think that&#x27;s more of a marketing spin. Most companies we work with are compute-bound. So per-TB pricing helps them very little. They want to crunch their data as fast as possible. More CPUs give them more I&#x2F;O. If you feel you&#x27;re storage-bound - take a hard look at the data that really needs to be available in Redshift for analysis, and move everything else to S3.<p>For inspiration, watch the AWS Reinvent videos on S3 &#x2F; Redshift &#x2F; Athena from Netflix and NASDAQ. Like this one:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=o52vMQ4Ey9I&amp;t=256s" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=o52vMQ4Ey9I&amp;t=256s</a><p>If you&#x27;re entire data is already within AWS, I think Redshift is the way to go. But then again I&#x27;m biased :)