Hadoop “fails us”
I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and then we migrated to other Big Data technologies. Just wanted to add my thoughts here.<p>Hadoop (and its general ecosystem, which includes HBase), is a fairly good idea. Its core ideas - map&#x2F;reduce on Hadoop, and a large distributed key&#x2F;value store for HBase - are actually pretty solid.<p>And for many years, there were simply no alternatives to Hadoop. Think of the years 2008 to 2012&#x2F;13. If you had to process terabytes or petabytes of data, what were your solutions? No wonder Yahoo and Facebook (and others) put in so much effort into their Hadoop solutions.<p>But, IMHO, there were several issues with Hadoop and their ilk.<p>1. The core infrastructure wasn&#x27;t stable enough. Hadoop &#x2F; HBase were supposed to be distributed systems, and they worked well, but small failures could bring down your entire cluster. Given that Hadoop and HBase were being used in mission-critical systems in the cloud, and given the amount of DevOps or sys-admin work that went into maintaining these, I&#x27;m not surprised people eventually migrated to distributed systems that were easier to maintain and run.<p>2. There are now plenty of &quot;hosted on the cloud&quot; solutions such as Amazon DynamoDB or similar cloud solutions. When your company depends on 99.99% or similar SLAs, you don&#x27;t want to have downtime on your database systems and spend time debugging complicated core dumps on your Hadoop or HBase clusters when you can just store it &quot;on the cloud&quot; and be done with it. Sure, there&#x27;s a higher price point, but those are the trade-offs you live with.<p>3. If you want to be in-house, there are plenty of alternatives out there as well today. Apache Spark for processing, Kafka for a messaging bus &#x2F; streaming data, ElasticSearch for large scale storage, with multiple indices. Many of them are much more robust than Apache Hadoop &#x2F; HBase, and I&#x27;m not surprised they&#x27;ve gotten more traction recently.<p>Ultimately, I think Hadoop &#x2F; HBase are just showing their age. They were fantastic for the first wave of Big Data technologies, and you had little alternative if you were building large-scale systems circa 2008 to 2013, but now, you just have a plethora of choices from various vendors.