Note: This is based on solutions I have been researching for a current project and I haven&#x27;t used these in production.<p>Short answer: I think you&#x27;re looking in the wrong direction, this problem isn&#x27;t solved by a database but a full data processing system like Hadoop, Spark, Flink (my pick), or Google Cloud&#x27;s dataflow. I don&#x27;t know what kind of stack you guys are using (imo the solution to this problem is best made leveraging java) but I would say that you could benefit a lot from either using the hadoop ecosystem or using google cloud&#x27;s ecosystem. Since you say that you are not experienced with that volume of data, I recommend you go with google cloud&#x27;s ecosystem specifically look at google dataflow which supports autoscaling.<p>Long answer: To answer your question more directly, you have a bunch of data arriving that needs to be processed and stored every X minutes and needs to be available to be interactively analyzed or processed later in a report. This is a common task and is exactly why the hadoop ecosystem is so big right now.<p>The &#x27;easy&#x27; way to solve this problem is by using google dataflow which is a stream processing abstraction over the google cloud that will let you set your X minute window (or more complex windowing) and automatically scale your compute servers (and pay only for what you use, not what you reserve). For interactive queries they offer google bigquery, a robust SQL based column database that lets you query your data in seconds and only charges you based on the columns you queried (if your data set is 1TB but the columns used in your query are only some short strings they might only charge you for querying 5GB). As a replacement for your mysql problems they also offer managed mysql instances and their own google bigtable which has many other useful features. Did I mention these services are integrated into an interactive ipython notebook style interface called Datalab and fully integrated with your dataflow code?<p>This is all might get a little expensive though (in terms of your cloud bill), the other solution is to do some harder work involving the hadoop ecosystem. The problem of processing data every X minutes is called windowing in stream processing. Your problems are solved by using Apache Flink, a relatively easy and fast stream processing system that makes it easy to set up clusters as you scale your data processing. Flink will help you with your report generation and make it easy to handle processing this streaming data in a fast, robust, and fault tolerant (that&#x27;s a lot of buzz words) fashion.<p>Please take a look at the flink programming guide or the data-artisans training sessions on this topic. Note that the problem of doing SQL queries using flink is not solved (yet) this feature is planned to be released this year. However, flink will solve all your data processing problems in terms of the cross table reports and preprocessing for storage in a relational database or distributed filesystem.<p>For storing this data and making it available you need to use something fast but just as robust as mysql, the &#x27;correct&#x27; solution at this time if you are not using all the columns of your table is using a columnar solution. From googles cloud you have bigquery, from the open source ecosystem you have drill, kudu, parquet, impala and many many more. You can also try using postgres or rethinkdb for a full relational solution or HDFS&#x2F;QFS + ignite + flink from the hadoop ecosystem.<p>For the problem of interactively working with your data, try using Apache Zeppelin (free, scala required I think) or Databricks (paid but with lots of features, spark only i think). Or take the results of your query from flink or similar and interactively analyze those using jupyter&#x2F;ipython(the solution I use).<p>The short answer is, dust off your old java textbooks. If you don&#x27;t have a java dev on your team and aren&#x27;t planning on hiring one, the google dataflow solution is way easier and cheaper in terms of engineering. If you help I do need an internship ;)<p>If you want to look at all the possible solutions from the hadoop ecosystem look at:
<a href="https:&#x2F;&#x2F;hadoopecosystemtable.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hadoopecosystemtable.github.io&#x2F;</a><p>For google cloud ecosystem it&#x27;s all there on their website.<p>Happy coding!<p>Oops, it seems I left out ingestion, I would use kafka or spring reactor.<p>P.S The flink mailing list is very friendly, try asking this question there.