Check out LinkedIn&#x27;s posts about log processing [0] and Apache Kafka.  Handling data as streams of events lets you avoid spikey query-based processing, and helps you scale out horizontally.  Partitioning lets you do joins, and you can still add databases as &quot;materialized views&quot; for query-ability.  Add Secor to automatically write logs to S3 so you can feel secure in the face of data loss, and use replication of at least 3 in your Kafka topics.  Also, start with instrumentation from DataDog or NewRelic from the start - it will show you the performance bottlenecks.<p>0: <a href="https:&#x2F;&#x2F;engineering.linkedin.com&#x2F;distributed-systems&#x2F;log-what-every-software-engineer-should-know-about-real-time-datas-unifying" rel="nofollow">https:&#x2F;&#x2F;engineering.linkedin.com&#x2F;distributed-systems&#x2F;log-wha...</a>