Title: Hello! For the past year I’ve been working on a fully-managed data warehouse built on Clickhouse. I built this because I was frustrated with how much work was required to run an OLAP database in prod: re-writing my app to do batch inserts, managing clusters and needing to look up special CREATE TABLE syntax every time I made a change. I found pricing for other warehouses confusing (what is a “credit” exactly?) and worried about getting capacity-planning wrong.<p>I was previously building accounting software for firms with millions of transactions. I desperately needed to move from Postgres to an OLAP database but didn’t know where to start. I eventually built abstractions around Clickhouse: My application code called an insert() function but in the background I had to stand up Kafka for streaming, bulk loading, DB drivers, Clickhouse configs, and manage schema changes.<p>This was all a big distraction when all I wanted was to save data and get it back. So I decided to build a better developer experience around it. The software is open-source: <a href="https:&#x2F;&#x2F;github.com&#x2F;scratchdata&#x2F;ScratchDB">https:&#x2F;&#x2F;github.com&#x2F;scratchdata&#x2F;ScratchDB</a> and and the paid offering is a hosted version: <a href="https:&#x2F;&#x2F;www.scratchdb.com&#x2F;">https:&#x2F;&#x2F;www.scratchdb.com&#x2F;</a>.<p>It&#x27;s called “ScratchDB” because the idea is to make it easy to get started from scratch. It’s a massively simpler abstraction on top of Clickhouse.<p>ScratchDB provides two endpoints [1]: one to insert data and another to query. When you send any JSON, it automatically creates tables and columns based on the structure [2]. Because table creation is automated, you can just start sending data and the system will just work [3]. It also means you can use Scratch as any webhook destination without prior setup [4,5]. When you query, just pass SQL as a query param and it returns JSON.<p>It handles streaming and bulk loading data. When data is inserted, I append it to a file on disk, which is then bulk loaded into Clickhouse. The overall goal is for the platform to automatically handle managing shards and replicas.<p>The whole thing runs on regular servers. Hetzner has become our cloud of choice, along with Backblaze B2 and SQS. It is written in Go. From an architecture perspective I try to keep things simple - want folks to make economical use of their servers.<p>So far ScratchDB has ingested about 2 TB of data and 4,000 requests&#x2F;second on about $100 worth of monthly server costs.<p>Feel free to download it and play around - if you’re interested in this stuff then I’d love to chat! Really looking for feedback on what is hard about analytical databases and what would make the developer experience easier!<p>[1] <a href="https:&#x2F;&#x2F;scratchdb.com&#x2F;docs">https:&#x2F;&#x2F;scratchdb.com&#x2F;docs</a><p>[2] <a href="https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;flatten-json&#x2F;">https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;flatten-json&#x2F;</a><p>[3] <a href="https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;scratchdb-email-signups&#x2F;">https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;scratchdb-email-signups&#x2F;</a><p>[4] <a href="https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;stripe-data-ingest&#x2F;">https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;stripe-data-ingest&#x2F;</a><p>[5] <a href="https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;shopify-data-ingest&#x2F;">https:&#x2F;&#x2F;scratchdb.com&#x2F;blog&#x2F;shopify-data-ingest&#x2F;</a>
Upvote: