Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
A big healthcare company I worked for did this. It worked extremely well, though it wasn&#x27;t without its drawbacks. They adopted the database-per-tenant pattern in the early &#x27;00s, and I truly think it was one of the major things that allowed them to scale to a large number of (increasingly larger in data&#x2F;access patterns) clients. It also made regulatory compliance a bit easier (everyone&#x27;s data is pretty firewalled off at the database-access-credentials layer) I think, but that wasn&#x27;t really my department.<p>We ended up in the &quot;thousands to tens of thousands&quot; of clients range, with thousands of tables per client and a pretty hairy schema.<p>Each customer had their own schema on one of a few dozen giant database servers. The company pushed this idea out to other parts of their infrastructure: separate webserver&#x2F;message broker&#x2F;cache tiers existed for each underlying database server, so outages or brownouts in one component couldn&#x27;t affect other customers&#x27; data that much.<p>Schema migrations, interestingly, weren&#x27;t much of a problem. The practice was nailed down early of &quot;everyone gets migrated during a release, no snowflakes&quot;. That, plus some pretty paranoid tooling and an acceptable-downtime (well, on paper it wasn&#x27;t acceptable, but everyone kinda understood that it was) in seconds-to-minutes during a release made migrations roughly as traumatic as migrations anywhere I&#x27;ve worked (which is to say, &quot;somewhat&quot;), but not too much more. It did take a lot of work to get the tooling right across multiple schemas in the same database server though. Investment in tooling--up to and including dedicated teams working on a single tool without a break-up date or firm mandate other than &quot;make this not suck and keep the lights on&quot;--is critical here, as in most areas.<p>Things that were hard:<p>- Connection management. Others on this thread have pointed that out. Connection pooling and long-lived queue workers were essential, and the web&#x2F;request tier couldn&#x27;t &quot;scale out&quot; too far without hitting connection limits. Scheduled jobs (and this company <i>loved</i> cron jobs, thousands of distinct invocations per tenant) were a problem in the connection-management department. Carefully written tooling around webserver connection reuse, cron-job execution harnesses (they didn&#x27;t really run as cron jobs, they got shipped to a worker already running the code with warm&#x2F;spare database connections and run there--all highly custom), and asynchronous jobs was needed. That occupied a team or three for awhile.<p>- The &quot;whale&quot; problem. When an individual tenant got big enough to start crowding out others on the same database server, it caused performance problems. We eventually worked on a migration tool that moved a client&#x27;s entire footprint (and remember, this isn&#x27;t just databases, but webs&#x2F;caches&#x2F;queue worker hosts&#x2F;etc.) onto another shard. Building this tool was a lot of work, but when it was done it worked surprisingly well. My advice in this area: build a good manually-initiated&#x2F;semi-supervised migration system. Leverage underlying database technology (binlog based replication). Don&#x27;t hesitate to get very dirty and custom with e.g. replication logfile formats, and don&#x27;t assume that $off_the_shelf_data_replicator isn&#x27;t gonna collapse when you want to do online per-schema replication in massive parallel from the same database (not even if that tool cost you millions of dollars). Do <i>NOT</i> succumb to the allure of &quot;we can automate the bin-packing and it&#x27;ll constantly rearrange clients&#x27; datasets for optimal resource usage!&quot; Manual is just fine for data migrations that big. Worst case, part of someone&#x27;s job is to initiate&#x2F;supervise them.<p>- SPOFs sucked. Some datasets weren&#x27;t per-tenant at all; sometimes client companies merged together or split up; some data arrived intended for a tenant but wasn&#x27;t tagged with that tenant&#x27;s ID, so it would have to go into some separate database before it found a home. These systems were, bar none, the biggest liabilities, causes of production issues, and hardest things to code around in the entire company. You&#x27;d think that having to write application code for thousands of logical databases across all the per-tenant schemas would suck, but in reality it wasn&#x27;t too hard. It was making sure your code didn&#x27;t accidentally talk to a SPOF that was the problem. My advice here: microservices do not help with this problem. HTTP, gRPC, or raw database wire protocol: if you have dependencies on a &quot;tragedy of the commons&quot;-type used-by-everyone server sneaking into your nicely sliced up per-tenant architecture, those callsites are going to be the cause of your sleepless nights. Get good visibility into where they occur. Favor &quot;push&quot; into per-tenant models over per-tenant code doing a blocking &quot;pull&quot;. Even if the push approach causes massive additional complexity and work. The costs of pull are too great.<p>- Some database specific shit (even on polished hosted offerings from AWS, or big-budget Oracle installs) will start acting really squirrely when you&#x27;re talking to thousands of identical schemas on the same database server (and thus tens or hundreds of thousands of identical tables with different data). If you double down on this route, be prepared to have a few really, really good database folks on staff. I don&#x27;t mean &quot;help me fix my slow giant reporting query&quot; people, I mean &quot;familiar with the internals&quot; folks. Example: query plans can be cached based on query <i>text</i>, globally, across an entire database server. Different schemas have <i>super</i> different clients, and thus super different data distribution among 100s of GBs of data. The plan that gets cached for query X against client A is the product of running heuristics&#x2F;histograms&#x2F;etc. across client A&#x27;s data. That plan might perform pathologically when query X runs against client B (on the same database server)&#x27;s data, and finding out how&#x2F;why is really annoying. Solution: bust the cache by a) happening to know that SQL comments aren&#x27;t stripped from query texts before the text is used as a plan-cache key and b) prepend each tenant&#x27;s identifier to each query at the database-driver level to prevent cache pollution. Result: you have traded a spooky query performance issue for a query-plan-cache-size issue; now your queries are all predictably slow because all your tenants&#x27; different queries are thrashing the plan cache. Tradeoffs abound.