FoundationDB: A Distributed Key-Value Store
FoundationDB has, in my experience, always been well regarded in DB development circles; I think their test architecture - developed to easily reproduce rare concurrency failures - is its best legacy, as mentioned in a comment above and frequently before.<p>However, since these topics are always filled with effusive praise in the comments, let me give an example of a distributed scenario where FDB has shortcomings: OLTP SQL.<p>First, FDB is clearly designed for “read often, update rarely” workloads, in a relative sense. It produces multiple consistent replicas which are consistently queryable at a past time stamp, without a transaction - excellent for that profile.  However, its transaction consistency method is both optimistic and centralized, and can lead to difficulty writing during high contention and (brief) system-wide transaction downtime if there is a failover; while it will work, it’s not optimal for “write often, read once” workloads.<p>Secondly, while it is an <i>ordered</i> key value store - facilitating building SQL on top of it - the popular thought of layering SQL <i>on top of the distributed layer</i> comes with many shortcomings.<p>My key example of this is schema changes. Optimistic application, and keeping schema information entirely “above” the transaction layer, can make it extremely slow to apply changes to large tables, and possibly require taking them partially offline during the update.  There are ways to manage this, but online schema changes will be a competitive advantage for other systems.<p>Even for read-only queries, you lose opportunities to push many types of predicates down to the storage node, where they can be executed with fewer round trips.  Depending on how distributed your system is, this could add up to significant additional latency.<p>Afaik, all of the spanner-likes of the world push significant schema-specific information into their transaction layers - and utilize pessimistic locking - to facilitate these scenarios with competitive performance.<p>For reasons like these, I think FDB will find (and has found) the most success in warehousing scenarios, where individual datum are queried often once written, and updates come in at a slower pace than the reads.