Apache Pulsar is an open-source distributed pub-sub messaging system
This looks promising. Is there such thing as a generalized SQL query engine that runs over any key-value store that provides certain minimal core operations?<p>For example, say you have a KV Store with basic mathematical Set operations like GET, SET, UNION, INTERSECT, EXCEPT, etc. The Engine would parse the SQL and then call the low-level KV Store Set operations, returning the result or updating KV pairs. This explains how Join relates to Set operations:<p><a href="https:&#x2F;&#x2F;blog.jooq.org&#x2F;2015&#x2F;10&#x2F;06&#x2F;you-probably-dont-use-sql-intersect-or-except-often-enough&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.jooq.org&#x2F;2015&#x2F;10&#x2F;06&#x2F;you-probably-dont-use-sql-i...</a><p>Another thing I&#x27;d like is if KV stores exposed a general purpose functional programming language (maybe a LISP or a minimal stack-based language like PostScript) for running the same SQL Set operations without ugly syntax. I don&#x27;t know the exact name for this. But if we had that, then we could build our own distributed databases, similar to Firebase but with a SQL interface as well, from KV stores like Pulsar. I&#x27;m thinking something similar to RethinkDB but with a more distinct&#x2F;open separation of layers.<p>The hard part would be around transactions and row locking. A slightly related question is if anyone has ever made a lock-free KV store with Set operations using something like atomic compare-and-swap (CAS) operations. There might be a way to leave requests &quot;open&quot; until the CAS has been fully committed. Not sure if this applies to ledger&#x2F;log based databases since the transaction might already be deterministic as long as the servers have exact copies of the same query log.<p>Edit: I wrote this thinking of something like Redis, but maybe Pulsar is only the message component and not a store. So the layering might look like: [Pulsar][KV Store (like Redis)][minimal Set operations][SQL query engine].