One SQL to Rule Them All: Management of Streams and Tables
A really interesting proposal to unify stream processing and relational querying and to extend SQL with a semantics over time-varying relations.In particular, the authors (from the Apache Beam/Flink/Calcite projects) stress on the need to:* smooth the frontier between streams and tables by using a single concept of time-varying relations;* make explicit the difference between the event and processing times, using monotonic watermarks to tie the two;* control when these continuously varying relations are materialized, avoiding unnecessary work and producing only timely outcomes.These ideas are not new per se, but there are here pushed further and nicely combined.However, I'm wondering if SQL is the right tool for the task. For instance, the listing 2 seems complex when compared to the query expressed in plain English. In particular, I disagree that "anyone who understands enough SQL to solve a problem in a non-streaming context still has the knowledge required to solve the problem in a streaming context as well.A must read paper by who is keen on stream processing, Spark, Flink, Beam, Kafka Stream ...