Fluvio: A programmable data platform
I have read the article and I do have some questions and observations.<p>&gt; Some admin operations such as setting up data sharing or re-balancing stream after config update require an IT ticket to be handled by the operation team.<p>No, they don&#x27;t. The organizations may have such requirements but the products do not require this. I don&#x27;t see how this would be different in an org that decided to deploy the Fluvio product.<p>&gt; Java-derived languages remains the only reliable way to customize stream processing.<p>There is a reason for that. Java has by far the most convenient method of providing and replacing dependencies at runtime while also providing type safety and being OS independent. Go modules do not count because they&#x27;re compiled into the program, go plugins - do they actually work? - have serious drawbacks. Things like goplugin from HashiCorp are really nice but they run separate process and communicate over a socket, thus add latency. I don&#x27;t think it&#x27;s different in the Rust world. With Java, I can have a program deployed and simply replace a jar dependency, there&#x27;s not even a need to restart a program in certain cases thanks to class loading mechanisms.<p>&gt; One of the most significant drawbacks of Java-based stream processing frameworks is the Jar wrapper required for distribution. Jars were designed at the dawn of the Internet when Browsers were rudimentary HTML readers and programs required runtime applets (aka. sandboxes) to operate. These sandboxes have been riddled with security vulnerabilities, and new browsers are gradually deprecating them.<p>What does the browser security have to do with jar being the distribution model for Java  programs?<p>&gt; Some frameworks resorted to container technologies such as Docker to add another layer of isolation and a workaround for dynamic loading. Unfortunately, the container introduces another layer of security issues and introduces more latencies and cold-startup time.<p>Sure, Fluvio says it&#x27;s cloud native and designed to run in Kubernetes. I guess it has the same problems?<p>&gt; The data-at-motion stream processing must handle an order of magnitude higher data than the products storing data-at-rest.<p>Are you sure about that? The storage layers I have seen have significantly more data than what&#x27;s in flight.<p>&gt; Consequently, stream process platforms must be small enough to boot within milliseconds and operate efficiently on any system architecture. Moreover, it must support a variety of deployments from small organizations to large enterprises. The ideal platform has: ...<p>What would be the need for booting within milliseconds? What benefit does it have considering that a data broker lifetime is long enough to justify a longer boot time? Do you take consistency into consideration? How is Fluvio handling replication? What is considered &quot;low latency&quot; and why would existing systems not be able to provide that?<p>&gt; Stream processing engines with access to data must have a robust sandboxing environment that can enforce access control and protect data records from impacting each other.<p>Aha, so this has nothing to do with Kafka or Pulsar. This has to do with Spark, Flink, Kafka Streams, Puslar Functions.<p>&gt; While SQL-based tools may be adequate for querying data-at-rest or a data lake, they offer limited functionality for developers who need full API access for automation.<p>All existing frameworks have an API.<p>&gt; Development APIs must be available in many widely used programming languages such as Node&#x2F;JavaScript, Python, Go, Ruby, etc.<p>I&#x27;d be happy to have a fully supported client. That would be enough.<p>&gt; ... consumer and producer as with legacy Java-based stream processing frameworks ...<p>That&#x27;s some &quot;modern vehicle manufacturer&quot; level cockery right there.<p>&gt; The Rust language powers Fluvio. Rust, a modern programming language built for speed, low overhead, cross-platform interoperability, and code safety. AWS, Mozilla, Google, Facebook, Discord, Dropbox, and others use Rust to create a new class of high-performance products, such as browsers, chat servers, network proxies, database servers, real-time systems, and more.<p>Whatever. Really, it does not matter to me what the product is written in. I&#x27;ll be interfacing with it via the client and as long as the platform works, I do not care what is it written in. A programming language the product is written in is not a selling point.<p>&gt; Rust is safe by default programming language, unlike ...<p>Again, why would I care when I am interfacing with the product via the client? Those are the reason why the company behind this product have selected Rust and that&#x27;s great. But it&#x27;s not a selling point.<p>&gt; With a Java-based streaming framework, it is challenging to provide a programmable pipeline with performance and security.<p>Why?<p>&gt; It is proven W3 technology to bring programmability to software such as Envoy Proxy, Cloudflare worker,<p>While WASM is great, it&#x27;s not a standard in those systems. WASM is one of the plugins in Envoy and one of the Workers runtime. WASM is cool but the tooling is not there yet to take it to the real masses.<p>&gt;  Since user modules can only access data supplied by SmartStream, it can’t access or modify protected information as PII data.<p>How does it prevent that if a stream being processed contains PII data?<p>&gt; WebAssembly supports any language with bindings to the LLVM toolchain - Rust, JavaScript, Python, Ruby, and Go. Fluvio offers abstractions, templates, utilities, and tools to make it easy and convenient to build and customize stream processing modules.<p>So basically, I need to buy into the ecosystem and I&#x27;m locked in into the WebAssembly stuff used by this platform to achieve the Zen of &quot;performance&quot;. Kafka clusters I work with ingest 400MB&#x2F;s easily and give me an end to end latency of 100ms with additional processing in the way. What is WebAssembly giving me here?<p>--<p>Now, I like the idea of having a lightweight stream processing engine where each topic could be its own raft group. Something that could be deployed close to the storage to avoid having to move volumes of data for processing across network boundaries just to do the computation. However, this article does not suggest that Fluvio would be just that. It mixes the stream processing part with storage part. Kafka is not &quot;stream processing&quot;, Pulsar is not &quot;stream processing&quot;. I do not really understand what does this do. It seems to be doing everything...<p>I&#x27;m not trying to say the product is bad, but this article does not give me any incentive to look at it. The amount of responsibilities packed in this write up make it sound &quot;complex&quot; and &quot;difficult&quot;. It completely glosses over the data storage concerns, replication and high availability of stream processing. I&#x27;ll keep an eye, though :)<p>Today, my first go to for a data mesh transport would be AMQP 1.0 because every participant can be both: a client and a server.