Ballista: Distributed compute platform implemented in Rust using Apache Arrow
It saddens me a little bit that a nascent protocol like arrow flight is using grpc + protobufs behind the scenes with some hacks on top to skip the cost of deserializing protobufs. It seems like a really common belief that protobufs have so much engineering time behind them and are cross-language that it's a no brainer to implement your new protocol on top of them.<p>In reality, all the engineering and optimization time is behind the implementations for the google internal languages, and even the python protobuf implementation is pretty bad.<p>Protobuf makes some <i>stunningly</i> bad decisions like using varints, etc that you shouldn't make the immediate assumption "google has tons of great engineers, google uses protobuf for everything internally, therefore, protobuf is a good foundation to build my new thing on top of"<p>In reality, path dependence and the (amazing) internal tooling ecosystem at google both play a huge part of why they use protobuf so extensively.<p>(Grpc is a little overly complicated to be a universal recommendation, but I could believe it's a good choice for Arrow Flight. But it seems like they didn't do grpc + arrow or grpc + flatbuffer + arrow in the hopes that "dumb" grpc + protobuf implementations would be able to still benefit. In my opinion, grpc implementations are so coupled, there's no reason to make this unnecessary concession to protobufs)