Don't Use Protobuf for Telemetry
I have written an in-house implementation of protobuf for C++ (sorry can&#x27;t share) and studied the wire format extensively.<p>Google&#x27;s implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they&#x27;re very good, but for a use case that I haven&#x27;t yet encountered). Don&#x27;t shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least <i>some</i> implementation available.<p>Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it&#x27;s not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting).<p>What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster.<p>The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.