Logging at Twitter
It’s fascinating to see companies solve the same problems over and over again.I hope in the near future these types of problems are “definitively solved” and more effort can be spent engineering novel features.There’s no way to do this, but I’d be interested to see if there’s a company that just has an extremely simple stack that is very performant and popular.I’m talking: their entire stack is Citus sharded over a few locations running two beefy servers per continent they have users in, that’s it. No dedicated search, logging or anything. Just dump everything in Postgres. If logs are lost then so be it.A lot of the problems at scale are problems of our own creation. For example is strong consistency really necessary for tweets? Do you really need to see a followers latest tweet instantly?Uncheck some boxes and the engineering becomes a lot simpler, very fast.