The hunt for a cluster-killer Erlang bug (2021)
Very interesting writeup.
Distributed systems problem solving is always a very interesting process. It very frequently uncovers areas ripe for instrumentation improvement.<p>The Erlang Ecosystem seemed very mature and iterated. It almost seemed like the "rails of distributed system" with things like Mnesia.<p>The one downside to that seemed to be that while I was working on grokking the system, the limits and observability of some of these built-in solutions was not so clear. What happens when a mailbox exceeds it's limit? Does the data get dropped? Or, how to recover from a network segmentation? These proved somewhat challenging to reproduce and troubleshoot (as distributed problems can be).<p>There are answers for all of these interesting scenarios, but in some cases it almost would have been simpler to use an external technology (redis/etc) with established scalability/observability.<p>I do say this knowing that there was plenty I did not get time to learn about the ecosystem in the depth that I desired, but was curious how more experienced Erlang engineers viewed the problem.