Optimizing M3: Halving Our Metrics Ingestion Latency by Forking the Go Compiler
Ah, memory management. Here’s my basic understanding:- Go initially allocates 2KB stack per routine. When it exceeds it it copies all of it into 2x the space.- This was happening once or twice per request. They didn’t explain exactly why all that stack memory was being used (maybe someone can chime in), but contributing factors were a 30 function deep call stack and a minor code change that tipped it into the next stack growth tier.- Also this doesn’t get freed up until garbage collection runs.- They worked around it by implementing a kind of go routine pool that keeps assigning work to the same (stack-expanded) routines, staying ahead of the garbage collector.My takeaways:1. Fantastic analysis and job well done.2. Pooling does not seem to be how things “should” work in Go. It’s more of a hack around undesirable allocator / garbage collector behavior.3. I’m really interested in reference counted languages like Swift on the server for these reasons. I know ARC means more predictable latency when it comes to garbage collector behavior (which is only indirectly the problem here). Now I’m really curious how Swift allocates the stack and whether it would avoid this “morestack” growth penalty that Go has.