GitLab Database Incident – Live Report
I&#x27;m a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale.<p>Nothing important of mine is allowed to live exclusively on Gitlab.com.<p>It seems like they are just growing too fast for their level of investment in their production environment.<p>One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much disruption if I needed to (yay open source!). Which I ended up forced to do on short notice when their CI system became unusable for people who use their own runners (overloaded system + an architecture which uses a database as a queue. ouch.).<p>Which put an end to what seemed like constant performance issues. It was overdue, and made me sleep well about things like backups :).<p>A while back one of their database clusters went into split brain mode, which I could tell as an outsider pretty quickly... but for those on the inside, it took them a while before they figured it out. My tweet on the subject ended up helping document when the problem had started.<p>If they are going to continue offering Gitlab.com I think they need to seriously invest in their talent. Even with highly skilled folks doing things efficiently, at some point you just need more people to keep up with all the things that need to be done. I know it&#x27;s a hard skillset to recruit for - us devopish types are both quite costly and quite rare - but I think operating the service as they do today seriously tarnishes the Gitlab brand.<p>I don&#x27;t like writing things like this because I know it can be hard to hear&#x2F;demoralizing. But it&#x27;s genuine feedback that, taken in the kind spirit is intended, will hopefully be helpful to the Gitlab team.