Title: I applaud the transparency of the GitLab team in their recent outage, but felt bad for the engineer who&#x27;s typo was called out. Anyone who&#x27;s done something similar will know the feeling immediately after realising your mistake...<p>To show that this sort of thing happens to the best of us, let&#x27;s share some of our horror stories :)<p>A few months ago, I joined a new team and was still finding my way around the environments. I was tasked with performing manual deployments to a Dev, QA and Staging environment that weren&#x27;t wired up to our automation system yet. We&#x27;d scheduled maintenance windows a week apart for the QA and Staging envs as we allow customers to test against these.<p>So the day of my QA deployment, I start by applying the database changes which all complete successfully.  
Next, I upload the new .ear files and deploy the new build of our web app. Again, all looks good, so I tell the QA team they can start testing.<p>Then the alerts started...<p>I deployed the app to the Staging env by mistake (and unexpectedly restarted the app server). I didn&#x27;t realise the naming scheme of the hostnames indicated the environment in this case :&#x2F;<p>Our UI broke immediately due to the schema changes, so my mistake was _very_ visible. I was lucky I could roll back the change easily, but I don&#x27;t think I&#x27;ll forget that day any time soon.
Upvote: