Gitlab 500 error
I gotta say I love but am surprised by the public google doc of the outages.  If I worked there I would personally advocate for not doing that, only because I feel like it would slow down resolution if I have to constantly be worrying about not putting &quot;secret&quot; things into the doc.  Unless they have someone who&#x27;s job is to take their real chat and transcribe it, in which case, that&#x27;s great because that person is useful for a postmortem as well.<p>Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up?  Shouldn&#x27;t there at least be a fallback that says &quot;We&#x27;re offline right now sorry&quot; when the database is unavailable?  Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage?<p>My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer?  Do we at least have a practiced procedure for manual failover if not an automatic failover?<p>My third question would be what sort of alerts do we have, if any, on replication lag?<p>It&#x27;s been a while since I&#x27;ve had to run a major outage, this was an interesting exercise for me!