Postmortem of database outage of January 31
&gt; Hot standby is also supported just fine out of the box, you just need something third party for the actual load balancing.<p>I&#x27;m aware that hot standby is supported, though it&#x27;s not the default configuration for the standby server (default and safest is a standby mode that you can&#x27;t query at all; hot standby introduces possible conflicts between hot read queries and write transactions coming in from the WAL, so if failover is your primary intention, you should be cold standbying). I&#x27;m saying that mixing read queries in and dispersing them over hot standbys is not well-supported, which is why you need third-party tools to do it.<p>It can also be risky if your replication lag gets out of control, and you&#x27;ve indicated that it easily does. PgSQL replication is eventually consistent and you risk returning stale data on reads, which could cause all sorts of havoc if it&#x27;s not accounted for by the application internally.<p>&gt; We&#x27;ve had a few too many cases like this in the past. We&#x27;re aiming to resolve those, but unfortunately this is rather tricky and time consuming.<p>This may take some upfront work, but it&#x27;s pretty routine. A serious commercial-level offering should not need to take itself offline without announcement in order to restart the single database server and apply a configuration tweak.<p>&gt; Code is written by developers, and developers are humans. Humans in turn make mistakes. Most project removal related code also existed before we started enforcing stricter performance guidelines.<p>The point is not that humans make mistakes, nor that bugs exist. The point is that such a feature was released without considering its easily-exploitable potential and the permanent consequences of its exploitation (permanent removal of data). That should trigger a process review.<p>&gt; There&#x27;s no point in hiding it. Spending a few minutes digging through the code and you&#x27;ll find it, and probably plenty other similar problems. If somebody tries to abuse it we&#x27;ll deal with it on a case by case basis.<p>There&#x27;s a lot of risk in drawing attention to this type of vulnerability. I think GitLab should be taking this more seriously. All code has bugs, but this isn&#x27;t a bug; it&#x27;s an incomplete, dangerously-designed feature that can be easily used by a malicious actor to permanent destroy large quantities of user data. Your CEO has just highlighted it before the whole world while it&#x27;s still active and exploitable on the public web site.<p>Reading the code isn&#x27;t a dead giveaway because it takes a lot of effort to find the specific code in question and realize what it means, and because the general assumption would be that GitLab.com is running a souped-up or specialized flavor of the code and that such dangerous design flaws <i>must</i> have already been resolved on a presumably high-traffic site. However, this post highlights that it hasn&#x27;t been, and that&#x27;s bad. This is effectively irresponsible self-disclosure of a very high-grade DoS exploit.<p>&gt; Probably just a naive configuration value since we have plenty of storage available.<p>Having the storage readily available means that the hard part is already done! Each WAL segment is 16MB. You have about 350 GB of unused disk. Set wal_keep_segments and min_wal_size to something reasonable and you won&#x27;t need to do this obviously-risky resync operation every time you have a couple of hours of heavy DB load.<p>&gt; Revealing hostnames isn&#x27;t really a big deal, neither is SSH running on port 22. In the worst case some bots will try to log in using &quot;admin&quot; usernames and the likes, which won&#x27;t work. All hosts use public key authentication, and password authentication is disabled.<p>See discussion at <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13621027" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13621027</a>. The worst case is not a bruteforced login, it&#x27;s an exploited daemon that leads to an exploited box that leads to an exploited network that leads to an exploited company. The secondary concern would be a DoS attack; everyone now knows that you have only one functioning database server that everything depends on, and that that server&#x27;s IP is x.x.y.y. That&#x27;s enough to cause trouble even without exploits or zero days.<p>&gt; When using psycial disks not used by anything else, maybe. However, we&#x27;re talking about disks used in a cloud environment. Are they actually physical? Are they part of larger disks shared with other servers? Who knows. The chance of data recovery using special tools in a cloud environment is basically zero.<p>Yes, this complicates things significantly. Something like EBS may be able to be used pretty similarly to a dd image, though there is no way to &quot;pull the plug&quot; on an EC2 server afaik (maybe it&#x27;s exposed through the API). I&#x27;ve never used Azure so I don&#x27;t know if this would be practicable there.<p>&gt; That only works for files still held on to by PostgreSQL. PostgreSQL doesn&#x27;t keep all files open at all times, so it wouldn&#x27;t help.<p>Indeed. While PgSQL doesn&#x27;t keep all files open at all times, it <i>does</i> keep some files open, and they may or may not have contained useful data. I personally would&#x27;ve also been interested in trying to freeze the memory state (something you <i>can</i> do with a lot of raw VMs that you can&#x27;t do with physical servers, but admittedly probably not something the cloud provider exposes).