Shutting Down NavHere
Speaking purely about the technical side of things, I’m curious about the use of six servers at $55&#x2F;month. I’d expect to do such a thing with one and $5&#x2F;month, or two and $10&#x2F;month if I wanted to absolutely minimise downtime.<p>It’s well documented that making systems distributed to achieve high availability very often actually <i>reduces</i> availability, because distributed systems failure modes are gnarly, and so there’s just a lot more <i>complexity</i> in the system as a whole, a lot more moving pieces to go wrong. (Relatedly, it is also well understood that error-handling code is about the buggiest code out there.)<p>For starters, I can’t think of any reason to separate web application server from forwarding servers, so that’d lop it by one.<p>Then I’d lop the three database servers and put them on the forwarding servers as well, with something like a simple streaming replica arrangement if I was keeping both of the forwarding servers rather than reducing it to one.<p>This is pretty much guaranteed to be more robust, because now requests are depending on one machine only, perhaps with a second just hanging out in the background paying attention to what’s going on; whereas in your distributed system, there are at least <i>two</i> machines that can take things down by failing (if the DB server you’re talking to goes down, it’ll still be a short time before the system sorts itself out and switches to another one).<p>I would love kernel live patching to be better understood, tooled and used; then you could safely keep your single machine up indefinitely, just gracefully restarting your HTTP servers and databases as needed. As it is, if you go down to a single box, you’ll want to do the occasional quick restart. Ten seconds of downtime every so often doesn’t feel <i>too</i> terrible to me, and if you happened to have an HTTP proxy in front of you it might even turn that into just a slow request. But yeah, the second server might be warranted so you can avoid this altogether: just switch which server is the primary and direct all traffic to it, and then freely reboot what is now the standby machine.