We&#x27;ve been mitigating against this kind of thing with backups at other datacenters or colos for a while. They can be hot standby, cold standby, slightly degraded in performance, whatever. I also recommend the backup be on a different part of the overall power grid in case <i>it</i> cascades in failure. The good colo&#x27;s often have connections to multiple backbones, too, which is extra redundancy.<p>That all assumes there&#x27;s a total and catastrophic failure at main datacenter. If not, there&#x27;s local backup batteries to sustain a smoother, fail-over plus shutdown. Plus, there&#x27;s tricks like isolating the monitoring systems from main systems and power supply using things data diodes over octocouplers or infrared. At least <i>one</i> thing will still be working and feeding you reliable information over a wireless connection after the full failure.<p>NonStop and VMS setups from late 80&#x27;s did better than Github. My own setups involving a minimum of servers plus apps with loose coupling could fail-over in such a situation. So, this just has to be bad architecture caused by who knows what. Examples below of OpenVMS in catastrophic situations having either no downtime or short downtime due to good architecture plus disaster planning.<p>Case study of active-active at World Trade Center
<a href="http:&#x2F;&#x2F;h71000.www7.hp.com&#x2F;openvms&#x2F;brochures&#x2F;commerzbank&#x2F;commerzbank.pdf" rel="nofollow">http:&#x2F;&#x2F;h71000.www7.hp.com&#x2F;openvms&#x2F;brochures&#x2F;commerzbank&#x2F;comm...</a><p>Marketing piece where HP straight-up detonates a datacenter. Guess who was number 1 in recovery. :)
<a href="https:&#x2F;&#x2F;youtu.be&#x2F;bUwthF9x210?t=34s" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;bUwthF9x210?t=34s</a>