The database servers powering Let's Encrypt
&gt; We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from &#x2F;proc&#x2F;stat) averaged over 90%<p>This strikes me as odd. In my experience, traditional OLTP row stores are I&#x2F;O bound due to contention (locking and latching). Does anyone have an explanation for this?<p>&gt; Once you have a server full of NVMe drives, you have to decide how to manage them. Our previous generation of database servers used hardware RAID in a RAID-10 configuration, but there is no effective hardware RAID for NVMe, so we needed another solution... we got several recommendations for OpenZFS and decided to give it a shot.<p>Again, traditional OLTP row stores have included a mechanism for recovering from media failure: place the WAL log on separate device from the DB. Early MySQL used a proprietary backup add-on as a revenue model so maybe this technique is now obfuscated and&#x2F;or missing. You may still need&#x2F;want a mechanism to federate the DB devices and incremental volume snapshots are far superior to full DB backup but placing the WAL log on a separate device is a fantastic technique for both performance and availability.<p>The Let&#x27;s Encrypt post does not describe how they implement off-machine and off-site backup-and-recovery. I&#x27;d like to know if and how they do this.