Details of the Cloudflare outage on July 2, 2019
TL;DR<p>Root cause was a bad regex generating excessive backtracking using all CPU on nodes.<p>The meta-cause is the process workflow:<p>> But, by design, the WAF doesn’t use this process because of the need to respond rapidly to threats.<p>The above is in reference to how WAF deployment doesn't use the graduated DOG(fooding)/(guinea)PIG/canary flow.<p>> We responded quickly to correct the situation and are correcting the process deficiencies that allowed the outage to occur [...]<p>Live and learn. Not all WAF deployments are emergency rollouts.