How “let it fail” leads to simpler code
One of the pieces of software I&#x27;m most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect.<p>The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn&#x27;t try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric.<p>The system fails all the time at this scale, but heals itself pretty quickly. In over 1 year of operation it hasn&#x27;t caused a single incident, and it has survived all outages.