Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this.Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this.Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout retry which is not what you want!Increasing `proxy_connect_timeout` is not an option, because then you risk filling up too many connections in the nginx instance if the upstream server swallows SYN packets or whatnot.The real workaround: Use haproxy. Serioously.