We killed our end-to-end test suite
This is a very bad list of complaints and it actually makes me angry to read it.<p>&gt; Engineers had to wait more and more to get feedback from this long-running suite<p>So speed up your tests. Run them in parallel. Find better frameworks for running tests.<p>&gt; Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative;<p>Fix your flaky tests! Why anyone just accepts that &quot;Oh, sometimes that test fails and we have to restart everything&quot; is beyond me. Root cause the problem and FIX IT.<p>&gt; Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge;<p>Tests should not rely on pre-existing state. Have a setup phase for each test that creates new data in the state you want it to be in. As it makes this data, also note down a reference to it with a Time To Live so that a follow-up process can clean up this unneeded data.<p>&gt; Test failures were very hard to debug<p>That&#x27;s not the fault of the tests, that&#x27;s the fault of a complex system that is hard to debug. Improve your tracing between services.<p>&gt; Queueing of commits in the End-to-End suite resulted in less frequent deployments<p>There are well known solutions to this problem. Lots of companies have overcome this already.<p>&gt; Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug<p>If your tests have false-positives, you need to adjust your tests. Accepting that the test failed but there isn&#x27;t a problem, and then not fixing the reason the test failed means that you don&#x27;t have reliable tests.<p>&gt; Bugs were still being found in production<p>Bugs will always make it to production. But after you fix a bug, you <i>write a test</i> so that this bug cannot happen again. Over time, the number of possible bugs that can make it to production shrinks.<p>And lastly:<p>&gt; The main difference to the old E2E is that they encompass only a subset of services and don’t require spinning a production-like environment (the services run in memory on a single JVM and HTTP&#x2F;Kafka communication is replaced by in-process communication). They are used in specific flows that we find too critical to only rely on Contract Tests.<p>Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won&#x27;t in your fake infra?