Question: This is part of my quest how to test my tests and test data. I looked at the techniques that would help improve my test cases. While *code/branch coverage* can tell me whether my test suites cover my code base sufficiently, it will not tell me whether my test oracles are able to detect all faults. So, here comes [mutation testing](http://en.wikipedia.org/wiki/Mutation_testing) with help: 

> 
> *Mutation testing* involves modifying a program's source code or byte
>  code in small ways. A test suite that does not detect and reject the
>  mutated code is considered defective.
> 
> 
> 

In extreme case it will detect useless assertions `1==1` in your tests :-)

There are [tools](https://stackoverflow.com/questions/246495/what-mutation-testing-frameworks-exist) for performing mutation testing (in Java and other programming languages), some well integrated with xUnit frameworks. However, configuring those tools and interpreting their results seems time-consuming to me, and the tools [might return many false negatives](http://dl.acm.org/citation.cfm?id=261062.261067) (called equivalent mutants) that I must filter out manually. Hence, I'm curious whether it is worth investing more time on learning that technique. 

1. Have you found those tools, or mutation testing in general, *useful* for improving your tests?
2. What kind of *deficiencies* have you found in your tests with mutation testing?

I've read about some [hands-on experiences](https://stackoverflow.com/questions/242650/is-mutation-testing-useful-in-practice) with this technique but no details what deficiencies were found in real test cases and how serious they were.

Answer: 
Mutation testing is generally interesting from an academic perspective. I have never seen or heard of anyone using mutation testing on software that is delivered to customers on a schedule. The time and expense involved in mutation testing is huge compared to the perceived value returned. In complex projects with several developers contributing to the product mutation testing of even a small component can block continuous integration cycles. In general, mutation testing is probably one of the most expensive solutions to a goal of improving test case effectiveness (assuming that you have a problem of ineffective test cases).

Mutation testing may be useful in training, or smaller projects with limited participants and no timelines, but it is generally more complicated than flipping a single bit and analyzing the results of a subset of tests. 

Also, it is a faulty assumption that an oracle (human or automated) will detect all faults. Automated oracles can only detect problems they are programmed to check for.
Mutation testing has been used in some small systems e.g.: see article ["An intuitive approach to determine test adequacy in safety-critical software"](http://se.kaist.ac.kr/wp-content/uploads/2013/03/An-Intuitive-Approach-to-Determine-Test-Adequacy-in-Safety-critical-Software.pdf). Here's the abstract:

> 
> Safety-critical software must adhere to stringent quality standards
> and is expected to be thoroughly tested. However, exhaustive testing
> of software is usually impractical. The two main challenges faced by a
> software testing team are generation of effective test cases and
> demonstration of testing adequacy.
> 
> 
> This paper proposes an intuitive and conservative approach to
> **determine the test adequacy in safety-critical software**. The approach is demonstrated through a case study: the core temperature
> monitoring system of a nuclear reactor. We combine conservative test
> coverage of unique execution path test cases, and **the results from
> mutation testing to determine the test adequacy**.
> 
> 
> Although mutation testing is a powerful technique, the **difficulty in
> identifying equivalent mutants has limited its practical utility**. To
> gain confidence on the computed test adequacy:
> 
> 
> 1. faults during mutation testing must be induced at all possible execution paths of the code,
> 2. properties of unkilled mutants must be studied, and
> 3. all equivalent mutants must be detected.
> 
> 
> In this regard;
> 
> 
> * results of static, dynamic and **coverage analysis of the mutants** is presented, and
> * a **technique to identify the likely equivalent mutants** is proposed.
> 
> 
>