Question: I've heard and read enough programmers firmly advocating automatic tests. According to many, tests are themselves part of a code's functionality, untested code is broken and/or legacy by definition, long-term manual testing is more time-consuming and provides far weaker guarantees against failures than automatic testing... etc.

I'm trying to hobbystically develop a turn-based Pokemon-like game. While talking to a far more experienced developer I said that when I add a new attack I run my game, go into 1v1 combat, use this move and see if it does what its supposed to do. His answer was: *Why won't you instead write a test that goes into a 1v1 combat, uses this move and checks if it does what it's supposed to do? Doing what you're doing manually may be faster than writing tests if you do it once, twice, thrice. But after 10 times? One hundred times?*

I'm thinking about what he said. I'm thinking and I still don't feel convinced. About any other kind of project, perhaps. But a game?

Assume I'm adding lifesteal to a monster's attack. Problem is I'll have to **anyway** boot my game and play it! To see if it feels right, if it plays right, if not for anything else.

Tests on the other hand add maintenance cost: for example, once I (for balancement reasons) change the lifesteal coefficient, I'll have to reflect that change in tests. (Or I won't have to reflect this change in tests if I copy&paste the formula, parametrized by attack strength, lifesteal coefficient, etc, but doesn't this defeat the purpose of testing?)

Automatic tests are supposed to replace manual tests; but given I have to manually play my game anyway, isn't this duplication of work?

Or is my thinking wrong? What am I missing?

Answer: 
Testing is an investment in your future. While an individual test might duplicate some aspect of manual testing you are about to do in any given run of the game, a robust suite of tests can in the long run cover far more scenarios than that small bit of overlapping work (manual testing can also more effectively test more complex scenarios where writing automatic tests might be too difficult).

Writing tests is an overhead, yes. So is maintaining them when the things you are testing change. It's also much harder to retrofit tests onto a big project not built from scratch with that goal in mind, as there are some technical design choices that are less compatible with easy, isolated testing than others.

Whether or not that overhead is worth the benefit depends on you and the scale and scope of your project.

The payoff is that automated tests can be run automatically and require far, far less work on your part that equivalent manual testing coverage would. In your example, sure, you manually run the game and ad-hoc test your change.

But do you do it on every platform you are shipping on? On every build configuration? Automated tests can do so easily, which means you can catch errors that only manifest on one or two platforms/configurations that you don't regularly test more quickly.

Similarly, while you may ad-hoc test that one ability change, do you manually go back and ad-hoc test *every other ability*? Probably not, because you're operating under the assumption that you didn't make a change that could impact those other abilities. But you are a programmer, and therefore, you make mistakes ("bugs") and therefor you *could have* accidentally made your change in a such a way as to have unintentionally broke something you were not expecting. An automated test suite could catch that.

Granted, the value of the test suite is dependent on what you test and how, and there is a point of diminishing returns in the investment (100% test coverage is often impractical, for example). 

For example, it's probably worthwhile to write a test to validate the results of a math function that presumes a left- or right-handed convention, as change in that function to prefer the other convention will probably destabilize a lot and should be flagged. But a test for the outcome of some damage calculation may not be a good candidate for a test, at least not early on when you are iterating rapidly on game balance. That sort of test is perhaps best added later, once balance has hardened a little, to alert you that you may have made a change that has balance implications.
> 
> Problem is I'll have to anyway boot my game and play it! To see if it feels right, if it plays right, if not for anything else.
> 
> 
> 

Yes, you need to do that while you are iterating on that specific situation you are implementing right now. But think ahead a couple years in the future. You might be working on a completely different aspect of the game and accidentally break this situation you though you had done. Are you going to test every single thing in your game after every change? Certainly not manually all by yourself. So how long will it take you to notice the bug? Will you then be able to easily connect it to that specific change you made? But if you have an automated test suit, you *can* easily test your entire codebase after every change.

> 
> Tests on the other hand add maintenance cost: for example, once I (for balancement reasons) change the lifesteal coefficient, I'll have to reflect that change in tests.
> 
> 
> 

Yes, and that's a good thing, because now you have to be aware of every ramification of your balance change.

Does the tutorial still play out the way you intended? Is that one boss fight still winnable with the intended strategy? Is that other boss still immune to life steal? In order to detect those problems you might have to play through your whole game from start to finish. But an automated test can tell you within seconds to minutes.

So an automated test allows you to go through every situation where the outcome is affected and confirm one by one that this is still the outcome you intended.

> 
> Automatic tests are supposed to replace manual tests; but given I have to manually play my game anyway, isn't this duplication of work?
> 
> 
> 

Automatic tests are not supposed to replace manual tests. They are supposed to augment them. While it can not replace the "how does it feel?" playtesting, it can save you a ton of work in regression testing (testing again and again if the things which used to work still work).