Answers: <p>It looks like everybody here assumes an IAC tool always run as expected, but I can tell (from my own experience) this is not always the case, otherwise unit test would actually be useless.</p>
  
  <p>I remember a picture saying "Ansible playbook ran, everything is fine" with a building burning in the background ...</p>
  
  <p>Running a declarative state and having the server in this actual declared state are 2 different thing from my point of view and experience at least.</p>
  
  <p>A broad and heterogeneous environment, spread across multiple DC, reachable through public network etc... There are multiple reason for which a state could not be applied, either fully or partially.</p>
  
  <p>For all these reasons, there is room for unit test allowing one to get a snapshot of the actual server state, which, again, might differ from the aimed state.</p>
  
  <p>So I'd say yes, unit test are useful even in a IAC managed environment. </p>
  
  <p><strong>EDIT</strong></p>
  
  <p>What about the non-regression side of the dev branch of the IaC code base ? so you'd make changes to your code in dev branch and merge it to the prod branch hoping for not breaking everything ? unit test are so valuable, and usually simple to implement I do not see why one would code without this feature.</p>
  
  <p>Reference (in french sorry for that): <a href="https://fr.slideshare.net/logilab/testinfra-pyconfr-2017" rel="nofollow noreferrer">https://fr.slideshare.net/logilab/testinfra-pyconfr-2017</a></p>
  
Answers: CC BY-SA 3.0
Answers: 2018-03-22T14:18:37.837Z
Answers: That'd be at least polite to add some comment with a down vote, don't you think down voter ? Especially on this kind of question where the debate could be very informative or even constructive.
Answers: Pier
Answers: CC BY-SA 3.0
Answers: 2018-03-22T15:12:34.890Z
Answers: I assume the tone of your answer which is quite aggressive to everyone who has interacted with this question added to the fact you're not giving any reference out of your own exemple nor describing any observed malfunction was the reason for the downvoter. You're finally ending up saying the same thing as other answers says, have smoke tests in your provisioning system to ensure the system is OK, which most systems does by failing if they can't converge the system in desired state. Concerning your edit, usually the merge is done after deploying to staging and ensuring the smoke tests pass ...
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-03-22T15:26:38.617Z
Answers: I definitely did not intend to be agressive, I am not using my natibve language here (this is obvious I guess :) ).
Answers: Pier
Answers: CC BY-SA 3.0
Answers: 2018-03-22T16:00:32.797Z
Answers: We may discuss it in [chat] if you wish, I think I saw this presentation or one like it at a devoxx event few years ago. I slightly disagree with calling that unit test, that's more functional tests.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-03-22T17:16:33.447Z
Answers: Some one from my dev team told me the very same this was not unit test, I am no dev so I may be wrong about calling this unit test, definitely
Answers: Pier
Answers: CC BY-SA 3.0
Answers: 2018-03-22T17:25:14.747Z
Answers: For exemple, Chef cookbooks unit tests are done with ChefSpec, just converging the recipe with no-op actions to test the overall behavior. Once this one is ok (code sounds like it will do what is expected), you launch a test machine and run smoke tests (port 80 is listening) then functional tests (Our application API answer X when sent Y), usually smoke tests and functional tests are done with the same tool, in chef ecosystem it's inspec nowadays as test-kitchen already include all those steps as default behavior.
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2018-02-27T14:25:46.173Z
Answers: 2018-03-22T15:34:22.457Z
Answers: 2018-03-22T15:34:22.457Z
Answers: Pier
Answers: <p>IMHO it's rather redundant to write TDD tests for items entirely covered by the IaaC state specification. Doing so implies that the effectiveness of the IaaC is questionable - why would you use it if so?</p>
  
  <p>Looking at it from a different prospective IaaC itself (if/when done properly) incorporates capabilities already tested and considered to be functioning reliably. Which is what makes it attractive and which makes writing TDD matching tests redundant.</p>
  
  <p>For example, an IaaC configuration specifying a system with SSH being installed already incorporates reliable checking for SSH being correctly installed and, if not, mechanisms for properly installing it. Which makes a TDD test for checking if SSH is installed redundant. If your IaaC config also specifies sshd to be started and listening on a specific port then TDD tests for sshd running and listening to the respective port would also be redundant.</p>
  
  <p>Note that my answer is not targeting TDD or any other type of testing that checks if your IaaC configuration as a whole fits a certain purpose. That remains valid and can be used in TDD, CI or similar checks during development of that IaaC specification - I believe @AnoE's answer is applicable in such case.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-12-04T18:28:37.540Z
Answers: Do you apply same thinking to ensuring SSH (or whatever) is enabled on a specific custom port, which is parsed from an external configuration file? That’s not resting on tested units, that’s novel logic.
Answers: Jon Lauridsen
Answers: CC BY-SA 3.0
Answers: 2017-12-04T19:06:46.330Z
Answers: It should be part of IaaC, if it supports it. If not - then this discussion doesn't really apply. Yes, this could slide into how much stuff can be covered by IaaC, but that's a different discussion.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-12-04T19:08:25.370Z
Answers: I'm also assuming we're not in a context where redundant checks are required - in some cases redundant checks following completely different code paths or even infrastructure may be required.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-12-04T16:15:27.203Z
Answers: 2017-12-05T03:07:22.103Z
Answers: 2017-12-05T03:07:22.103Z
Answers: Dan Cornilescu
Answers: <p>In my experience one of the main differences between Dev and Ops are "heavy run time dependencies". Installing packages heavily depends on repositories, networks or valid keys, or lets say instantiating a new cloud server - it depends on your provider resources.</p>
  
  <p>In terms of server-provisioning even if you did not change your provision code, your image will be valid most of the times but sometimes not. So I think testing is really essential for providing working images.</p>
  
  <p>If you go beond single servers things get even worse ... how you will test reachability in whole network-setups? Including DNS-resolution, routing and firewalling? Even if your IaaC providers API works as expected (I've seen wired issues in this area) I really like TDD in this case.</p>
  
  <p>Because I've not found any testing tools in this area, we wrote one in our spare-time: 
  <a href="https://github.com/DomainDrivenArchitecture/dda-serverspec-crate" rel="nofollow noreferrer">https://github.com/DomainDrivenArchitecture/dda-serverspec-crate</a></p>
  
  <p>So I think TDD is really important in the DevOps world!</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-10-26T14:26:22.903Z
Answers: 2018-10-26T14:26:22.903Z
Answers: jerger
Answers: <p>In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.</p>
  
  <p>First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.</p>
  
  <p>Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended <code>apt-get install</code> side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the <code>ps</code> outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.</p>
  
  <p>These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-05-18T11:32:54.647Z
Answers: So after some time, this is exactly the stance I took. Parts that are executed by ansible are not tested by itself, but the side-effects of those actions are tested using `goss`. 
  So for e.g., RPM is installed (ansible) and then tested if the expected default file is put in place, or the service is running and listening for a specific port. I don't want to fix such issue automatically, but be notified and stop the progress. Sure Ansible can test the system for you as well, you just need to be explicit about it, but in our case, we use `goss` to test the behaviour of service within a container
Answers: JackLeo
Answers: CC BY-SA 3.0
Answers: 2017-12-04T18:29:00.683Z
Answers: 2017-12-04T18:29:00.683Z
Answers: AnoE
Body: <p>Infrastructure as code tells us to use tools that automate your builds. Great. Tools like <a href="https://www.ansible.com/" rel="noreferrer">ansible</a>, <a href="https://www.chef.io/" rel="noreferrer">chef</a>, <a href="https://puppet.com/" rel="noreferrer">puppet</a>, <a href="https://saltstack.com/" rel="noreferrer">salt stack</a> and others push us towards writing how infrastructure looks like, while resolving the differences. </p>
  
  <p>In Salt Stack those bits are called <a href="https://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html" rel="noreferrer">states</a>. If state does not match the reality, the tool will resolve it for us. In other words - we are writing a test for our infrastructure and if the test fails, the tool will fix it on its own. At least that's the idea.</p>
  
  <p>XP teaches us to use TDD and the question is if it's applicable to infrastructure? The tooling suggests it is.</p>
  
  <p>I can imagine few type of tests that can be very useful. </p>
  
  <p>We write smoke tests that are bundled with the deployed service to ensure that end-to-end the deployed service works and runs as expected. This would be an API call or/and systemctl check to make sure what we just deployed works. A lot of this functionality can be covered in the same states since tools like ansible have states to make sure a service is running.</p>
  
  <p>There is project <a href="https://molecule.readthedocs.io/en/latest/" rel="noreferrer">Molecule</a> that allows running individual <a href="https://docs.ansible.com/ansible/latest/playbooks_reuse_roles.html" rel="noreferrer">roles</a> (as ansible calls its states) against docker or another temporary virtualisation engine. This forces to decouple roles and allows to execute them in isolation from the playbook while working on them. Tests mostly allow mocking the variables that the role is supposed to work with. Other examples seem like a duplication of the ansible engine though (assert a file belongs to a user...).</p>
  
  <p><a href="https://www.thoughtworks.com/radar/tools" rel="noreferrer">ThoughtWorks</a> tech radar right now praises tools like <a href="https://www.inspec.io/" rel="noreferrer">inspec</a>, <a href="http://serverspec.org/" rel="noreferrer">serverspec</a> or <a href="https://github.com/aelsabbahy/goss" rel="noreferrer">goss</a> for validating that the server meets the spec. But we are writing a spec, aren't we?</p>
  
  <p>So is there a point in further testing infrastructure if we are describing infrastructure in states/roles? I could suspect this becomes more required in larger organisations where one team provides the spec and other follows, or if there is a large set of roles maybe you want to run a subset of those and get a speed benefit from tests? I'm struggling to see why you would write a test if you could have a role/state for the same question in mind.</p>
  
Tags: <deployment><testing><infrastructure-as-code><serverspec><tdd>
Title: Infrastructure as code and TDD
UserDisplayName: JackLeo
ThreadText: # Infrastructure as code and TDD
  
  (Asked by: username_0 on 2017-12-04)
  
  Infrastructure as code tells us to use tools that automate your builds. Great. Tools like [ansible](https://www.ansible.com/), [chef](https://www.chef.io/), [puppet](https://puppet.com/), [salt stack](https://saltstack.com/) and others push us towards writing how infrastructure looks like, while resolving the differences.
  
  In Salt Stack those bits are called [states](https://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html). If state does not match the reality, the tool will resolve it for us. In other words - we are writing a test for our infrastructure and if the test fails, the tool will fix it on its own. At least that's the idea.
  
  XP teaches us to use TDD and the question is if it's applicable to infrastructure? The tooling suggests it is.
  
  I can imagine few type of tests that can be very useful.
  
  We write smoke tests that are bundled with the deployed service to ensure that end-to-end the deployed service works and runs as expected. This would be an API call or/and systemctl check to make sure what we just deployed works. A lot of this functionality can be covered in the same states since tools like ansible have states to make sure a service is running.
  
  There is project [Molecule](https://molecule.readthedocs.io/en/latest/) that allows running individual [roles](https://docs.ansible.com/ansible/latest/playbooks_reuse_roles.html) (as ansible calls its states) against docker or another temporary virtualisation engine. This forces to decouple roles and allows to execute them in isolation from the playbook while working on them. Tests mostly allow mocking the variables that the role is supposed to work with. Other examples seem like a duplication of the ansible engine though (assert a file belongs to a user...).
  
  [ThoughtWorks](https://www.thoughtworks.com/radar/tools) tech radar right now praises tools like [inspec](https://www.inspec.io/), [serverspec](http://serverspec.org/) or [goss](https://github.com/aelsabbahy/goss) for validating that the server meets the spec. But we are writing a spec, aren't we?
  
  So is there a point in further testing infrastructure if we are describing infrastructure in states/roles? I could suspect this becomes more required in larger organisations where one team provides the spec and other follows, or if there is a large set of roles maybe you want to run a subset of those and get a speed benefit from tests? I'm struggling to see why you would write a test if you could have a role/state for the same question in mind.
  
  **Tags:** deployment, testing, infrastructure-as-code, serverspec, tdd
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_6 on 2017-12-04. Score: 6 | Accepted answer
  
  In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.
  
  First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.
  
  Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended `apt-get install` side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the `ps` outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.
  
  These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.
  
  > **Comments:**
  > **username_0**: So after some time, this is exactly the stance I took. Parts that are executed by ansible are not tested by itself, but the side-effects of those actions are tested using `goss`.  So for e.g., RPM is installed (ansible) and then tested if the expected default file is put in place, or the service is running and listening for a specific port. I don't want to fix such issue automatically, but be notified and stop the progress. Sure Ansible can test the system for you as well, you just need to be explicit about it, but in our case, we use `goss` to test the behaviour of service within a container  
  
  ---
  
  ### Answer by username_1 on 2018-02-27. Score: 2
  
  It looks like everybody here assumes an IAC tool always run as expected, but I can tell (from my own experience) this is not always the case, otherwise unit test would actually be useless.
  
  I remember a picture saying "Ansible playbook ran, everything is fine" with a building burning in the background ...
  
  Running a declarative state and having the server in this actual declared state are 2 different thing from my point of view and experience at least.
  
  A broad and heterogeneous environment, spread across multiple DC, reachable through public network etc... There are multiple reason for which a state could not be applied, either fully or partially.
  
  For all these reasons, there is room for unit test allowing one to get a snapshot of the actual server state, which, again, might differ from the aimed state.
  
  So I'd say yes, unit test are useful even in a IAC managed environment.
  
  **EDIT**
  
  What about the non-regression side of the dev branch of the IaC code base ? so you'd make changes to your code in dev branch and merge it to the prod branch hoping for not breaking everything ? unit test are so valuable, and usually simple to implement I do not see why one would code without this feature.
  
  Reference (in french sorry for that): <https://fr.slideshare.net/logilab/testinfra-pyconfr-2017>
  
  > **Comments:**
  > **username_1**: That'd be at least polite to add some comment with a down vote, don't you think down voter ? Especially on this kind of question where the debate could be very informative or even constructive.  
  > **username_2**: I assume the tone of your answer which is quite aggressive to everyone who has interacted with this question added to the fact you're not giving any reference out of your own exemple nor describing any observed malfunction was the reason for the downvoter. You're finally ending up saying the same thing as other answers says, have smoke tests in your provisioning system to ensure the system is OK, which most systems does by failing if they can't converge the system in desired state. Concerning your edit, usually the merge is done after deploying to staging and ensuring the smoke tests pass ...  
  > **username_1**: I definitely did not intend to be agressive, I am not using my natibve language here (this is obvious I guess :) ).  
  > **username_2**: We may discuss it in [chat] if you wish, I think I saw this presentation or one like it at a devoxx event few years ago. I slightly disagree with calling that unit test, that's more functional tests.  
  > **username_1**: Some one from my dev team told me the very same this was not unit test, I am no dev so I may be wrong about calling this unit test, definitely  
  > **username_2**: For exemple, Chef cookbooks unit tests are done with ChefSpec, just converging the recipe with no-op actions to test the overall behavior. Once this one is ok (code sounds like it will do what is expected), you launch a test machine and run smoke tests (port 80 is listening) then functional tests (Our application API answer X when sent Y), usually smoke tests and functional tests are done with the same tool, in chef ecosystem it's inspec nowadays as test-kitchen already include all those steps as default behavior.  
  
  ---
  
  ### Answer by username_3 on 2017-12-04. Score: 1
  
  IMHO it's rather redundant to write TDD tests for items entirely covered by the IaaC state specification. Doing so implies that the effectiveness of the IaaC is questionable - why would you use it if so?
  
  Looking at it from a different prospective IaaC itself (if/when done properly) incorporates capabilities already tested and considered to be functioning reliably. Which is what makes it attractive and which makes writing TDD matching tests redundant.
  
  For example, an IaaC configuration specifying a system with SSH being installed already incorporates reliable checking for SSH being correctly installed and, if not, mechanisms for properly installing it. Which makes a TDD test for checking if SSH is installed redundant. If your IaaC config also specifies sshd to be started and listening on a specific port then TDD tests for sshd running and listening to the respective port would also be redundant.
  
  Note that my answer is not targeting TDD or any other type of testing that checks if your IaaC configuration as a whole fits a certain purpose. That remains valid and can be used in TDD, CI or similar checks during development of that IaaC specification - I believe @username_6's answer is applicable in such case.
  
  > **Comments:**
  > **username_4**: Do you apply same thinking to ensuring SSH (or whatever) is enabled on a specific custom port, which is parsed from an external configuration file? That’s not resting on tested units, that’s novel logic.  
  > **username_3**: It should be part of IaaC, if it supports it. If not - then this discussion doesn't really apply. Yes, this could slide into how much stuff can be covered by IaaC, but that's a different discussion.  
  > **username_3**: I'm also assuming we're not in a context where redundant checks are required - in some cases redundant checks following completely different code paths or even infrastructure may be required.  
  
  ---
  
  ### Answer by username_5 on 2018-10-26. Score: 1
  
  In my experience one of the main differences between Dev and Ops are "heavy run time dependencies". Installing packages heavily depends on repositories, networks or valid keys, or lets say instantiating a new cloud server - it depends on your provider resources.
  
  In terms of server-provisioning even if you did not change your provision code, your image will be valid most of the times but sometimes not. So I think testing is really essential for providing working images.
  
  If you go beond single servers things get even worse ... how you will test reachability in whole network-setups? Including DNS-resolution, routing and firewalling? Even if your IaaC providers API works as expected (I've seen wired issues in this area) I really like TDD in this case.
  
  Because I've not found any testing tools in this area, we wrote one in our spare-time:
  <https://github.com/DomainDrivenArchitecture/dda-serverspec-crate>
  
  So I think TDD is really important in the DevOps world!
  
  
  ---
  
  