Answers: <p><a href="https://github.com/pytest-dev/pytest-testinfra" rel="nofollow noreferrer">Testinfra</a> lets you use the <a href="https://docs.pytest.org/" rel="nofollow noreferrer">pytest</a> test framework to interact with and run tests against remote hosts.</p>
  <p>A simple test suite might look like:</p>
  <pre><code>def test_mariadb_query(host):
      '''Connect to a remote host, run a SQL query using the `mariadb`
      CLI, record the result, and verify it matches our expectations.
      '''
  
      expected = &quot;count\n15\n&quot;
      res = host.run(
          '''mariadb example -e &quot;select count from widgets where name='doodad'&quot;'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  </code></pre>
  <p>The above would run the <code>test_mariadb_query</code> test on any hosts specified on the command line, e.g:</p>
  <pre><code>pytest --hosts=ssh://root@node1
  </code></pre>
  <p>You can specify a list of hosts in the test file instead of on the command line:</p>
  <pre><code>import pytest
  import testinfra
  
  testinfra_hosts = [&quot;ssh://root@node1&quot;]
  
  
  def test_mariadb_query(host):
      expected = &quot;count\n15\n&quot;
      res = host.run(
          '''mariadb example -e &quot;select count from widgets where name='doodad'&quot;'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  </code></pre>
  <p>With this, you can just run <code>pytest</code> and it will do what you expect.</p>
  <p>If you have tests that interact with <em>specific</em> hosts, you can create pytest fixtures for those hosts:</p>
  <pre><code>import pytest
  import testinfra
  
  
  @pytest.fixture
  def node1():
      return testinfra.get_host(&quot;ssh://root@node1&quot;)
  
  
  def test_kernel_version(host):
      res = host.run(&quot;uname -r&quot;)
      assert res.stdout.startswith(&quot;5.17&quot;)
  
  
  def test_mariadb_query(node1):
      expected = &quot;count\n15\n&quot;
      res = node1.run(
          '''mariadb example -e &quot;select count from widgets where name='doodad'&quot;'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  </code></pre>
  <p>The above will run the <code>test_kernel_version</code> test against all hosts specified on the command line, but <code>test_mariadb_query</code> will run only against host <code>node1</code>, regardless of the command line.</p>
  <p>The downside to this solution is that it's built on top of a unit testing framework; since tests are supposed to be independent, there's not a built-in mechanism to run them in a specific order. However, there are <a href="https://pytest-ordering.readthedocs.io/en/develop/" rel="nofollow noreferrer">plugins that attempt to address this issue</a>.</p>
  <p>If your ordering requirements can be written as setup/teardown tasks, <a href="https://docs.pytest.org/en/6.2.x/fixture.html" rel="nofollow noreferrer">pytest fixtures</a> may be a viable solution.</p>
  <hr />
  <p>You can use <a href="https://www.ansible.com" rel="nofollow noreferrer">Ansible</a> itself to run your tests, and this gives you complete control over the order of execution of things, but you will generally &quot;fail fast&quot; rather than &quot;fail last&quot; (execution stops with the first failure, rather than continuing and showing you all test failures).</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-10T15:33:36.873Z
Answers: Thank you for the answer! Please note that `pytest-ordering` is no longer maintained, and has been superseded by [`pytest-order`](https://github.com/pytest-dev/pytest-order).
Answers: vpseg
Answers: CC BY-SA 4.0
Answers: 2022-08-10T16:55:28.303Z
Answers: Right, that was't meant as a particular recommendation, just that "there exist things that attempt to solve this problem". I don't have personal experience with either of them -- all of the tests I've need to run haven't been order dependent.
Answers: larsks
Answers: CC BY-SA 4.0
Answers: 2022-08-10T01:36:11.033Z
Answers: 2022-08-10T01:59:14.847Z
Answers: 2022-08-10T01:59:14.847Z
Answers: larsks
Body: <p>I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc. This is done using SaltStack, which allows us to completely set up blank VMs (in development they are built on QEMU using Vagrant, on a linux host). To date, regression testing has been done by hand, and the extent of our automation has been Test Kitchen ensuring that highstates return clean, coupled with some basic InSpec profiles in some cases (is port listening, etc).</p>
  <p>It has become necessary to perform more testing as these systems become more complicated. For example, take a cluster of three VMs, which each have services used by the others, all heavily configured with SaltStack. It seems that the standard advice for testing is to test each system individually with tools like Chef's InSpec, or testinfra. However, I'm looking for a way to test the cluster overall.</p>
  <p>I'm looking for a way to test all my services in different cases. For example, I should be able to kick off a test that:</p>
  <ol>
  <li>Highstates one blank machine</li>
  <li>Uses a client VM to access the public service of the first (i.e. a basic MySQL query)</li>
  <li>Highstates a second server VM</li>
  <li>Tests both services again</li>
  <li>Disables a network interface on one VM to test split-brain functionality</li>
  </ol>
  <p>... etc</p>
  <p>I could do all of this with a long bash or Python script and many SSH commands to each VM, but I would prefer to not reinvent the wheel. Is there a testing framework that can run tests on multiple machines, in order as I define, that allows me to test the whole system in aggregate?</p>
  
Tags: <testing><chef><vagrant><salt>
Title: How does one perform systems testing against multiple interdependent machines?
UserDisplayName: vpseg
ThreadText: # How does one perform systems testing against multiple interdependent machines?
  
  (Asked by: username_0 on 2022-08-04)
  
  I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc. This is done using SaltStack, which allows us to completely set up blank VMs (in development they are built on QEMU using Vagrant, on a linux host). To date, regression testing has been done by hand, and the extent of our automation has been Test Kitchen ensuring that highstates return clean, coupled with some basic InSpec profiles in some cases (is port listening, etc).
  
  It has become necessary to perform more testing as these systems become more complicated. For example, take a cluster of three VMs, which each have services used by the others, all heavily configured with SaltStack. It seems that the standard advice for testing is to test each system individually with tools like Chef's InSpec, or testinfra. However, I'm looking for a way to test the cluster overall.
  
  I'm looking for a way to test all my services in different cases. For example, I should be able to kick off a test that:
  
  1. Highstates one blank machine
  2. Uses a client VM to access the public service of the first (i.e. a basic MySQL query)
  3. Highstates a second server VM
  4. Tests both services again
  5. Disables a network interface on one VM to test split-brain functionality
  
  ... etc
  
  I could do all of this with a long bash or Python script and many SSH commands to each VM, but I would prefer to not reinvent the wheel. Is there a testing framework that can run tests on multiple machines, in order as I define, that allows me to test the whole system in aggregate?
  
  **Tags:** testing, chef, vagrant, salt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-10. Score: 1
  
  [Testinfra](https://github.com/pytest-dev/pytest-testinfra) lets you use the [pytest](https://docs.pytest.org/) test framework to interact with and run tests against remote hosts.
  
  A simple test suite might look like:
  
  ```
  def test_mariadb_query(host):
      '''Connect to a remote host, run a SQL query using the `mariadb`
      CLI, record the result, and verify it matches our expectations.
      '''
  
      expected = "count\n15\n"
      res = host.run(
          '''mariadb example -e "select count from widgets where name='doodad'"'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  
  ```
  
  The above would run the `test_mariadb_query` test on any hosts specified on the command line, e.g:
  
  ```
  pytest --hosts=ssh://root@node1
  
  ```
  
  You can specify a list of hosts in the test file instead of on the command line:
  
  ```
  import pytest
  import testinfra
  
  testinfra_hosts = ["ssh://root@node1"]
  
  def test_mariadb_query(host):
      expected = "count\n15\n"
      res = host.run(
          '''mariadb example -e "select count from widgets where name='doodad'"'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  
  ```
  
  With this, you can just run `pytest` and it will do what you expect.
  
  If you have tests that interact with *specific* hosts, you can create pytest fixtures for those hosts:
  
  ```
  import pytest
  import testinfra
  
  @pytest.fixture
  def node1():
      return testinfra.get_host("ssh://root@node1")
  
  def test_kernel_version(host):
      res = host.run("uname -r")
      assert res.stdout.startswith("5.17")
  
  def test_mariadb_query(node1):
      expected = "count\n15\n"
      res = node1.run(
          '''mariadb example -e "select count from widgets where name='doodad'"'''
      )
      assert res.rc == 0
      assert res.stdout == expected
  
  ```
  
  The above will run the `test_kernel_version` test against all hosts specified on the command line, but `test_mariadb_query` will run only against host `node1`, regardless of the command line.
  
  The downside to this solution is that it's built on top of a unit testing framework; since tests are supposed to be independent, there's not a built-in mechanism to run them in a specific order. However, there are [plugins that attempt to address this issue](https://pytest-ordering.readthedocs.io/en/develop/).
  
  If your ordering requirements can be written as setup/teardown tasks, [pytest fixtures](https://docs.pytest.org/en/6.2.x/fixture.html) may be a viable solution.
  
  ---
  
  You can use [Ansible](https://www.ansible.com) itself to run your tests, and this gives you complete control over the order of execution of things, but you will generally "fail fast" rather than "fail last" (execution stops with the first failure, rather than continuing and showing you all test failures).
  
  > **Comments:**
  > **username_0**: Thank you for the answer! Please note that `pytest-ordering` is no longer maintained, and has been superseded by [`pytest-order`](https://github.com/pytest-dev/pytest-order).  
  > **username_1**: Right, that was't meant as a particular recommendation, just that "there exist things that attempt to solve this problem". I don't have personal experience with either of them -- all of the tests I've need to run haven't been order dependent.  
  
  ---
  
  