<p dir="auto">We have accumulated a <em>lot</em> of dev automation tools, and our homegrown <code class="notranslate">runtests.py</code> CLI which is slightly clumsy. Here is an overview:</p>
<ul dir="auto">
<li><code class="notranslate">runtests.py</code> - and its sibling <code class="notranslate">dev.py</code> for Meson</li>
<li><code class="notranslate">doc/Makefile</code> - build docs and upload them</li>
<li><code class="notranslate">pavement.py</code> - uses Paver for some release tasks</li>
<li><code class="notranslate">tox.ini</code> - contains <code class="notranslate">flake8</code> and <code class="notranslate">pycodestyle</code> config, we don't use Tox itself anymore</li>
<li><code class="notranslate">tools/authors.py</code> / <code class="notranslate">tools/gh_lists.py</code> / <code class="notranslate">tools/download-wheels.py</code> release scripts</li>
<li>tasks for which we have no CLI, just docs that we hope people read before doing the thing they need:
<ul dir="auto">
<li>dev environment setup - <code class="notranslate">apt-get xxx &amp;&amp; pip install yyy</code> or <code class="notranslate">conda env create -n environment.yml</code></li>
<li><code class="notranslate">git</code> - for a lot of things, but with submodules it is getting less intuitive</li>
<li><code class="notranslate">docker</code> - for Gitpod config, see <code class="notranslate">tools/docker_dev</code></li>
<li><code class="notranslate">act</code> - for running GitHub Actions jobs locally</li>
</ul>
</li>
</ul>
<p dir="auto">Many of these things are badly maintained or only half in use. <code class="notranslate">runtests.py</code> has been the best maintained so far, but it's not ideal because task dependencies are manually dealt with in <code class="notranslate">if args.some_task: ...</code> logic.</p>
<p dir="auto"><strong>Proposal</strong>: let's create a new and more ergonomic CLI for all development tools.</p>
<p dir="auto">Ideally it has these properties:</p>
<ul dir="auto">
<li>Completeness: we have all normal/recurring development tasks implemented in it</li>
<li>Conciseness: (for example) <code class="notranslate">do bench</code> instead of <code class="notranslate">python runtests.py --bench</code></li>
<li>Self-documenting: it should be easy to read the CLI implementation and figure out what actual commands are being run (or maybe the CLI can print the command)</li>
<li>Hierarchical: if there are a number of docs-related tasks, they should be discoverable and runnable like: <code class="notranslate">do doc build</code>, <code class="notranslate">do doc show</code>, <code class="notranslate">do doc upload</code>, etc. Same for CI and release tasks; top-level tasks should be for regular contributions/development</li>
<li>Genericity: it should make as few assumptions as possible (e.g., don't assume you're developing in a virtualenv), it should be implemented in a way where the non-SciPy-specific parts can be reused in other projects that use <code class="notranslate">pytest</code>/<code class="notranslate">asv</code>/<code class="notranslate">meson</code>/<code class="notranslate">sphinx</code>, and it should work in harmony with the tools it wraps so one can use those directly</li>
</ul>
<p dir="auto">There will be tools where it doesn't make sense to wrap them (e.g., <a href="https://cli.github.com/">gh</a>) however it would be great if those tools could be made discoverable as well so it doesn't take new contributors years to find out about all of them.</p>
<p dir="auto">I don't have a proposal (yet) for which framework or task runner to use. There are many. It looks like we need a straightforward task runner that understands dependencies between tasks and makes it easy to implement a CLI. Something like:</p>
<ul dir="auto">
<li><a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html" rel="nofollow">Airflow</a></li>
<li><a href="https://docs.celeryproject.org/en/stable/index.html" rel="nofollow">Celery</a></li>
<li><a href="https://pydoit.org/contents.html" rel="nofollow">doit</a></li>
<li><a href="https://www.pyinvoke.org/" rel="nofollow">Invoke</a></li>
</ul>
<p dir="auto">Ones that I know of but aren't good choices: Paver (obsolete), Fabric (superceded by Invoke), Luigi (superceded by Airflow), Tox/Nox (they insist on environment activation/updates for every task).</p>
<p dir="auto">Airflow is on the complicated (but powerful) side, <code class="notranslate">doit</code> is on the simple side. I don't have a lot of experience with any of these tools, so don't have a good sense for the features vs. simplicity tradeoff or other pros/cons. Suggestions very welcome.</p>
<h3 dir="auto">Is your feature request related to a problem? Please describe.</h3>
<p dir="auto">The RegularGridInterpolator <a href="https://github.com/scipy/scipy/blob/main/scipy/interpolate/tests/test_rgi.py">tests</a> often test multiple conditions inside the same test. This underreports the number and type of test failures. For example, <code class="notranslate">test_nearest</code> tests multiple different conditions, each of which should be its own test.</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scipy/scipy/blob/633172ca3241ac06d8122e3a3d090ebe8bef84cd/scipy/interpolate/tests/test_rgi.py#L160-L177">scipy/scipy/interpolate/tests/test_rgi.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 160 to 177
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scipy/scipy/commit/633172ca3241ac06d8122e3a3d090ebe8bef84cd">633172c</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L160" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="160"></td>
          <td id="LC160" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">test_nearest</span>(<span class="pl-s1">self</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L161" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="161"></td>
          <td id="LC161" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">points</span>, <span class="pl-s1">values</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_get_sample_4d</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L162" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="162"></td>
          <td id="LC162" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">interp</span> <span class="pl-c1">=</span> <span class="pl-v">RegularGridInterpolator</span>(<span class="pl-s1">points</span>, <span class="pl-s1">values</span>, <span class="pl-s1">method</span><span class="pl-c1">=</span><span class="pl-s">"nearest"</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L163" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="163"></td>
          <td id="LC163" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sample</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>([<span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">.9</span>, <span class="pl-c1">.9</span>]) </td>
        </tr>

        <tr class="border-0">
          <td id="L164" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="164"></td>
          <td id="LC164" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">wanted</span> <span class="pl-c1">=</span> <span class="pl-c1">1100.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L165" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="165"></td>
          <td id="LC165" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">assert_array_almost_equal</span>(<span class="pl-en">interp</span>(<span class="pl-s1">sample</span>), <span class="pl-s1">wanted</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L166" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="166"></td>
          <td id="LC166" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sample</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>([<span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>]) </td>
        </tr>

        <tr class="border-0">
          <td id="L167" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="167"></td>
          <td id="LC167" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">wanted</span> <span class="pl-c1">=</span> <span class="pl-c1">0.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L168" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="168"></td>
          <td id="LC168" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">assert_array_almost_equal</span>(<span class="pl-en">interp</span>(<span class="pl-s1">sample</span>), <span class="pl-s1">wanted</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L169" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="169"></td>
          <td id="LC169" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sample</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>([<span class="pl-c1">0.</span>, <span class="pl-c1">0.</span>, <span class="pl-c1">0.</span>, <span class="pl-c1">0.</span>]) </td>
        </tr>

        <tr class="border-0">
          <td id="L170" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="170"></td>
          <td id="LC170" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">wanted</span> <span class="pl-c1">=</span> <span class="pl-c1">0.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L171" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="171"></td>
          <td id="LC171" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">assert_array_almost_equal</span>(<span class="pl-en">interp</span>(<span class="pl-s1">sample</span>), <span class="pl-s1">wanted</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L172" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="172"></td>
          <td id="LC172" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sample</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>([<span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>, <span class="pl-c1">1.</span>]) </td>
        </tr>

        <tr class="border-0">
          <td id="L173" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="173"></td>
          <td id="LC173" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">wanted</span> <span class="pl-c1">=</span> <span class="pl-c1">1111.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L174" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="174"></td>
          <td id="LC174" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">assert_array_almost_equal</span>(<span class="pl-en">interp</span>(<span class="pl-s1">sample</span>), <span class="pl-s1">wanted</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L175" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="175"></td>
          <td id="LC175" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sample</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>([<span class="pl-c1">0.1</span>, <span class="pl-c1">0.4</span>, <span class="pl-c1">0.6</span>, <span class="pl-c1">0.9</span>]) </td>
        </tr>

        <tr class="border-0">
          <td id="L176" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="176"></td>
          <td id="LC176" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">wanted</span> <span class="pl-c1">=</span> <span class="pl-c1">1055.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L177" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="177"></td>
          <td id="LC177" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">assert_array_almost_equal</span>(<span class="pl-en">interp</span>(<span class="pl-s1">sample</span>), <span class="pl-s1">wanted</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<h3 dir="auto">Describe the solution you'd like.</h3>
<p dir="auto">I'd like the tests to be parameterised so that each tested condition is its own test. This issue is for someone to go through the above tests and identify which tests can be parameterised and do it.</p>
<h3 dir="auto">Describe alternatives you've considered.</h3>
<p dir="auto">We could leave it as is.</p>
<h3 dir="auto">Additional context (e.g. screenshots, GIFs)</h3>
<p dir="auto"><em>No response</em></p>