<p dir="auto">When we add new <code class="notranslate">.py</code> files, we are careful to add underscores to the file names to ensure everyone sees at a glance that this is not a new public namespace. However, we have lots of files that are missing that underscore. This is only because of historical reasons; in the early days of SciPy no one paid attention to this. We had a discussion about it once, probably about 9-10 years ago, and decided to not clean things up back then because the concrete benefits were unclear.</p>
<p dir="auto">The public namespaces are documented in <a href="http://scipy.github.io/devdocs/reference/index.html#api-definition" rel="nofollow">http://scipy.github.io/devdocs/reference/index.html#api-definition</a>. No one reads the docs though. This has become much more of a problem recently, because authors of other libraries have started to reimplement parts of the SciPy API, for example:</p>
<ul dir="auto">
<li>CuPy, see <a href="https://docs.cupy.dev/en/stable/reference/scipy.html" rel="nofollow">https://docs.cupy.dev/en/stable/reference/scipy.html</a></li>
<li>JAX, see <a href="https://jax.readthedocs.io/en/latest/jax.scipy.html" rel="nofollow">https://jax.readthedocs.io/en/latest/jax.scipy.html</a></li>
<li><code class="notranslate">dask_image</code>, see <a href="http://image.dask.org/en/latest/api.html" rel="nofollow">http://image.dask.org/en/latest/api.html</a></li>
<li>PyTorch is adding a <code class="notranslate">torch.special</code> namespace that mirrors <code class="notranslate">scipy.special</code>: <a href="https://pytorch.org/docs/master/special.html" rel="nofollow">https://pytorch.org/docs/master/special.html</a></li>
</ul>
<p dir="auto">Those libraries can get it wrong by accident, or they just mirror what our actual filenames are even though we tell them it's not meant to be a public namespace. An example of each:</p>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="447182052" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/10204" data-hovercard-type="issue" data-hovercard-url="/scipy/scipy/issues/10204/hovercard" href="https://github.com/scipy/scipy/issues/10204">#10204</a> discusses <code class="notranslate">dask_image</code> restructuring its namespace to match <code class="notranslate">scipy.ndimage</code></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="851639822" data-permission-text="Title is private" data-url="https://github.com/cupy/cupy/issues/5047" data-hovercard-type="pull_request" data-hovercard-url="/cupy/cupy/pull/5047/hovercard" href="https://github.com/cupy/cupy/pull/5047">cupy/cupy#5047</a> is about <code class="notranslate">scipy.stats.distributions</code></li>
</ul>
<p dir="auto">Long story short: we should clean this up. There's two ways we can go about this:</p>
<ol dir="auto">
<li>don't deprecate anything. Just move <code class="notranslate">somefile.py</code> to <code class="notranslate">_somefile.py</code>, and then add a new <code class="notranslate">somefile.py</code> which reimports the public functions from <code class="notranslate">_somefile.py</code> and adds a big comment with warnings about this not being a public namespace at the top of the file.</li>
<li>do (1), and then deprecate the functions in <code class="notranslate">somefile</code></li>
</ol>
<p dir="auto">I'd suggest to do (2) by default, and only make exceptions in case something is quite heavily used and adding deprecations would be too disruptive.</p>
<p dir="auto">We should prioritize modules that are being duplicated in CuPy, JAX, PyTorch et al.</p>
<p dir="auto">EDIT: a test should also be added that no new private-but-public-looking namespaces are added. The approach can be taken from <a href="https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py">https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py</a></p>
<p dir="auto">EDIT 2: a basic API design principle is: <em>a public object should only be available from one namespace</em>. Having any function in two or more places is just extra technical debt, and with things like dispatching on an API or another library implementing a mirror API, the cost goes up.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from scipy import ndimage
&gt;&gt;&gt; ndimage.filters.gaussian_filter is ndimage.gaussian_filter  # :(
True"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">from</span> <span class="pl-s1">scipy</span> <span class="pl-k">import</span> <span class="pl-s1">ndimage</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">ndimage</span>.<span class="pl-s1">filters</span>.<span class="pl-s1">gaussian_filter</span> <span class="pl-c1">is</span> <span class="pl-s1">ndimage</span>.<span class="pl-s1">gaussian_filter</span>  <span class="pl-c"># :(</span>
<span class="pl-c1">True</span></pre></div>
<p dir="auto">EDIT 3: now that it's decided we're going ahead with this, here is a tracker of which modules are done:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> cluster</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> constants</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> fft</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> fftpack</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> integrate</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> interpolate</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> io</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> linalg</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> misc</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> ndimage</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> odr</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> optimize</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> signal</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> sparse</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> sparse.csgraph</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> sparse.linalg</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> spatial</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> special</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> stats</li>
</ul>
<p dir="auto">Also, we should take over the public API test from NumPy:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked="">  <a href="https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py">https://github.com/numpy/numpy/blob/main/numpy/tests/test_public_api.py</a></li>
</ul>
<p dir="auto">EDIT 4:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> right before closing this issue, we should compare master with 1.7.0 and use some version of <code class="notranslate">sorted([x for x in dir(fitpack) if not (x.startswith('_') or x in ('np', 'warnings'))]</code> to compare for each file/namespace we gave an underscore that everything that looks like a regular SciPy function/object is present in the listings from which we raise deprecation warnings. That way, even if people used clearly private things that were not in <code class="notranslate">__all__</code> in that file, they will still get the warnings rather than a hard break.</li>
</ul>
<p dir="auto">I should mention first that this is an enhancement request rather than a bug issue.</p>
<p dir="auto">The return type of the <a href="https://github.com/scipy/scipy/blob/b5d8bab88af61d61de09641243848df63380a67f/scipy/stats/_stats_py.py#L5944"><code class="notranslate">stats.ttest_ind</code></a> function, testing for equal means two samples, comprises the t statistic and its associated p value. However, the degree of freedom parameter, computed in the function to estimate the p value from the t statistic, is missing in the return type, although it could be useful to the user (it is to me for instance).</p>
<p dir="auto">In the <a href="https://github.com/scipy/scipy/issues/9485" data-hovercard-type="issue" data-hovercard-url="/scipy/scipy/issues/9485/hovercard">distantly related issue</a>, the issue author points out to the <a href="https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/t.test" rel="nofollow">R function performing the t test</a> (factorized in one function instead of 3 functions, depending on function arguments). Efforts could be combined to factorize the functions and enhance the return type by getting inspired by the R function (other values could be useful for the user).</p>