Using D and std.ndslice as a Numpy Replacement
The central claim of the post seems summarized by this quote:<p>&gt; For example, when using a non-numpy API or functions that don&#x27;t use Numpy that return regular arrays, you either have to use the normal Python functions (slow), or use np.asarray which copies the data into a new variable (also slow).<p>but I disagree strongly with this.<p>First of all, if there is a common use case for some set of operations that need to be performed on very large data (the type of data you&#x27;d look to NumPy to handle), then generally there is already a subpackage within numpy&#x2F;scipy&#x2F;scikits&#x2F;pandas&#x2F;etc that already deals with that use case and natively handles it with NumPy arrays, with no switching cost to convert back forth between lists or tuples or whatever.<p>And, of course, when a list&#x2F;tuple-heavy API is only meant to deal with small data, it&#x27;s not a problem to use NumPy&#x27;s facilities for converting between ndarray and the builtin array types. In cases where you&#x27;re dealing with a huge breadth of small data, then that casts doubt on whether you should be using NumPy; it wouldn&#x27;t be casting doubt on whatever the other list&#x2F;tuple-heavy API is. And probably parallelization (or even the buffer stuff I mention below) is a fine solution in that case.<p>Second, in a lot of cases you can make use of the Python Buffer Protocol to share the underlying data of a NumPy array without copying it. This won&#x27;t help if some other API expects Python lists or tuples, but the great thing about dynamic typing in Python is that all that really matters is that whatever underlying buffer type you need implements whatever methods that other API expects to call.<p>You can always write your own extension type that adheres to the Buffer Protocol and also provides whatever API is needed to conform to some other library, so the power to create these double-sided adapters (one side sharing data with NumPy, the other side appearing like a drop-in acceptable data structure for the other library API) is very powerful and generic. It might take some getting used to the first few times you do this, but if you use tools like Cython to help, it&#x27;s really quite easy to do, easy to maintain, and solves a surprisingly wide range of NumPy integration problems. In fact, these things generally already exist for most problems you will run into and ultimately they often boil down to simple Cython-based wrappers around C bindings to the other Python API you&#x27;re working with.<p>I would argue that the existence of this Buffer Protocol adapter strategy alone is enough to say that the switching cost to D is virtually never worth it, and still pretty speculative even if you&#x27;re starting a brand new numerical computing project.<p>Finally, most Python libraries that heavily rely on the list or tuple APIs are not meant for large data (those APIs mostly already just use NumPy, as I mentioned, or else they use generators and let the end user decide which array type will eventually be instantiated as the results are consumed). It&#x27;s not common, by intentional design, for list&#x2F;tuple-heavy APIs to need to cope with large data, so when someone says something off the cuff like &quot;What do you do when some library API needs lists and you&#x27;ve got NumPy arrays?&quot; it <i>sounds</i> like a worrisome case, but in practice it&#x27;s really, really uncommon that such a situation arises and no one else ran into it before you and no one has created a NumPy-compliant solution already. It&#x27;s not impossible, of course. Just unlikely, and probably not important enough to use as a basis for language choice, unless you&#x27;re facing a really special sort of API problem.<p>Edit: None of this should read at all as a criticism of the D language or this particular implementation of ndarray data structures. All that stuff is great and anyone wishing to use D absolutely should.<p>I&#x27;m <i>only</i> arguing against the post&#x27;s central thesis insofar as it is used to justify considering D as an alternative to scientific Python. The problem that the post points out already has solutions solely in the Python ecosystem, many projects have handled that problem before, and the problem is pretty rare and esoteric anyway, so it&#x27;s probably not a good thing to use as the basis of an important choice like which language to use, or whether to switch languages.<p>There could be many reasons to prefer D over scientific Python depending on a given use case, and there could be certain situations where switching from Python to D is a good idea. Whatever those cases may be, the central issue of this post, performance degradation caused by NumPy-to-other-API compatibility, is <i>not</i> one of them.