Python C API, PyPy and the road into the future
This post highlights an interesting dichotomy in the Python scientific computing community.  Everyone knows that PyPy runs faster than CPython for many common tasks [1].<p>[1] PyPy is on average 7x faster than CPython: <a href="http:&#x2F;&#x2F;speed.pypy.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;speed.pypy.org&#x2F;</a><p>But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy &amp; Scipy, which provide hand-coded C implementations of most matrix-related operations.  Everyone knows that Python for-loops are &quot;slow&quot; [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory &amp; indirection.  So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements.  Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type.<p>[2] Python for-loops are slow: <a href="https:&#x2F;&#x2F;jakevdp.github.io&#x2F;blog&#x2F;2014&#x2F;05&#x2F;09&#x2F;why-python-is-slow&#x2F;" rel="nofollow">https:&#x2F;&#x2F;jakevdp.github.io&#x2F;blog&#x2F;2014&#x2F;05&#x2F;09&#x2F;why-python-is-slow...</a><p>So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy &amp; Scipy operations, which run in compiled C (and additionally, often make use of Blas&#x2F;Atlas&#x2F;LAPACK&#x2F;etc, for even greater speedups in sustained number-crunching).<p>But unfortunately, Numpy &amp; PyPy do not naturally work together.  Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its <i>own</i> C-API [3]!  The official Numpy package doesn&#x27;t work with PyPy; the PyPy project very thoughtfully provides its own PyPy-compatible Numpy package [4].<p>[3] Numpy C-API: <a href="http:&#x2F;&#x2F;docs.scipy.org&#x2F;doc&#x2F;numpy-1.10.0&#x2F;reference&#x2F;c-api.html" rel="nofollow">http:&#x2F;&#x2F;docs.scipy.org&#x2F;doc&#x2F;numpy-1.10.0&#x2F;reference&#x2F;c-api.html</a><p>[4] PyPy-compatible Numpy package: <a href="http:&#x2F;&#x2F;pypy.org&#x2F;download.html#installing-numpy" rel="nofollow">http:&#x2F;&#x2F;pypy.org&#x2F;download.html#installing-numpy</a><p>Furthermore, Numpy is fantastic, but it can&#x27;t offer all possible permutations of matrix operations.  In particular, there are certain image-processing operations that are awkward (and thus, computationally-inefficient) to express using Numpy operations.  So you might ultimately need to go to the Numpy C-API anyway.<p>This is why we created (and, just a few days ago, open-sourced) Pymod: <a href="https:&#x2F;&#x2F;github.com&#x2F;jboy&#x2F;nim-pymod" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jboy&#x2F;nim-pymod</a><p>Pymod is a Nim+Python project that auto-generates all the Python C-API boilerplate &amp; auto-compiles a Python C extension module that wraps the functions in a Nim module.  Pymod enables us to write our Numpy array-processing code in Nim, then compile it (for C++-like speeds) as a well-behaved Python module.  Nim made this very easy, because it compiles to C.<p>After considering our Python-integration options (CPython C-API, `ctypes` and `cffi`), we decided to go with the CPython C-API &amp; Numpy C-API.  We explained this decision in greater detail in the &quot;Implementation details&quot; section of the Pymod README [5]; the executive summary is that `ctypes` seems better suited to wrapping C types in Python, rather than exposing existing Python types in C, while the CPython C-API code could be generated &amp; compiled with the C code that Nim was going to produce anyway.<p>[5] Pymod implementation details: <a href="https:&#x2F;&#x2F;github.com&#x2F;jboy&#x2F;nim-pymod#implementation-details" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jboy&#x2F;nim-pymod#implementation-details</a><p>That said, we would be delighted for Pymod-produced Python modules to be able to run under PyPy.  We&#x27;ve been strongly considering implementing a `cffi` back-end for Pymod, but this won&#x27;t necessarily solve the Numpy issue.  It would be even better if PyPy could support all the CPython C-API extension modules in the world in one fell swoop.