What to do about GPU packages on PyPI?
The fundamental issue here is that Python users expect things to be install and go. That has actually never been the case, but it’s got more common over time - before wheels came along, you ended up getting compilation errors all the time from missing packages with pip. Enthought Canopy and later Conda came along and tried to fix that, and then later pip implemented binary wheels, but the wheels are often not a great solution because as someone in this thread said - every package ends up including duplicate stuff, so you can end up with like 5 versions of OpenBLAS and whatever other C&#x2F;C++&#x2F;FORTRAN dependencies your packages depend on. Intel released an MKL package onto PyPi which has reduced a lot of that somewhat but it is obviously a bit problematic, because lots of things now use this instead of OpenBLAS and fail to compile on other architectures (cough, PyTorch, cough).<p>The issue we now have is that scientific <i>users</i> almost always turn to Conda. Scientific authors also have increasingly turned to conda, because it solves a lot of distribution problems for them - it’s a lot easier to say to people ‘conda install -c conda-forge XYZ’ than it is to say ‘install the CUDA toolkit &gt;v10.1 and then download dependency X and then install it from PyPi.’ Look for e.g. at Software Carpentry which aims to teach Python to researchers, or any undergrad course in Python outside of perhaps Comp Sci, and these days they almost always tell people to use Conda. In addition to that, it’s so much easier from a build perspective to add packages to Conda Forge than it is to build the various types of wheels that you need to do for PyPi. The manylinux instructions aren’t even particularly up to date.<p>To me, a lot of this stems back from Guido basically saying to the scientific community ‘it sounds like your problems are different to that of a lot of the Python ecosystem’ but without seeing that the whole compiled dependency thing affects everyone at some point. I’m not sure what the fix is - do we want people to be able to use PyPi to bundle compiled external libraries? It sounds like you’d just end up replicating Conda at the end of the day.<p>For me personally, this fragmentation causes a lot of issues because I help users install packages into our HPC cluster. I need to be able to install things from source and PyPi usually offers that, but increasingly, users bring packages with a chain of dependencies that have no source PyPi package and a long list of conda dependencies. I don’t blame package authors at all - they are doing what is easiest for them and most of their users - but I do think it needs a lot of thought and work by PyPa.