Python packaging

The pip team has released pip 24.1b1 which contains a lot of significant improvements and bug fixes. I’d like to highlight a major change in this blog post: the removal of support for “legacy” versions and dependency specification, or as I like to call it, nonsensical versions and dependencies. We’re releasing this as a beta to get feedback from the community on how this change affects their workflows. Nonsensical versions and dependencies For a bunch of historical reasons, pip has allowed many arbitrary strings as versions and dependency specification....

I maintain quite a few Python packages and they all have development workflows as well as release processes that are different in various ways. This is basically my research document as I am exploring what I want the release process (and supporting development workflow) to look like for the Python packages I maintain, so that I can reduce the overhead caused by the various differences in these projects as well as the need to make all these decisions myself....

Note: I’ve updated this to reflect how this happened, on recommendation from PDM’s author. PDM claims to implement PEP 582. However, if you look at what it implements, it is something completely different from the standard. PDM’s file system structure <root> __pypackages__ 3.10 bottle myscript.py PDM will scan for the __pypackages__ directory up to 5 folders above the “current” one. PEP 582’s file system structure <root> __pypackages__ lib python3.10 site-packages bottle myscript....

My response to the discussion topic posed in Python Packaging Strategy Discussion Part 1 had become quite long, so I decided to move it to write a blog post instead. This post then started absorbing various draft posts I’ve had on this topic since this blog was started, morphing to include my broader thoughts on where we are today. Note: I’ve updated this to cover an aspect of the recent LWN article on the topic as well....

The way the Python packaging community is organised is something that I’ve explained in multiple places, in multiple contexts. I figure that it’ll be useful to actually write it down in a single place, so that I don’t have to repeat myself. The Python Packaging Authority The Python Packaging Authority (PyPA) is a fairly loose group of projects that happen to be related to Python packaging. While the PyPA has a formal governance model, there’s no “oversight” or “enforcement” on projects that are part of the PyPA – they are only required to adopt the PSF Code of Conduct and to be accepted by the existing members....

When installing with pip (or from PyPI in general), wheels are much faster than source distributions, even for pure-Python projects. Packages with native code are a clearer win, because the wheel file will contain pre-compiled binaries for the platform you’re installing on. This means that you don’t need to have a compiler and non-Python build dependencies installed, and you don’t need to wait for the compiler to do its thing....