pip 24.1 betas -- help us test a major upcoming change!

The pip team has released pip 24.1b1 which contains a lot of significant improvements and bug fixes.

I’d like to highlight a major change in this blog post: the removal of support for “legacy” versions and dependency specification, or as I like to call it, nonsensical versions and dependencies. We’re releasing this as a beta to get feedback from the community on how this change affects their workflows.

Nonsensical versions and dependencies

For a bunch of historical reasons, pip has allowed many arbitrary strings as versions and dependency specification. There’s so many kinds of bad behaviours that this “feature” has enabled.

My favourite example is one I wrote back in 2022 on the packaging¹ project’s issue tracker:

>>> packaging.version.parse("This is a completely random string")
<LegacyVersion('This is a completely random string')>

Or providing weird version comparisons:

>>> packaging.version.parse("1.0") > packaging.version.parse("99999.0.whatever")
True
>>> packaging.version.parse("1.0")
<Version('1.0')>
>>> packaging.version.parse("99999.0.whatever")
<LegacyVersion('99999.0.whatever')>

Or hiding mistakes in dependency declarations:

>>> all_dependencies = [
...     "package == 1"
...     "another-package",
...     "yet-another-package",
]
>>> [packaging.requirements.Requirement(s) for s in all_dependencies]
[<Requirement('package==1another-package')>, <Requirement('yet-another-package')>]

Over a decade ago, the Python packaging ecosystem adopted PEP 440 which defined a standard for versions and dependency specification. While this standard has been adopted by basically every single tool in the ecosystem, support for “legacy” versions and requirements has been maintained even though it has caused a lot of user confusion and bugs in the Python ecosystem.

Getting rid of the nonsense

Over the course of the last decade, across multiple projects in the Python packaging ecosystem, many PyPA projects have been working to make this behaviour not possible. The package index, https://pypi.org, has not accepted uploads with bad versions for many years now and is also enforcing validity of the requirements on all uploads. The tooling that generate packages (build-backends) also now refuse to generate packages with such bad metadata in them.

This effort has been undertaken by many people across the Python packaging ecosystem, and I’m very grateful for all the work that has gone into this. It regularly surprises me how much work goes into making sure that the foundational pieces of Python’s packaging ecosystem are improving, even absent any institutional investment into it.

This pip release marks the next major step in the direction of improving behaviours in this area: updating the package installer to stop accepting such packages. Starting with this pip release, pip will refuse to install packages with bad versions or dependency specification and will ignore any files with such bad versions or dependency specification during the dependency resolution step.

Where we can’t see the nonsense

While a substantial amount of effort has been put into making sure pip can work correctly when enforcing correctness around these versions and dependency specification, we want to be cautious in rolling out this change. There are many places that Python is used where we can’t access the packages (e.g. private package indexes hosting proprietary code), and thus can’t ensure that packages have appropriate metadata in those spaces.

We’re releasing this as a beta to get feedback from the community on any issues that might arise from this change. This will help us better understand how this change affects user workflows and what actionable guidance can be provided to users in various contexts.

Help us test the changes

We’re looking for feedback from Python users on this beta release. Notably, we’d like for users who are using package indexes other than PyPI to test their workflows against this beta release of pip and provide us with feedback.

You can let us know about issues related to this change on pip’s issue tracker (please check for duplicates before filing, and use reactions rather than saying “me too”!). Please also feel welcome to tell us about the good stuff, like if this release made dependency resolves quicker – we like hearing about that too! :)

Acknowledgements

While there’s a lot of people who have worked on things I mentioned in this blog post, I can’t really list them all here – you know who you are, and thank you for all the work you’ve done toward this as well as the other cool things you’ve worked on.

I do want to thank Paul Moore, Stéphane Bidoul, Richard Si and Pavithra Eswaramoorthy for reviewing various drafts of this post and providing feedback on it.