Open Source Updates

I’m trying to post these roughly once a month. Here’s the January post. I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver. Work I did (Jan 6 - Feb 5) Technical Co-worked with another developer, in person, for 1 week, on pip! Triaged pip’s issue tracker (a lot). Spend some time improving pip’s test suite infrastructure. Investigated Python 2 usage, to identify anomalies. Helped with virtualenv 20.0 release (kinda!). Invested effort to improve pip’s test suite Helped aggregate test cases for pip’s next generation resolver. Communication Managed the pip 20.0 release fiasco. Helped the UX folks get started with working on pip. Additional notes on challenges January has been a very productive month. ...

I’m trying to post these roughly once a month. Here’s the December post. I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver. Work I did (Dec 11 - Jan 5) Technical Wrore and adopted vendoring for use in pip. Initiated conversation for pipenv to switch to it too. Made progress toward TOML 1.0, via useful clarifications in the specification Continued tackling pip’s build logic refactoring. Refactoring of pip’s wheel building logic. Designing a “package builder” library – figuring out what Python Packaging’s “build” step should look like to end users. Started reviewing virtualenv’s rewrite. PR Reviews: ~25, including improvements to pip’s installation logic, better wheel-related error messages and more. Communication A few release manager-y things: (I’ll be the release manager for pip 20.0) Identified tasks that need to happen prior to pip’s release. Discussions on discuss.python.org: Giving pip an API https://discuss.python.org/t/2833 Python Packaging in GSoC 2020 https://discuss.python.org/t/2830 PyPA governance PEP https://discuss.python.org/t/2619 Discussions on issue trackers: New Resolver: technical choices https://github.com/pypa/pip/issues/7406 Switch to a dedicated tool for vendoring pip’s dependencies https://github.com/pypa/pip/pull/7485 Provided helper/starter for pip’s UX review work. Onboarded a new triager to pip: Welcome Stéphane Bidoul! :) Additional notes on challenges December was fairly productive, until the end of the month, when the holidays came in and I also wasn’t in the best of health. ...

I’m trying to post these roughly once a month. Here’s the November post. This update has been a little late, since I have been traveling and recovering from an illness. I also have good news: I will be working on open source projects related to Python Packaging (full time) starting early Dec 2019 till May/Jun 2020. Work I did (Nov 6 - Dec 10) Technical Worked to separate pip’s dependency vendoring logic from rest of pip This would enable other projects that use similar mechanisms (eg. pipenv) Made some progress for preparing for TOML’s 1.0 release Communication Finalized my internship, and got the necessary approvals at college PyConf Hyderabad 2019! (yay!) I gave a talk: Rethinking Python Packaging – a thought experiment. I ran a sprint! ~5 new contributors worked on pip! Initiated a [discussion for Python Packaging at PyCon US and in GSoC 2020. Additional notes on challenges As I’d anticipated, November has been low productivity. I didn’t have much time to put toward working on “Open Source stuff”, due to various other commitments – broadly: exam preparation, attending a family wedding and talk preparation for PyConf Hyderabad. ...

I’m trying to post these roughly once a month. Here’s the October post. Work I did (Oct 6 - Nov 5) Technical Wrapped up refactoring metadata generation within pip. (yay!) Started refactoring wheel building logic within pip. A few minor fixes/cleanup within pip. Helped out with pip 19.3’s release. Significantly simplified pip’s release processes. Reviewed >50 PRs. Fixed a “major” regression in pip’s detection of virtual environments. Got some work done, progressing closer to TOML 1.0.0. Communication ...

I’m trying to post these roughly once a month. Here’s the September post. Work I did (Sept 6 - Oct 5) Technical Figured out a rough structure for approaching pip’s build logic refactor. Discussed and sketched a rough “goal” model for pip’s build logic, to adopt after initial cleanups and decoupling refactor work. Made significant progress on refactoring pip’s metadata handling and generation logic. (yay!) Identified a quirk in how pip’s current resolution logic, which makes it conceptually intertwined with pip’s index interaction logic (sigh). out-of-scope for the ongoing “build logic refactor”, but we do need to resolve this at some point. Revamped pip’s linting pipeline to be… simpler (i.e. better). Multiple miscellaneous fixes/cleanup to pip. A lots of PRs reviewed (>80). Reviewed and merged ABNF-related PRs, for TOML. Sunk a good day to debugging 500s on GitHub (it was due to emojis). :( Communication ...

I’m trying to post these roughly once a month. Here’s the August post which includes the work I was planning to do this month. Work I did (Aug 6 - Sept 5) Technical Began working on splitting the code for handling of the two kinds of source distributions. Wrap up pip 19.2, with pip 19.2.3 release, unblocking Python 3.8’s final beta. Hooray! Fix documentation builds for pip, on ReadTheDocs – this took way longer to figure out than I’d expected. Multiple miscellaneous fixes that I could do in gaps between classes. Lots of PRs reviewed (30+) In August, I wanted to get further on separating “prepare distribution” logic from the resolution logic and on turning distribution-type-specific build code into dedicated modules within the new “distributions” sub-package in pip. I didn’t get there. ...

I’m going to try to post these roughly once a month. Here’s the late June post. Work I was planning to do Technical Clearly determine what parts of the build system are intertwined with pip’s resolution logic. Communication Get confirmation from my college about required dates/duration of internships in final semester. Work I did (24 June - 6 Aug) Technical Identified portion of build system intertwined with pip’s resolver: “Get abstract distribution”: building installable artifacts is intertwined with pip’s resolver. Specifically, the resolver currently passes information about how it would handle a distribution, to the “preparer” that fetches and builds it. Marker evaluation and handling is done by “RequirementSet” and not the resolver. Started moving code for pip’s handling of installable artifacts into a pip._internal.distributions package. ...

I’m glad you asked. What’s up? Over the past few months, I’ve worked on a prototype for a resolver in pip, spent some time planning what rolling out a new dependency resolver in pip might look like, and continue to work on pip’s build logic to decouple it from the dependency resolution logic. The prototype has helped me better understand the relationships and contracts between various portions of the codebase. This understanding has led me to conclude that it is a worthwhile investment to better decouple the build logic from the resolution logic, then introduce the abstractions defined in resolvelib/zazo, and then proceed to adopting better dependency resolution. ...