Scribbled Notes

Alrighty! Let’s start doing this again. ...

I’m actually really good at procrastinating things when I’m uncomfortable. - me, back in Feb 2020 ...

This is an attempt to summarize the broader software architecture around dependency resolution in pip and how testing is being done around this area. The motivation behind writing this, is to make sure all the developers working on this project are on the same page, and to have a written record about the state of affairs. Architecture The “legacy” resolver in pip, is implemented as part of pip’s codebase and has been a part of it for many years. It’s very tightly coupled with the existing code, isn’t easy to work with and has severe backward compatibility concerns with modifying directly – which is why we’re implementing a separate “new” resolver in this project, instead of trying to improve the existing one. ...

I’m trying to post these roughly once a month. Here’s the January post. I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver. Work I did (Jan 6 - Feb 5) Technical Co-worked with another developer, in person, for 1 week, on pip! Triaged pip’s issue tracker (a lot). Spend some time improving pip’s test suite infrastructure. Investigated Python 2 usage, to identify anomalies. Helped with virtualenv 20.0 release (kinda!). Invested effort to improve pip’s test suite Helped aggregate test cases for pip’s next generation resolver. Communication Managed the pip 20.0 release fiasco. Helped the UX folks get started with working on pip. Additional notes on challenges January has been a very productive month. ...

I’m trying to post these roughly once a month. Here’s the December post. I am working on open source projects, as part of an internship at FOSSEE and as a part of grant-funded work on pip’s dependency resolver. Work I did (Dec 11 - Jan 5) Technical Wrore and adopted vendoring for use in pip. Initiated conversation for pipenv to switch to it too. Made progress toward TOML 1.0, via useful clarifications in the specification Continued tackling pip’s build logic refactoring. Refactoring of pip’s wheel building logic. Designing a “package builder” library – figuring out what Python Packaging’s “build” step should look like to end users. Started reviewing virtualenv’s rewrite. PR Reviews: ~25, including improvements to pip’s installation logic, better wheel-related error messages and more. Communication A few release manager-y things: (I’ll be the release manager for pip 20.0) Identified tasks that need to happen prior to pip’s release. Discussions on discuss.python.org: Giving pip an API https://discuss.python.org/t/2833 Python Packaging in GSoC 2020 https://discuss.python.org/t/2830 PyPA governance PEP https://discuss.python.org/t/2619 Discussions on issue trackers: New Resolver: technical choices https://github.com/pypa/pip/issues/7406 Switch to a dedicated tool for vendoring pip’s dependencies https://github.com/pypa/pip/pull/7485 Provided helper/starter for pip’s UX review work. Onboarded a new triager to pip: Welcome Stéphane Bidoul! :) Additional notes on challenges December was fairly productive, until the end of the month, when the holidays came in and I also wasn’t in the best of health. ...

I’m trying to post these roughly once a month. Here’s the November post. This update has been a little late, since I have been traveling and recovering from an illness. I also have good news: I will be working on open source projects related to Python Packaging (full time) starting early Dec 2019 till May/Jun 2020. Work I did (Nov 6 - Dec 10) Technical Worked to separate pip’s dependency vendoring logic from rest of pip This would enable other projects that use similar mechanisms (eg. pipenv) Made some progress for preparing for TOML’s 1.0 release Communication Finalized my internship, and got the necessary approvals at college PyConf Hyderabad 2019! (yay!) I gave a talk: Rethinking Python Packaging – a thought experiment. I ran a sprint! ~5 new contributors worked on pip! Initiated a [discussion for Python Packaging at PyCon US and in GSoC 2020. Additional notes on challenges As I’d anticipated, November has been low productivity. I didn’t have much time to put toward working on “Open Source stuff”, due to various other commitments – broadly: exam preparation, attending a family wedding and talk preparation for PyConf Hyderabad. ...

I’m trying to post these roughly once a month. Here’s the October post. Work I did (Oct 6 - Nov 5) Technical Wrapped up refactoring metadata generation within pip. (yay!) Started refactoring wheel building logic within pip. A few minor fixes/cleanup within pip. Helped out with pip 19.3’s release. Significantly simplified pip’s release processes. Reviewed >50 PRs. Fixed a “major” regression in pip’s detection of virtual environments. Got some work done, progressing closer to TOML 1.0.0. Communication ...

bugs.python.org is the issue tracker for CPython. “bpo”, as it is commonly referenced, is where implementation bugs, smaller feature requests, and documentation issues are tracked as part of Python’s development. PEP 581 proposes sunsetting bugs.python.org, in favor of GitHub issues. “Why not focus on improving Roundup / bpo?” section made me wonder: What is needed to improve Roundup / bpo? This blog post is my attempt at summarizing the state of bpo and how it could be improving in the future, based on a discussion with folks maintaining bpo. ...

I’m trying to post these roughly once a month. Here’s the September post. Work I did (Sept 6 - Oct 5) Technical Figured out a rough structure for approaching pip’s build logic refactor. Discussed and sketched a rough “goal” model for pip’s build logic, to adopt after initial cleanups and decoupling refactor work. Made significant progress on refactoring pip’s metadata handling and generation logic. (yay!) Identified a quirk in how pip’s current resolution logic, which makes it conceptually intertwined with pip’s index interaction logic (sigh). out-of-scope for the ongoing “build logic refactor”, but we do need to resolve this at some point. Revamped pip’s linting pipeline to be… simpler (i.e. better). Multiple miscellaneous fixes/cleanup to pip. A lots of PRs reviewed (>80). Reviewed and merged ABNF-related PRs, for TOML. Sunk a good day to debugging 500s on GitHub (it was due to emojis). :( Communication ...

I’m trying to post these roughly once a month. Here’s the August post which includes the work I was planning to do this month. Work I did (Aug 6 - Sept 5) Technical Began working on splitting the code for handling of the two kinds of source distributions. Wrap up pip 19.2, with pip 19.2.3 release, unblocking Python 3.8’s final beta. Hooray! Fix documentation builds for pip, on ReadTheDocs – this took way longer to figure out than I’d expected. Multiple miscellaneous fixes that I could do in gaps between classes. Lots of PRs reviewed (30+) In August, I wanted to get further on separating “prepare distribution” logic from the resolution logic and on turning distribution-type-specific build code into dedicated modules within the new “distributions” sub-package in pip. I didn’t get there. ...