Testing your Python package releases

What follows is my adventure debugging a release process and discovering that things could be improved.

The problem

The Django Debug Toolbar uses a release.yml script from Django Commons which is effectively the example for the Trusted Publisher Workflow.

This works as follows:

1. [manual] The version numbers are updated and committed.
2. [manual] A tag is created with the new version number
3. [manual] The commit is pushed to origin/main
4. [manual] The tag is pushed to origin
5. [GitHub Action] The wheel and tarball are built
6. [GitHub Action] If the push was a tag, the wheel and tarball are uploaded to Test PyPI
7. [manual] Approval is given to release to PyPI
8. [GitHub Action] If the push was a tag, the wheel and tarball are uploaded to PyPI
9. [GitHub Action] The distributions are signed, a GitHub release is created and the distributions are uploaded
10. People can use the new version

With the 5.0.0 release for the Django Debug Toolbar, there was a problem with the uploading to Test PyPI and PyPI. To make matters worse, I manually created a GitHub release for the tag and that release tagged several contributors. So I was left in a place where I had a tag and released published on GitHub, but no matching version on PyPI. Additionally, I wasn’t sure how to fix the problem.

This entire process revealed how brittle the toolbar’s release workflow is and how useless our integration with Test PyPI is¹.

What I needed was the ability to test the action that publishes packages to Test PyPI. However, in the current state, this meant I needed to create a new tag. I wasn’t entirely comfortable with that approach because the toolbar doesn’t have a history of test tags.

What is needed for testing releases?

The main thing that’s needed is the ability to test the release process regularly and on demand. Testing the release process as you’re releasing the software is a recipe for pain.

Ideally, there would be a CI process that runs once per week that would make a release to Test PyPI. This would uncover any problems with dependency changes.

Next, there would be a CI process that could run as needed that would make an additional release to Test PyPI. This would allow for testing immediate changes to your own release process.

What blocks testing releases?

The current infrastructure can only release packages with version numbers from your pyproject.toml / setup.py.

However, it appears that python -m build does support specifying a configuration setting which could likely be fetched within the code to generate a version. This may be the path forward.

How did I solve my original problem?

For those wondering how I solved the problem, well it was a mess.

1. I downloaded the distributions compiled by GitHub and uploaded them to PyPI via twine
2. I changed the release.yml definition to run on any push to main and always upload to Test PyPI
3. I upgraded the Trusted Publisher action version (this fixed the underlying bug)
4. I pushed those commits to main and confirmed the fix worked
5. I incremented the version and added a tag, then pushed those to main
6. I discovered that the PyPI release process doesn’t run here since it would only run on tag pushes, but the entire action only ran on pushes to main
7. I deleted the tag from origin
8. I reverted the release.yml to the original version, but with the Trusted Publisher action version update
9. I committed the changes, recreated the 5.0.1 tag and pushed to both to origin
10. I confirmed the entire release process worked²