Best practices for using GitHub Actions to deploy to PyPI/Anaconda

General
,
1

I’m pretty new to setting up CI/CD workflows via GitHub Actions, so I thought this would be a good place to get a sense check on whether I’m doing “the right thing” (or not!).

I have a fairly simple task: publish a pure Python package to PyPI and Anaconda.org. This is the workflow I have created:

It works fine (after a fair bit of experimentation), but feels a bit long-winded/faffy for what I thought would be a common enough task. This might be because I’ve constrained myself in a few of (possibly misguided) ways:

  • I’m building the Python package using Poetry. I should have listened to @mcflugen’s advice (Python packaging references · csdms · Discussion #23 · GitHub) that Poetry is really slow in CI/CD environments…
  • I’m building the Conda package using rattler-build. This is a tonne faster than conda-build, but means I can’t use this useful looking action: Build and upload conda packages · Actions · GitHub Marketplace · GitHub
  • I decided to implement the two builds and two deploys into separate jobs, but this gives a bit of overhead in setting up the environment for each. I also hit issues installing anaconda-client using pip and hence I’m using micromamba there instead. So I’ve ended up with an eclectic mix of “setup” stuff across the jobs (installing Python, installing Poetry, installing Micromamba…).

If anyone has any thoughts on how this could be better implemented, I would be grateful to know!

Also, is there any way of testing workflows before pushing them to GitHub? It is a bit annoying having to pollute my commit history with a load of broken commits whilst I figure things out! I tried act to run the workflow locally but got a load of Docker permission errors and gave up… Maybe I should have used a temporary branch.

2

The one I worked out for our Python TopoToolbox package looks quite similar to yours:

so I too would be interested if there is a better way to do it. Though I often prefer more verbose but explicit steps over “use some action published by a random user.”

I usually work on a feature branch and then set

on: 
  push: 
    branches: ["feature-branch-name"]

at the top, push a bunch of stuff to try it out, and then (remembering to switch feature-branch-name for main at the last minute!) rebase the feature branch before merging it into main to clean up the commit history. This works but is probably not optimal. I spent a good chunk of the last week waiting for GitHub Actions to run while I tweaked things in one of our workflows.

1 Like
3

Thanks, that’s good advice on branching and rebasing to clean up the commit history. I might give that ago!

As an aside, I seem to be slightly vindicated in using rattler-build instead of conda-build as it looks like Conda Forge are moving towards using the newer Conda recipe format that rattler-build uses: Announcing the new recipe format on conda-forge | conda-forge | community-driven packaging for conda

4

I just used this method for another repo and it worked a charm :slight_smile:

5

Over time I have used a few ways to push new package versions to PyPI, but nowadays I would recommend to follow the official python packaging guide on this topic.

As for conda, I only have experience with publishing to conda-forge. This has been quite easy using “grayskull” and the conda-forge guide . For these workflows I don’t need to build anything or maintain anything on the original repository. The recipe lives on the conda-forge github org (e.g.; GitHub - conda-forge/era5cli-feedstock: A conda-smithy repository for era5cli.).

In modern Python packaging python -m build should be sufficient, the actual build system is defined in the pyproject.toml file (PEP-517). You can do this with poetry too.
I personally use hatch, which uses python established standards instead of custom solutions.

2 Likes
6

Thanks, useful info!

Ultimately I think we will move to conda-forge - perhaps I should have gone with this first instead of writing the workflow to push to another channel!

I always forget this - thanks!

I might be tempted to try, though having tried Poetry before moving onto Pixi/rattler, I might give it a while before introducing yet another thing to learn :laughing: Having said that, I have just noticed that Pixi now supports PEP517 backends like hatchling…