Best practices for using GitHub Actions to deploy to PyPI/Anaconda

General
,
1

I’m pretty new to setting up CI/CD workflows via GitHub Actions, so I thought this would be a good place to get a sense check on whether I’m doing “the right thing” (or not!).

I have a fairly simple task: publish a pure Python package to PyPI and Anaconda.org. This is the workflow I have created:

It works fine (after a fair bit of experimentation), but feels a bit long-winded/faffy for what I thought would be a common enough task. This might be because I’ve constrained myself in a few of (possibly misguided) ways:

  • I’m building the Python package using Poetry. I should have listened to @mcflugen’s advice (Python packaging references · csdms · Discussion #23 · GitHub) that Poetry is really slow in CI/CD environments…
  • I’m building the Conda package using rattler-build. This is a tonne faster than conda-build, but means I can’t use this useful looking action: Build and upload conda packages · Actions · GitHub Marketplace · GitHub
  • I decided to implement the two builds and two deploys into separate jobs, but this gives a bit of overhead in setting up the environment for each. I also hit issues installing anaconda-client using pip and hence I’m using micromamba there instead. So I’ve ended up with an eclectic mix of “setup” stuff across the jobs (installing Python, installing Poetry, installing Micromamba…).

If anyone has any thoughts on how this could be better implemented, I would be grateful to know!

Also, is there any way of testing workflows before pushing them to GitHub? It is a bit annoying having to pollute my commit history with a load of broken commits whilst I figure things out! I tried act to run the workflow locally but got a load of Docker permission errors and gave up… Maybe I should have used a temporary branch.

2

The one I worked out for our Python TopoToolbox package looks quite similar to yours:

so I too would be interested if there is a better way to do it. Though I often prefer more verbose but explicit steps over “use some action published by a random user.”

I usually work on a feature branch and then set

on: 
  push: 
    branches: ["feature-branch-name"]

at the top, push a bunch of stuff to try it out, and then (remembering to switch feature-branch-name for main at the last minute!) rebase the feature branch before merging it into main to clean up the commit history. This works but is probably not optimal. I spent a good chunk of the last week waiting for GitHub Actions to run while I tweaked things in one of our workflows.

1 Like
3

Thanks, that’s good advice on branching and rebasing to clean up the commit history. I might give that ago!

As an aside, I seem to be slightly vindicated in using rattler-build instead of conda-build as it looks like Conda Forge are moving towards using the newer Conda recipe format that rattler-build uses: Announcing the new recipe format on conda-forge | conda-forge | community-driven packaging for conda

4

I just used this method for another repo and it worked a charm :slight_smile: