Continuous integration testing of Arch User Repository packages

I maintain a couple of ArchLinux user-contributed packages on the Arch User Repository (AUR), and over time I’ve built out a bit of infrastructure around that to make that maintenance easier (and hopefully the results better). The core of it is automated building of packages in Continuous Integration, which catches a number of issues which otherwise would be more difficult.

This write-up will go through the entire packaging process to make it easily reproducible.

Contributing a package

AUR is a great resource for Arch Linux users, and  it is pretty easy to create and contribute new packages.

Packages are created by cloning an empty git repository with the desired package name. I do it in a slightly different setup compared to the wiki that’s linked just above, as:

git clone ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Add your PKGBUILD and any other required files, run mksrcinfo, and git commit, and push… If everything went well, your package is now visible in the AUR search.

Next time that repository is cloned, it will contain the code, and changes (i.e. package updates) can be pushed just as well too.

Keeping track of packages

As more packages are contributed, it is increasingly hard to keep track of them as separate repositories. One way to improve on this, is creating a “meta” repository (or repo), where all the contributed packages are linked as git submodules.

This organization is achieved by creating your meta-repo, and add your package as a submodule:

git submodule add ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Then you’d make package updates in that submodule, and the meta repo would contain all your packages as a collection.

My packages’ meta repo that show this arrangement is on Github at imrehg/aur.

Continuous integration testing

What we can do with this setup now, is to automatically check out, build, analyze, and test (including installation) of the all the packages.  I’ve set that up as CircleCI build jobs for each of the packages: each of them built and installed in a clean Arch Linux environment.

The clean Arch Linux environment is provided by a Docker image, that I’ve created for this purpose, archlinux-makepkg-docker. That image builds on an upstream Arch Linux image, and sets a few things up:

• updates the image with the latest base build system
• creates a “builder” user that can run sudo
• installs two packages from scratch that are sometimes needed for working with AUR packages: “package-query” and “yaourt”
• installs “namcap” to analyze the package

Each AUR package is set up with its own CircleCI build job as part of a workflow.

Since most of the work for each package is pretty much the same, we can simplify things with templates, such as this:

# Common sections
defaults: &defaults
  working_directory: ~/aur
  docker:
    - image: imrehg/archlinux-makepkg

updatepackage: &updatepackage
  name: Update packages
  command: sudo pacman -Syu --noconfirm

gitupdate: &gitupdate
  name: Git repo updates
  command: |
    sed -i "s#ssh+git://aur@aur.archlinux.org#https://aur.archlinux.org#" .gitmodules
    git submodule update --init
pkgbuildtest: &pkgbuildtest
  name: Testing PKGBUILD
  command: |
    cd ~/aur/${CIRCLE_JOB}
    namcap PKGBUILD
buildtest: &buildtest
  name: Building package
  command: |
    cd ~/aur/${CIRCLE_JOB}
    makepkg -sci --noconfirm

# Main
version: 2
jobs:
  my-package:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest

workflows:
  version: 2
  build:
    jobs:
      - my-package

The sample CircleCI “config.yml” here is set up to build an AUR package called “my-package”:

• it pulls the Arch Linux Docker image mentioned earlier
• updates any outdated OS package
• checks out meta repo that we are working from
• updates the submodule configuration to be able to pull the required submodule without authentication. the “ssh+git://” setup requires the maintainer’s SSH credentials, while switching to “https://” the CI environment is allowed to check the package’s code out (and won’t be able to push back upstream, which is safer)
• runs “namcap” on the PKGBUILD to catch any obvious issues
• builds and installs the package (including dependencies)

As “my-package” is set up above, it does not have any line specific in to that package in the build steps. The specifics are set up using CircleCI variables (CIRCLE_JOB) and YAML Merge Key Language-Independent Types (the “foo: &foo” and “<< : *foo” section). Thus if there’s “another-package”, it’s easy to clone the “my-package” section as it is, naming that “another-package”, and adding a new build job to the end of the file called “another-package”. With this “templating” when the build steps need to be modified, they can be updated in the header, and all the packages will pick that up.

Workflows are also useful, as  jobs can be made dependent on each other, if they are related, such as my “gnushogi” and “xshogi” packages, or likely any AUR package that requires other AUR packages that need to be built.

...
workflows:
  version: 2
  build:
    jobs:
      <other jobs>
      - gnushogi
      - xshogi:
          requires:
            - gnushogi

This would result in a dependency in the jobs as:

The workflows also allow for jobs to give files to each other. E.g. as above “xshogi” depends on “gnushogi” to be installed, I could build all the required dependencies again in “xshogi”, but it was already built, I could just pass on the created package from the earlier job to the next, using CircleCI workspaces.

  gnushogi:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest
      - persist_to_workspace:
          root: gnushogi
          paths: gnushogi-*.pkg.tar.xz

  xshogi:
    <<: *defaults
    steps:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - attach_workspace:
          at: /tmp/workspace
      - run:
          name: Installing gnushogi
          command: sudo pacman -U --noconfirm /tmp/workspace/gnushogi*.pkg.*
      - run:
          <<: *buildtest

The meta repo is now ready to go with such “.circleci/config.yml”, and on each push, it will build all the packages defined in the job list. You can check how the results look for my AUR packages in CircleCI’s build job view (one entry by build job, ie. package-per-push) or workflow view (one entry per push, aggregating all jobs).

One of the advantages of this setup, is that if a build fails on any of the package (e.g. a source file is no longer available) it’s easy to see, and I can catch a number of out-of-date packages sooner than someone reports it on AUR.

Keeping the build image up to data

The  Arch Linux Docker image is automatically built on Docker Hub (and can be found at imrehg/archlinux-makepkg. It is kept fresh by an If This Than That applet, which triggers the build every morning.

That applet just uses the Date & Time and Webhooks recipes. The webhook points to the Trigger URL provided by the “Build Settings / Build Triggers” section on Docker Hub for the image, and it’s a POST request with payload of:

{"docker_tag": "latest"}

Keeping the image fresh like this shortens the build time when running the jobs on CircleCI (fewer packages need to be updated), which especially important as free users have limited CPU time available each month.

Not many packages which use other AUR packages, which likely need more setup here.

Update workflow

As an aside, the process to update any given package with this setup as follows:

• Update the “PKGBUILD” for the package, quite often it’s just the version number
• Update the checksums easily with “updpkgsums” (part of “pacman” so it should be always available)
• Build the package
• If everything goes well, update the required “.SRCINFO” with “mksrcinfo” (part of “pkgbuild-introspection”)
• git add, commit (signed if you can:), and push to AUR
• Clean up the package directory (“git clean -d -f && rm -rf src”)
• Going back up in the folder hierarchy to the meta repo git add and commit the changes to the submodules
• Push to github, and enjoy the build!

Future

Many things can be improved on this setup (one day), here are some ideas

It should be possible to publish the build artifacts to somewhere (say S3) and set it up as a custom Arch Linux package repository, thus can be reused without everyone needing to build from scratch every time.

If that publishing would happen, I’m guessing it would be good to also sign the built packages, which might be a bit trickier to set up safely, but would make package distribution nicer and more robust.

In my list of packages there are not that many that depend on other AUR packages. Other packages with more AUR dependencies might need even more custom setup than shown above, besides the templated sections, to make them speedy and logical.

In the package testing steps, probably should run “namcap” on the finished package too, to catch other issues (e.g. dependencies required but not included).

What’s your experience with maintaining AUR packages, or with CircleCI? Have any feedback on how to make this above even more useful?