This is a debugging tale for those working with containers, and a reminder that things aren’t always what they seem in Linux, all with a big pinch reminder to Read the Fine Manual, Always!

ARM with Achiveteam v2

Recently I’ve got an email from a reader of the ARM images to help the Archive Team blogpost from years ago, asking me about refreshing that project to use again. There I was recompiling the ArchiveTeam’s Docker images to support ARM, and thus I was looking how things changed in the intervening time. I also got more lazy pragmatic since then, I was was wondering if the Archiveteam just made some ARM or multi-arch images as I believe(d) they should. That lead me to their FAQ entry about ARM images:

Can I run the Warrior on ARM or some other unusual architecture?

Not directly. We currently do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident that they run correctly under (among other things) different endiannesses. […]

Set up QEMU with your Docker install and add –platform linux/amd64 to your docker run command.

This actually seems like a sensible thing – if they dug that deep that they’ve seen issues in wget, I’ve definitely been doing things naively before.

The guidance of installing QEMU seems sensible as well (we were doing a lot of those at balena), and it goes roughly like.

1. install binfmt
2. install QEMU with statically compiled binaries
3. load those binaries to emulate the platforms you want with the F / fix_binary flag

For those unfamiliar, binfmt_misc is a Linux kernel feature that allows non-native binary formats to be recognized and passed to user space applications. It’s what makes it possible to run ARM binaries on x86 systems and vice versa through emulation. The various flags are how the actual behaviour of binfmt is adjusted (F, P, C, O)

Docker advised to use a image to set things up, that is, for example for the x86_64/amd64 platform like this:

docker run --privileged --rm tonistiigi/binfmt --install amd4

My Raspberry Pi is running ArchLinuxARM which installs systemd-binfmt to load the relevant emulation settings at boot time, which seemed handy: with the docker method I had to run that every time before I could run an emulated container, with systemd I would have thing ready by every time the time Docker is ready to run (ie. keeping the Archiveteam containers always on and restarting after reboot.) So I have a strong incentive to use the systemd-based approach instead of the docker run based one.

Now comes the kicker :

• the docker installed binfmt setup worked and allowed to run linux/amd64 containers
• systemd-binfmt initiated binfmt setup worked for the x86_64 binaries in the file system, but not in Docker where the binaries just failed to run
• both setups had identical output when looking at the config in /proc/sys/fs/binfmt_misc

When Same’s Not the Same

To see whether emulation works, the tonistiigi/binfmt container can be invoked without any arguments and it shows the status. For example setting things up with docker would show:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/amd64/v2",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-x86_64"
  ]
}

Here the supported section shows amd64 as it should, and their test of running an amd64 image to check if the binaries are run has the expected output:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
x86_64

Going back to the alternative, after uninstalling that emulatior I start up systemd-binfmtI can test the status again:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
[...snip...]
    "qemu-x86_64",
[...snip...]
  ]
}

This shows that while the emulator is installed, Docker doesn’t find that the linux/amd64 platform is supported, and this checks out with running the alpine image again as above:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
exec /bin/uname: exec format error

Well, this doesn’t work.

The binfmt_misc docs in the Linux Kernel wiki do have plenty of info on the setup and use of the that emulation function. For example to check the configuration of the emulation setup, we can look at the contents of a file in /proc filesystem:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /usr/bin/qemu-x86_64
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

This was the almost the same whether I the docker based setup or used systemd-binfmt with a slight difference: the flags bit is only PF when run with systemd-binfmt, and POCF when set things up with docker run. Even if the Docker docs are asking for the F flag, I wanted to make sure we are on equal footing, so I’ve tried to modify the QEMU setup to be the same. This means overriding the qemu-x86_64.conf that is shipped by default:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :FP to :FPOC
• restart systemd-binfmt

After this the output of the the runtime info in /proc/sys/fs/binfmt_misc/qemu-x86_64 was completely the same. Why’s the difference?

More debugging steps ensued:

More Debugging Ensued

I’ve read through the source code of tonistiigi/binfmt on GitHub and seen that it doesn’t do anything fancy, it’s quite clear implementation of the `binfmt_misc` usage docs and the same magic values as QEMU shipped on my system. Good that no surprise, but no hints of any difference

I’ve tried to replicate that process of setting up QEMU through translating it into Python and running, still the same

I’ve recompiled the binary on my system, and run it outside of docker: it worked the same way as the systemd-binfmt setup: x86_64 static binaries¹ worked outside of Docker but not inside of it

A sort-of breakthrough came when I’ve tried out dbhi/qus Docker images, that promises “qemu-user-static (qus) and containers, non-invasive minimal working setups”, and can do the similar emulator & platform support setup with:

docker run --rm --privileged aptman/qus -s -- -p x86_64

It was a lot slower to run (coming back to this later), but worked like the charm, just like Docker’s own recommendation. However there was a difference in the outcome when I checked the runtime config info:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /qus/bin/qemu-x86_64-static
flags: F
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

It has just the apparently required F flag, but the interpreter points to /qus/bin/qemu-x86_64-static … which is not in the regular file system. Nevertheless alpine happily runs, just as my local static binaries.

How does this actually work, then?

Everything’s Illuminated

With this above, and with a better understanding what the docs say, we have everything in place to understand the all the behaviours above, things we had pointers throughout, but not enough experience to put them together:

So, the F flag was required by the Docker docs, what does that actually do?

F – fix binary

The usual behaviour of binfmt_misc is to spawn the binary lazily when the misc format file is invoked. However, this doesn’t work very well in the face of mount namespaces and changeroots, so the F mode opens the binary as soon as the emulation is installed and uses the opened image to spawn the emulator, meaning it is always available once installed, regardless of how the environment changes.

Because of this, if F is set, the interpreter entry in the runtime settings doesn’t mean the path of the interpreter it will be called, but where it was called at the time – ie. it’s irrelevant for the actual runtime.

The tonistiigi/binfmt image ships its own static-compiled qemu-* binarlies, as well as aptman/qus container gets the right ones at runtime (hence the slowness), and the interpreter path is the binary inside the container when the command is run. The binary is then kept in memory, and the container can go away, the interpreter path’s not refering anything that exists any longer.

Why does systemd-binfmt fail then? Well of course because it’s a dynamically linked binary:

$ file /usr/bin/qemu-x86_64
/usr/bin/qemu-x86_64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=a4b8a93a4361be61dfa34a0eab40083325853839, for GNU/Linux 3.7.0, stripped

… and because it’s dynamically linked, even if the F flag makes it stay in memory, its lib dependencies aren’t, so when in run in Docker (which uses namespaces) it doesn’t have everything to run…

And of course, ArchLinux spells this out:

Note: At present, Arch does not offer a full-system mode and statically linked variant (neither officially nor via AUR), as this is usually not needed.

Yes, “as this is usually not needed”. :)

Updated Setup and Looking Forward

Sort of lobbying ArchLinux to have static QEMU² what options do I have?

• set up a systemd service to run the tonistiigi/binfmt container on startup (which is possible)
• get some static QEMU binaries and override the settings that systemd-binfmt uses
• switch to anothe Linux Distro that supports the Pi, the software I run, but also ships static QEMU builds

All three are suboptimal, potentially fragile, and the third is way too much work. Still the second one was kinda fine:

cd $(mktemp -d)
docker create --name="tmp_$$"  tonistiigi/binfmt
docker export tmp_$$ -o tonistiigi.tar.gz
docker rm tmp_$$
tar -xf tonistiigi.tar.gz --wildcards "*/qemu-x86_64"
# Copy along the binaries folder:
sudo cp usr/bin/qemu-x86_64 /usr/bin/qemu-x86_64-static

Then just like we’ve overridden the upstream qemu-x86_64.conf we do it again:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :/usr/bin/qemu-x86_64:FP to :/usr/bin/qemu-x86_64-static:FPOC (that is updating the binary it points at and the flags for good measure too
• As a bonus, can update the :qemu-x86_64: in the front too, say to :qemu-x86_64-static:, to change the display name of the emulator without affecting any of the functionality, it will just rename the entrin in /proc/sys/fs/binfmt_misc
• restart systemd-binfmt

Then the check again:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64-static
enabled
interpreter /usr/bin/qemu-x86_64-static
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

And the alpine-based checks work once more.

Lessons Learned

The details were all in plain sight, but not enough experience to piece these things together. The Docker-recommended image ships its own QEMU? What does that F flag actually do? Can you run binaries while you don’t have them anymore? Dynamic and static linking and the signs of their misbehaviours to provide hints… However this is coupled with confusion when expectations are broken (say the interpreter doesn’t have to refer to an actual file path that exists now), until I started to question my expectations. Also, just being a heavy user of Docker doesn’t mean I’m knowledgeable of the relevant kernel functionality, and probably I should be more…

This whole process underlined my previous thoughts on Software Engineering when AI seems Everywhere, as I did try to debug things by rubber ducking with Claude: this time the hallucinations were through the roof (a metric tonne of non-existent systemd funcionality, non-existent command line flags), definitely got me on a wild goose chase in a few cases. So even more care’s needed, maybe a version of Hofstadter’s Law:

Imreh’s Law³: LLMs are always more wrong than you expect, even when you take into account Imreh’s Law.

In the end, Don’t Panic, make theories and try to prove them, and talk with anyone who listens, even when they are wrong, and you are more likely to get there⁴.

1. I’ve download static binaries from andrew-d/static-binaries, recommend strings as something that’s quick and simple to use ./strings /bin/sh | head for example, allowing fast iteration. ︎
2. ArchLinux is x86 by default, for them it would be to emulate linux/arm64, linux/arm/v7, linux/arm/v6 images. For ArchLinux ARM it would be a different work the other direction. If only the main Arch would support ARM, it would be a happier world (even if even more complex). ︎
3. Tongue-in-cheek, of course. ︎
4. And with this we just rediscovered the Feynman Algorithm, I guess. ︎

The post The curious case of binfmt for x86 emulation for ARM Docker appeared first on ClickedyClick.

While trying out the instructions, it was quickly clear that the Archive Team provided images out of the box were not working on the Pi. Could they be made to work, though? (Spoiler: it did, and you can check the results on GitHub.)

Container ARM-ification

The issue was that the Archive Team-provided project container, atdr.meo.ws/archiveteam/yahooanswers-grab is only available for amd64 (x64-64) machines. Fortunately the source code is available on GitHub and thus could check out the relevant Dockerfile. It uses a specific base image, atdr.meo.ws/archiveteam/grab-base, whose repo reveals that internally it pulls in atdr.meo.ws/archiveteam/wget-lua containing a modified version of wget from yet another repo (adding Lua scripting support, plus a bunch of other stuff added by Archive Team). But fortunately this is the bottom of the stack. And this bottom of the stack is based on debian:buster-slim which does come built as multiarchitecture images, providing arm64, armv7, armv6 versions, the architectures that are relevant for us if building for the Raspberry Pi (and other ARM boards, since why not).

Thus the challenge is now to rebuild the whole stack of wget-lua → grab-base → yahooanswers-grab images with these extra architectures. This would allow our humble Pi to work, since the main Archive Team payload seems to be Python, only the (forked) wget in there is architecture dependent.

There was a very handy blogpost on the Docker site on how to actually do that using buildx: it’s beautifully straightforward:

• create a “builder” for a set of compatible architectures
• run the build command through buildx with extra flags defining the architectures you need

This simple setup hides a few gotchas, though.

Firstly, when building for multiple architectures, you most likely need the relevant target docker repository set up, as buildx will need to push the images somewhere. They are normally in the “builder” container, and they won’t show up locally in “docker images”.

Secondly, for simplicity all the builds are happening in parallel and for “n” architectures I’ve kinda seen more-than-n-times slowdown (anecdotal, I haven’t measured it:). This slowdown had some unexpected effects, that while my Raspberry Pi 3 running Arch Linux ARM (armv8 or also called arm64) and it could build all 3 relevant architectures listed above, 2 of them together worked fine while 3 of them resulted in various timeouts that prevented the successful finish of the build – after the 2 hours or so it took to build the “wget-lua” image… Not fun.

This needed a change of approach, and started to go down the emulated build path instead, using Docker with QEMU and thus enabling x86 machines to build ARM images. Here I enlisted GitHub actions to have a build architecture remotely, and the Docker Setup QEMU and the Docker Setup Buildx actions. They allowed me to potentially build any of these platforms: linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6

Then the whole thing came down to:

• If the stage uses a base image (in the FROM statement) that has a build by me, replace the archive team image name with my images
• Since some images install Python packages, they might not exists in prebuilt wheel format, thus the Dockerfile need to be patched to install the build requirements for that “pip install” to succeed.

These steps were packaged up in a build script (the link is to the current snapshot, just FYI), and used that to successfully build all images one by one and push to Docker Hub. From here I could set up the archiving task on the Pi finally and start contributing to the scrape again!

Are we done yet, though? I’ve noticed that once the upstream code for the task changes, my running task just stops doing anything until it was restated with the latest version. The Archive Team uses an extra Watchtower container that checks running containers for newer versions for images, and if found, it will update & restart things. Fortunately Watchtower containers are provided in multi-architecture fashion, so I didn’t need to fork/compile that, I only needed to rebuild the images if the upstream projects have changed…

Keeping things up to date

So let’s rebuild the images, automatically for good measure, so we won’t have to keep an eye on when it is due. But how to best do that?

The current idea I settled on uses Docker image tags to communicate which versions are available and which aren’t (yet).

• when building an image, also push a tag that is the upstream source repo’s commit hash
• when preparing to build next time, check the upstream latest commit hash, and check if a corresponding image exists on Docker Hub? Only build if no such image exists.

This is complicated however by the fact, that we also want to rebuild stuff automatically if not just the last image in the stack has a new commit, but the upstream projects as well! I’ve tried to put together some logic in a Makefile that would handle all this rebuilding in a dependency-aware way, but it stated to become very hairy very quick. So instead settled on Invoke to have some resemblance to make-like tasks, but more flexibly due to Python. With a bit of trial and error I’ve ended up with a crude dependency management and external script calls (git rev-parse for remote repo status, docker manifest for image status, and the build script).

The first build was roughly 1:45h, afterwards with caching it takes 30-60s for the GitHub action to run when it has nothing to build. That’s fast enough to set it on a schedule, and try to rebuild every half an hour or so.

And with this, we are pretty much done, here’s the relevant GitHub repo all packaged up. So far it works pretty decently, and already had one automatic rebuild & update of images this week that I didn’t have to know about.

Future improvements

No programming project is ever really finished, and this one has a lot of room for improvements.

Could add images for more Archive Team projects (ideally all of them). This is complicated by that some of the images have multiple forms (e.g. wget-lua build with openssl or gnutls), and my current build system doesn’t take that into account. I would need to replicate support for those build args and expand on the “have this image been built?” checks to take those into account as well for robustness.

But the more ambitious and likely trickier and better bang for buck task would be folding this work into the official images by the Archive Team. This is likely challenging for a few reasons. The first, easier aspect is that they use another Docker registry (not Docker Hub) and I don’t know whether that supports multiarch images (though it should). The trickier part is that they use Drone as CI/CD tool, and their Docker plugin is not based on buildx (naturally, I guess) nor have they any other plugin that we could use here (unlike GitHub actions). So besides code change also some convincing needed to encourage tool changes. Much higher bar.

Finally, it would be interesting to dig in to the reasons of the slow builds. It seems like the wget-lua image is dead slow in buildx (about 1:20h), and not sure why, when regular build finishes at a fraction of that time. So far some steps that seem to take very long are git submodule init/update, and some of the config steps, I believe, but didn’t look too deep so far. Cutting this build time would help a lot with experimentation, though since the relevant image doesn’t need to be rebuilt often it might be low return on investment.

And this is all… I’d love to hear how did it work for you if you try the these ARM images!

The post ARM images to help the Archive Team appeared first on ClickedyClick.

This write-up will go through the entire packaging process to make it easily reproducible.

Contributing a package

AUR is a great resource for Arch Linux users, and  it is pretty easy to create and contribute new packages.

Packages are created by cloning an empty git repository with the desired package name. I do it in a slightly different setup compared to the wiki that’s linked just above, as:

git clone ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Add your PKGBUILD and any other required files, run mksrcinfo, and git commit, and push… If everything went well, your package is now visible in the AUR search.

Next time that repository is cloned, it will contain the code, and changes (i.e. package updates) can be pushed just as well too.

Keeping track of packages

As more packages are contributed, it is increasingly hard to keep track of them as separate repositories. One way to improve on this, is creating a “meta” repository (or repo), where all the contributed packages are linked as git submodules.

This organization is achieved by creating your meta-repo, and add your package as a submodule:

git submodule add ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Then you’d make package updates in that submodule, and the meta repo would contain all your packages as a collection.

My packages’ meta repo that show this arrangement is on Github at imrehg/aur.

Continuous integration testing

What we can do with this setup now, is to automatically check out, build, analyze, and test (including installation) of the all the packages.  I’ve set that up as CircleCI build jobs for each of the packages: each of them built and installed in a clean Arch Linux environment.

The clean Arch Linux environment is provided by a Docker image, that I’ve created for this purpose, archlinux-makepkg-docker. That image builds on an upstream Arch Linux image, and sets a few things up:

• updates the image with the latest base build system
• creates a “builder” user that can run sudo
• installs two packages from scratch that are sometimes needed for working with AUR packages: “package-query” and “yaourt”
• installs “namcap” to analyze the package

Each AUR package is set up with its own CircleCI build job as part of a workflow.

Since most of the work for each package is pretty much the same, we can simplify things with templates, such as this:

# Common sections
defaults: &defaults
  working_directory: ~/aur
  docker:
    - image: imrehg/archlinux-makepkg

updatepackage: &updatepackage
  name: Update packages
  command: sudo pacman -Syu --noconfirm

gitupdate: &gitupdate
  name: Git repo updates
  command: |
    sed -i "s#ssh+git://aur@aur.archlinux.org#https://aur.archlinux.org#" .gitmodules
    git submodule update --init
pkgbuildtest: &pkgbuildtest
  name: Testing PKGBUILD
  command: |
    cd ~/aur/${CIRCLE_JOB}
    namcap PKGBUILD
buildtest: &buildtest
  name: Building package
  command: |
    cd ~/aur/${CIRCLE_JOB}
    makepkg -sci --noconfirm

# Main
version: 2
jobs:
  my-package:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest

workflows:
  version: 2
  build:
    jobs:
      - my-package

The sample CircleCI “config.yml” here is set up to build an AUR package called “my-package”:

• it pulls the Arch Linux Docker image mentioned earlier
• updates any outdated OS package
• checks out meta repo that we are working from
• updates the submodule configuration to be able to pull the required submodule without authentication. the “ssh+git://” setup requires the maintainer’s SSH credentials, while switching to “https://” the CI environment is allowed to check the package’s code out (and won’t be able to push back upstream, which is safer)
• runs “namcap” on the PKGBUILD to catch any obvious issues
• builds and installs the package (including dependencies)

As “my-package” is set up above, it does not have any line specific in to that package in the build steps. The specifics are set up using CircleCI variables (CIRCLE_JOB) and YAML Merge Key Language-Independent Types (the “foo: &foo” and “<< : *foo” section). Thus if there’s “another-package”, it’s easy to clone the “my-package” section as it is, naming that “another-package”, and adding a new build job to the end of the file called “another-package”. With this “templating” when the build steps need to be modified, they can be updated in the header, and all the packages will pick that up.

Workflows are also useful, as  jobs can be made dependent on each other, if they are related, such as my “gnushogi” and “xshogi” packages, or likely any AUR package that requires other AUR packages that need to be built.

...
workflows:
  version: 2
  build:
    jobs:
      <other jobs>
      - gnushogi
      - xshogi:
          requires:
            - gnushogi

This would result in a dependency in the jobs as:

The workflows also allow for jobs to give files to each other. E.g. as above “xshogi” depends on “gnushogi” to be installed, I could build all the required dependencies again in “xshogi”, but it was already built, I could just pass on the created package from the earlier job to the next, using CircleCI workspaces.

  gnushogi:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest
      - persist_to_workspace:
          root: gnushogi
          paths: gnushogi-*.pkg.tar.xz

  xshogi:
    <<: *defaults
    steps:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - attach_workspace:
          at: /tmp/workspace
      - run:
          name: Installing gnushogi
          command: sudo pacman -U --noconfirm /tmp/workspace/gnushogi*.pkg.*
      - run:
          <<: *buildtest

The meta repo is now ready to go with such “.circleci/config.yml”, and on each push, it will build all the packages defined in the job list. You can check how the results look for my AUR packages in CircleCI’s build job view (one entry by build job, ie. package-per-push) or workflow view (one entry per push, aggregating all jobs).

One of the advantages of this setup, is that if a build fails on any of the package (e.g. a source file is no longer available) it’s easy to see, and I can catch a number of out-of-date packages sooner than someone reports it on AUR.

Keeping the build image up to data

The  Arch Linux Docker image is automatically built on Docker Hub (and can be found at imrehg/archlinux-makepkg. It is kept fresh by an If This Than That applet, which triggers the build every morning.

That applet just uses the Date & Time and Webhooks recipes. The webhook points to the Trigger URL provided by the “Build Settings / Build Triggers” section on Docker Hub for the image, and it’s a POST request with payload of:

{"docker_tag": "latest"}

Keeping the image fresh like this shortens the build time when running the jobs on CircleCI (fewer packages need to be updated), which especially important as free users have limited CPU time available each month.

Not many packages which use other AUR packages, which likely need more setup here.

Update workflow

As an aside, the process to update any given package with this setup as follows:

• Update the “PKGBUILD” for the package, quite often it’s just the version number
• Update the checksums easily with “updpkgsums” (part of “pacman” so it should be always available)
• Build the package
• If everything goes well, update the required “.SRCINFO” with “mksrcinfo” (part of “pkgbuild-introspection”)
• git add, commit (signed if you can:), and push to AUR
• Clean up the package directory (“git clean -d -f && rm -rf src”)
• Going back up in the folder hierarchy to the meta repo git add and commit the changes to the submodules
• Push to github, and enjoy the build!

Future

Many things can be improved on this setup (one day), here are some ideas

It should be possible to publish the build artifacts to somewhere (say S3) and set it up as a custom Arch Linux package repository, thus can be reused without everyone needing to build from scratch every time.

If that publishing would happen, I’m guessing it would be good to also sign the built packages, which might be a bit trickier to set up safely, but would make package distribution nicer and more robust.

In my list of packages there are not that many that depend on other AUR packages. Other packages with more AUR dependencies might need even more custom setup than shown above, besides the templated sections, to make them speedy and logical.

In the package testing steps, probably should run “namcap” on the finished package too, to catch other issues (e.g. dependencies required but not included).

What’s your experience with maintaining AUR packages, or with CircleCI? Have any feedback on how to make this above even more useful?

The post Continuous integration testing of Arch User Repository packages appeared first on ClickedyClick.