This is a debugging tale for those working with containers, and a reminder that things aren’t always what they seem in Linux, all with a big pinch reminder to Read the Fine Manual, Always!

ARM with Achiveteam v2

Recently I’ve got an email from a reader of the ARM images to help the Archive Team blogpost from years ago, asking me about refreshing that project to use again. There I was recompiling the ArchiveTeam’s Docker images to support ARM, and thus I was looking how things changed in the intervening time. I also got more lazy pragmatic since then, I was was wondering if the Archiveteam just made some ARM or multi-arch images as I believe(d) they should. That lead me to their FAQ entry about ARM images:

Can I run the Warrior on ARM or some other unusual architecture?

Not directly. We currently do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident that they run correctly under (among other things) different endiannesses. […]

Set up QEMU with your Docker install and add –platform linux/amd64 to your docker run command.

This actually seems like a sensible thing – if they dug that deep that they’ve seen issues in wget, I’ve definitely been doing things naively before.

The guidance of installing QEMU seems sensible as well (we were doing a lot of those at balena), and it goes roughly like.

1. install binfmt
2. install QEMU with statically compiled binaries
3. load those binaries to emulate the platforms you want with the F / fix_binary flag

For those unfamiliar, binfmt_misc is a Linux kernel feature that allows non-native binary formats to be recognized and passed to user space applications. It’s what makes it possible to run ARM binaries on x86 systems and vice versa through emulation. The various flags are how the actual behaviour of binfmt is adjusted (F, P, C, O)

Docker advised to use a image to set things up, that is, for example for the x86_64/amd64 platform like this:

docker run --privileged --rm tonistiigi/binfmt --install amd4

My Raspberry Pi is running ArchLinuxARM which installs systemd-binfmt to load the relevant emulation settings at boot time, which seemed handy: with the docker method I had to run that every time before I could run an emulated container, with systemd I would have thing ready by every time the time Docker is ready to run (ie. keeping the Archiveteam containers always on and restarting after reboot.) So I have a strong incentive to use the systemd-based approach instead of the docker run based one.

Now comes the kicker :

• the docker installed binfmt setup worked and allowed to run linux/amd64 containers
• systemd-binfmt initiated binfmt setup worked for the x86_64 binaries in the file system, but not in Docker where the binaries just failed to run
• both setups had identical output when looking at the config in /proc/sys/fs/binfmt_misc

When Same’s Not the Same

To see whether emulation works, the tonistiigi/binfmt container can be invoked without any arguments and it shows the status. For example setting things up with docker would show:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/amd64/v2",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-x86_64"
  ]
}

Here the supported section shows amd64 as it should, and their test of running an amd64 image to check if the binaries are run has the expected output:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
x86_64

Going back to the alternative, after uninstalling that emulatior I start up systemd-binfmtI can test the status again:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
[...snip...]
    "qemu-x86_64",
[...snip...]
  ]
}

This shows that while the emulator is installed, Docker doesn’t find that the linux/amd64 platform is supported, and this checks out with running the alpine image again as above:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
exec /bin/uname: exec format error

Well, this doesn’t work.

The binfmt_misc docs in the Linux Kernel wiki do have plenty of info on the setup and use of the that emulation function. For example to check the configuration of the emulation setup, we can look at the contents of a file in /proc filesystem:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /usr/bin/qemu-x86_64
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

This was the almost the same whether I the docker based setup or used systemd-binfmt with a slight difference: the flags bit is only PF when run with systemd-binfmt, and POCF when set things up with docker run. Even if the Docker docs are asking for the F flag, I wanted to make sure we are on equal footing, so I’ve tried to modify the QEMU setup to be the same. This means overriding the qemu-x86_64.conf that is shipped by default:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :FP to :FPOC
• restart systemd-binfmt

After this the output of the the runtime info in /proc/sys/fs/binfmt_misc/qemu-x86_64 was completely the same. Why’s the difference?

More debugging steps ensued:

More Debugging Ensued

I’ve read through the source code of tonistiigi/binfmt on GitHub and seen that it doesn’t do anything fancy, it’s quite clear implementation of the `binfmt_misc` usage docs and the same magic values as QEMU shipped on my system. Good that no surprise, but no hints of any difference

I’ve tried to replicate that process of setting up QEMU through translating it into Python and running, still the same

I’ve recompiled the binary on my system, and run it outside of docker: it worked the same way as the systemd-binfmt setup: x86_64 static binaries¹ worked outside of Docker but not inside of it

A sort-of breakthrough came when I’ve tried out dbhi/qus Docker images, that promises “qemu-user-static (qus) and containers, non-invasive minimal working setups”, and can do the similar emulator & platform support setup with:

docker run --rm --privileged aptman/qus -s -- -p x86_64

It was a lot slower to run (coming back to this later), but worked like the charm, just like Docker’s own recommendation. However there was a difference in the outcome when I checked the runtime config info:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /qus/bin/qemu-x86_64-static
flags: F
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

It has just the apparently required F flag, but the interpreter points to /qus/bin/qemu-x86_64-static … which is not in the regular file system. Nevertheless alpine happily runs, just as my local static binaries.

How does this actually work, then?

Everything’s Illuminated

With this above, and with a better understanding what the docs say, we have everything in place to understand the all the behaviours above, things we had pointers throughout, but not enough experience to put them together:

So, the F flag was required by the Docker docs, what does that actually do?

F – fix binary

The usual behaviour of binfmt_misc is to spawn the binary lazily when the misc format file is invoked. However, this doesn’t work very well in the face of mount namespaces and changeroots, so the F mode opens the binary as soon as the emulation is installed and uses the opened image to spawn the emulator, meaning it is always available once installed, regardless of how the environment changes.

Because of this, if F is set, the interpreter entry in the runtime settings doesn’t mean the path of the interpreter it will be called, but where it was called at the time – ie. it’s irrelevant for the actual runtime.

The tonistiigi/binfmt image ships its own static-compiled qemu-* binarlies, as well as aptman/qus container gets the right ones at runtime (hence the slowness), and the interpreter path is the binary inside the container when the command is run. The binary is then kept in memory, and the container can go away, the interpreter path’s not refering anything that exists any longer.

Why does systemd-binfmt fail then? Well of course because it’s a dynamically linked binary:

$ file /usr/bin/qemu-x86_64
/usr/bin/qemu-x86_64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=a4b8a93a4361be61dfa34a0eab40083325853839, for GNU/Linux 3.7.0, stripped

… and because it’s dynamically linked, even if the F flag makes it stay in memory, its lib dependencies aren’t, so when in run in Docker (which uses namespaces) it doesn’t have everything to run…

And of course, ArchLinux spells this out:

Note: At present, Arch does not offer a full-system mode and statically linked variant (neither officially nor via AUR), as this is usually not needed.

Yes, “as this is usually not needed”. :)

Updated Setup and Looking Forward

Sort of lobbying ArchLinux to have static QEMU² what options do I have?

• set up a systemd service to run the tonistiigi/binfmt container on startup (which is possible)
• get some static QEMU binaries and override the settings that systemd-binfmt uses
• switch to anothe Linux Distro that supports the Pi, the software I run, but also ships static QEMU builds

All three are suboptimal, potentially fragile, and the third is way too much work. Still the second one was kinda fine:

cd $(mktemp -d)
docker create --name="tmp_$$"  tonistiigi/binfmt
docker export tmp_$$ -o tonistiigi.tar.gz
docker rm tmp_$$
tar -xf tonistiigi.tar.gz --wildcards "*/qemu-x86_64"
# Copy along the binaries folder:
sudo cp usr/bin/qemu-x86_64 /usr/bin/qemu-x86_64-static

Then just like we’ve overridden the upstream qemu-x86_64.conf we do it again:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :/usr/bin/qemu-x86_64:FP to :/usr/bin/qemu-x86_64-static:FPOC (that is updating the binary it points at and the flags for good measure too
• As a bonus, can update the :qemu-x86_64: in the front too, say to :qemu-x86_64-static:, to change the display name of the emulator without affecting any of the functionality, it will just rename the entrin in /proc/sys/fs/binfmt_misc
• restart systemd-binfmt

Then the check again:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64-static
enabled
interpreter /usr/bin/qemu-x86_64-static
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

And the alpine-based checks work once more.

Lessons Learned

The details were all in plain sight, but not enough experience to piece these things together. The Docker-recommended image ships its own QEMU? What does that F flag actually do? Can you run binaries while you don’t have them anymore? Dynamic and static linking and the signs of their misbehaviours to provide hints… However this is coupled with confusion when expectations are broken (say the interpreter doesn’t have to refer to an actual file path that exists now), until I started to question my expectations. Also, just being a heavy user of Docker doesn’t mean I’m knowledgeable of the relevant kernel functionality, and probably I should be more…

This whole process underlined my previous thoughts on Software Engineering when AI seems Everywhere, as I did try to debug things by rubber ducking with Claude: this time the hallucinations were through the roof (a metric tonne of non-existent systemd funcionality, non-existent command line flags), definitely got me on a wild goose chase in a few cases. So even more care’s needed, maybe a version of Hofstadter’s Law:

Imreh’s Law³: LLMs are always more wrong than you expect, even when you take into account Imreh’s Law.

In the end, Don’t Panic, make theories and try to prove them, and talk with anyone who listens, even when they are wrong, and you are more likely to get there⁴.

1. I’ve download static binaries from andrew-d/static-binaries, recommend strings as something that’s quick and simple to use ./strings /bin/sh | head for example, allowing fast iteration. ︎
2. ArchLinux is x86 by default, for them it would be to emulate linux/arm64, linux/arm/v7, linux/arm/v6 images. For ArchLinux ARM it would be a different work the other direction. If only the main Arch would support ARM, it would be a happier world (even if even more complex). ︎
3. Tongue-in-cheek, of course. ︎
4. And with this we just rediscovered the Feynman Algorithm, I guess. ︎

The post The curious case of binfmt for x86 emulation for ARM Docker appeared first on ClickedyClick.

When I say I have a home server, that might be too generous description of the status quo: I have a pretty banged up Raspberry Pi 3B. It’s running ArchLinux ARM, the 64-bit, AAarch64 version, looking a bit more retro on the hardware front while pushing for more modernity on the software side – a mix that I find fun.

There are a handful of services running on the device — not that many, mostly limited by it’s *gulp* 1GB memory; plenty of things I’d love to run, doesn’t well co-locate in such a tiny compartment. Besides the memory, it’s also limited by storage: the Raspberry Pi runs off an SD card, and those are both fragile, and limited in size. If one wants to run a home file server, say using a handful of other SD cards lying around, to expand the available storage, that will be awkward very soon. To make that task less awkward (or replace one kind of awkward with a more interesting one), I’ve set out to set up a ZFS storage pool, using OpenZFS.

The idea

Why ZFS? In big part, to be able to credibly answer that question.

But with a single, more concrete reason: being able to build a more solid and expandable storage unit. ZFS cancombine different storage units

• in a way that combats data errors, e.g. mirroring: this addresses SD cards fragility
• in a way that data can expand across all of them in a single file system: this addresses the SD cards size limitations

This sounds great in theory and after a bit of trial-and-error, I’ve made the following setup, relying on dynamic kernel modules for support for flexibility, and a hodgepodge of drives at hand for the storage

The file system supports needs is provided by the zfs-dkms package dynamic kernel module (DKMS), which means the kernel module required for being able to manage that file system is recompiled for each new Linux kernel version as it is updated. This is handy in theory, as I can use the main kernel packages provided by the ArchLinux ARM team.

For storage, I’ve started off with two SD cards in mirror mode (going for data integrity first). Later I’ve found — and invested in — some large capacity USB sticks that bumped the storage size quite a bit. With these, the currentl ZFS pool looks like this:

It already saved me — or rather my data — once where an SD card was acting up, though that’s par for the course. One very large benefit is that the main system card is being used less, so hopefully will last longer.

The complications

Of course, it’s never this easy… With non-mainline kernel modules and with DKMS, every update is a bit of a gamble, that can suddenly not pay off. That’s exactly what happened last year, when suddenly the module didn’t compile anymore on a new kernel version, and thus all that storage was sitting dump and inaccessible. After digging into the issue, it down to:

1. the OpenZFS project being under Common Development and Distribution License (CDDL)
2. the Linux kernel deliberately breaking non-GPL licensed code by starting to withold certain floating point capabilities, because “this is not expected to be disruptive to existing users”.

This wasn’t great, as user being between pretty much a rock & a hard place, even if this is a hobby and not strictly speaking a production use case on my side.

Nonetheless, it worked by downgrading to a working version and skipping updates to the kernel packages.

Then based on a suggestion, patching the zfs-dkms package (rewriting the license entry in the META file) to make it look like it’s a GPL-licensed module — which is fair game for one doing on their own machine. This is hacky, or let’s call it pragmatic.

--- META.prev   2024-02-28 08:42:21.526641154 +0800
+++ META        2024-02-28 08:42:36.435569959 +0800
@@ -4,7 +4,7 @@
 Version:       2.2.3
 Release:       1
 Release-Tags:  relext
-License:       CDDL
+License:       GPL
 Author:        OpenZFS
 Linux-Maximum: 6.7
 Linux-Minimum: 3.10

Now, with the current 2.2.3 version, it seems like there’s an official fix-slash-workaround for being able to get the module to compile, even if it’s not a full fix. From the linked merge request message I’m not fully convinced that this is not a fragile status quo, but it’s at least front of mind – good going for wider ARM hardware usage that brings out people’s willingness to fix things!

Future development

Some while back, while working at an IoT software deploument & management company, I had a lot of interesting hardware at hand, naturally, to build things with (or wrestle with…). Nowadays I have things I best describe as spare parts, and thus loads of thingss are more fragile than they need to be, as well as gosh-it-takes-a-long-time to compile things on a Raspberry Pi 3 – making every kernel update some half-an-hour longer!

Likely the best move would be to upgrade to a (much more powerful) Raspberry Pi 5 and use an external NVMe drive, where I’d have much less need for ZFS, at least for the original reasons. It would likely be still useful for other aspects (such as snapshotting, or sending/receiving the drive data, compression, deduplication, etc…), changing the learning path away from multi-device support to the file system features.

If I wanted to use more storage in the existing system, I could also get rid of the mirrored SD cards and just just 4 large USB sticks (maybe in a RAIDZ setup), a poor-man’s NAS, I guess. Though there I’d worry a bit about using the sticks with the same sizes for this to work (unlike pooling, which has no same-size requirements), given the differences in the supposedly same sized products from different companies (likely locking me into a having the same brand and model across the board).

I also feel like I’m not using ZFS to its full potential. If I know enough just to be dangerous… maybe that’s the generalists natural habitat?

The post ZFS on a Raspberry Pi appeared first on ClickedyClick.

This write-up will go through the entire packaging process to make it easily reproducible.

Contributing a package

AUR is a great resource for Arch Linux users, and  it is pretty easy to create and contribute new packages.

Packages are created by cloning an empty git repository with the desired package name. I do it in a slightly different setup compared to the wiki that’s linked just above, as:

git clone ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Add your PKGBUILD and any other required files, run mksrcinfo, and git commit, and push… If everything went well, your package is now visible in the AUR search.

Next time that repository is cloned, it will contain the code, and changes (i.e. package updates) can be pushed just as well too.

Keeping track of packages

As more packages are contributed, it is increasingly hard to keep track of them as separate repositories. One way to improve on this, is creating a “meta” repository (or repo), where all the contributed packages are linked as git submodules.

This organization is achieved by creating your meta-repo, and add your package as a submodule:

git submodule add ssh+git://aur@aur.archlinux.org/<PACKAGENAME>.git

Then you’d make package updates in that submodule, and the meta repo would contain all your packages as a collection.

My packages’ meta repo that show this arrangement is on Github at imrehg/aur.

Continuous integration testing

What we can do with this setup now, is to automatically check out, build, analyze, and test (including installation) of the all the packages.  I’ve set that up as CircleCI build jobs for each of the packages: each of them built and installed in a clean Arch Linux environment.

The clean Arch Linux environment is provided by a Docker image, that I’ve created for this purpose, archlinux-makepkg-docker. That image builds on an upstream Arch Linux image, and sets a few things up:

• updates the image with the latest base build system
• creates a “builder” user that can run sudo
• installs two packages from scratch that are sometimes needed for working with AUR packages: “package-query” and “yaourt”
• installs “namcap” to analyze the package

Each AUR package is set up with its own CircleCI build job as part of a workflow.

Since most of the work for each package is pretty much the same, we can simplify things with templates, such as this:

# Common sections
defaults: &defaults
  working_directory: ~/aur
  docker:
    - image: imrehg/archlinux-makepkg

updatepackage: &updatepackage
  name: Update packages
  command: sudo pacman -Syu --noconfirm

gitupdate: &gitupdate
  name: Git repo updates
  command: |
    sed -i "s#ssh+git://aur@aur.archlinux.org#https://aur.archlinux.org#" .gitmodules
    git submodule update --init
pkgbuildtest: &pkgbuildtest
  name: Testing PKGBUILD
  command: |
    cd ~/aur/${CIRCLE_JOB}
    namcap PKGBUILD
buildtest: &buildtest
  name: Building package
  command: |
    cd ~/aur/${CIRCLE_JOB}
    makepkg -sci --noconfirm

# Main
version: 2
jobs:
  my-package:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest

workflows:
  version: 2
  build:
    jobs:
      - my-package

The sample CircleCI “config.yml” here is set up to build an AUR package called “my-package”:

• it pulls the Arch Linux Docker image mentioned earlier
• updates any outdated OS package
• checks out meta repo that we are working from
• updates the submodule configuration to be able to pull the required submodule without authentication. the “ssh+git://” setup requires the maintainer’s SSH credentials, while switching to “https://” the CI environment is allowed to check the package’s code out (and won’t be able to push back upstream, which is safer)
• runs “namcap” on the PKGBUILD to catch any obvious issues
• builds and installs the package (including dependencies)

As “my-package” is set up above, it does not have any line specific in to that package in the build steps. The specifics are set up using CircleCI variables (CIRCLE_JOB) and YAML Merge Key Language-Independent Types (the “foo: &foo” and “<< : *foo” section). Thus if there’s “another-package”, it’s easy to clone the “my-package” section as it is, naming that “another-package”, and adding a new build job to the end of the file called “another-package”. With this “templating” when the build steps need to be modified, they can be updated in the header, and all the packages will pick that up.

Workflows are also useful, as  jobs can be made dependent on each other, if they are related, such as my “gnushogi” and “xshogi” packages, or likely any AUR package that requires other AUR packages that need to be built.

...
workflows:
  version: 2
  build:
    jobs:
      <other jobs>
      - gnushogi
      - xshogi:
          requires:
            - gnushogi

This would result in a dependency in the jobs as:

The workflows also allow for jobs to give files to each other. E.g. as above “xshogi” depends on “gnushogi” to be installed, I could build all the required dependencies again in “xshogi”, but it was already built, I could just pass on the created package from the earlier job to the next, using CircleCI workspaces.

  gnushogi:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - run:
          <<: *buildtest
      - persist_to_workspace:
          root: gnushogi
          paths: gnushogi-*.pkg.tar.xz

  xshogi:
    <<: *defaults
    steps:
    <<: *defaults
    steps:
      - run:
          <<: *updatepackage
      - checkout
      - run:
          <<: *gitupdate
      - run:
          <<: *pkgbuildtest
      - attach_workspace:
          at: /tmp/workspace
      - run:
          name: Installing gnushogi
          command: sudo pacman -U --noconfirm /tmp/workspace/gnushogi*.pkg.*
      - run:
          <<: *buildtest

The meta repo is now ready to go with such “.circleci/config.yml”, and on each push, it will build all the packages defined in the job list. You can check how the results look for my AUR packages in CircleCI’s build job view (one entry by build job, ie. package-per-push) or workflow view (one entry per push, aggregating all jobs).

One of the advantages of this setup, is that if a build fails on any of the package (e.g. a source file is no longer available) it’s easy to see, and I can catch a number of out-of-date packages sooner than someone reports it on AUR.

Keeping the build image up to data

The  Arch Linux Docker image is automatically built on Docker Hub (and can be found at imrehg/archlinux-makepkg. It is kept fresh by an If This Than That applet, which triggers the build every morning.

That applet just uses the Date & Time and Webhooks recipes. The webhook points to the Trigger URL provided by the “Build Settings / Build Triggers” section on Docker Hub for the image, and it’s a POST request with payload of:

{"docker_tag": "latest"}

Keeping the image fresh like this shortens the build time when running the jobs on CircleCI (fewer packages need to be updated), which especially important as free users have limited CPU time available each month.

Not many packages which use other AUR packages, which likely need more setup here.

Update workflow

As an aside, the process to update any given package with this setup as follows:

• Update the “PKGBUILD” for the package, quite often it’s just the version number
• Update the checksums easily with “updpkgsums” (part of “pacman” so it should be always available)
• Build the package
• If everything goes well, update the required “.SRCINFO” with “mksrcinfo” (part of “pkgbuild-introspection”)
• git add, commit (signed if you can:), and push to AUR
• Clean up the package directory (“git clean -d -f && rm -rf src”)
• Going back up in the folder hierarchy to the meta repo git add and commit the changes to the submodules
• Push to github, and enjoy the build!

Future

Many things can be improved on this setup (one day), here are some ideas

It should be possible to publish the build artifacts to somewhere (say S3) and set it up as a custom Arch Linux package repository, thus can be reused without everyone needing to build from scratch every time.

If that publishing would happen, I’m guessing it would be good to also sign the built packages, which might be a bit trickier to set up safely, but would make package distribution nicer and more robust.

In my list of packages there are not that many that depend on other AUR packages. Other packages with more AUR dependencies might need even more custom setup than shown above, besides the templated sections, to make them speedy and logical.

In the package testing steps, probably should run “namcap” on the finished package too, to catch other issues (e.g. dependencies required but not included).

What’s your experience with maintaining AUR packages, or with CircleCI? Have any feedback on how to make this above even more useful?

The post Continuous integration testing of Arch User Repository packages appeared first on ClickedyClick.

Hardware

To have a really good really good, satellite-powered reference clock, I have to have access to a 1-pulse-per-second (1PPS) signal from the receiver. The pure USB-connected receivers don’t really seem to do that yet (looks like plenty of opportunities there!), instead I have to use separate hardware for it.

The Navspark module has a 1PPS pin (GPO3 below), and the only other pin I’ll really need is a serial pin to receive the NMEA stream of the satellite lock data (TXD1 below).

These pins have to be connected to a Raspberry Pi 2’s GPIO pins.

Based on the pinout above, and some digging around, the connections I needed to make between the Pi (left) and Navspark (right) were:

• 5V  BATTERY
• Ground GND
• GPIO15 (UART0_RX) TXD1
• GPIO8 P1PPS (GPO3)

Some things to remember here, that on serial connections one side’s RX line is connected to the other’s TX  and vice versa. Also, as much as I gather any other GPIO pins on the Pi could be used to receive the 1PPS signal, I used GPIO8 because it’s been used by other projects and playing it safe…

The adapter board was easy enough to put together (though single-side pads are a pain when the components are on both sides, as it is the case here).

The adapter works well enough, though I’m sure that it could be done nicer.

Software

I found a lot of examples and help from other people’s similar projects but there were still some differences to be figured along the way. The overall setup was to be as follows:

• ArchLinux ARM for base platform
• gpsd for interpreting the Navspark NMEA messages
• LinuxPPS for 1PPS signal passing from GPIO to the userspace
• chrony as time server (as opposed to the usual ntpd)

Installing the system was straightforward enough (well, because I’ve done it so many times, but I digress), and added a few required basic packages: gpsd, chrony, git, the base-devel group, avahi (for easier network discovery).

GPSD

UART0 serial communication is done through the /dev/ttyAMA0 device in Pi, though the default settings needed some change for gpsd to be able to receive the data from Navspark.

Most of the changes our outline in a blogpost where I have learned from as well. First remove all config settings from /boot/cmdline.txt, basically these parts:

console=ttyAMA0,115200 kgdboc=ttyAMA0,115200

Next disable the automatic serial console on that device with removing relevant systemd unit with

sudo systemctl disable serial-getty@ttyAMA0.service

Then have to set up the serial channel’s baud rate to enable communication, and here I’ve done that with udev: create a new rule file at /etc/udev/rules.d/AMA0_serial.rules with the content of

KERNEL=="ttyAMA0", RUN+="/bin/stty -F /dev/ttyAMA0 115200 raw -echo"

which will set the 115200 baud rate used by Navspark when the ttyAMA0 device comes up.

Finally have to add gpsd’s default settings in the /etc/gpsd file:

# Default settings for gpsd.
START_DAEMON="true"
GPSD_OPTIONS="-b -n"
DEVICES="/dev/ttyAMA0"
USBAUTO="true"

Now it’s ready, and should be able to run gpsd and see some communication.

sudo systemctl start gpsd

and if want to start it on boot, then run the same command with “enable” instead “start”. Use gpsmon to see what NMEA data Navspark sends, and hopefully it will lock on soon too (mind the sky visibility!). Can also use cgps to monitor which satellites the system is locked onto.

1PPS

For 1PPS I had to add an item to /boot/config.txt:

dttoverlay=pps-gpio,gpiopin=8

This adds the relevant device tree overlay to enable the pps-gpio driver on GPIO8. Upon boot there was a /dev/pps0 device that can be poked at e.g. with ppstest from pps-tools. Since pps-tools did not have a package on ArchLinux, I made one, and the PKGBUILD is available on Github. That package also adds the timepps.h header file which is required for the chrony PPS driver to work.

The pps signal can be checked by running ppstest:

$ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1448436177.999366423, sequence: 199 - clear  0.000000000, sequence: 0
source 0 - assert 1448436178.999373610, sequence: 200 - clear  0.000000000, sequence: 0
...

but it needs a 3D fix for the signal to show up.

Chrony

Next up is the time server itself. The satellite time will be pulled in using two “reference clock”: the GPS timestamps provided by gpsd through a memory interface, and the PPS data that locks those timestamps. Unfortunately the PPS driver is not built in by default because it needs the timepps.h header. This means after installing pps-tools chrony needs to be recompiled.

Update the chrony configuration by editing the /etc/chrony.conf file. I usually leave the defaults alone, and add my modifications to the end of the file for clarity. Here’s my current setup:

lock_all

logdir /var/log/chrony
log measurements statistics tracking refclocks

# being a server
allow 192.168/16

server 2.tw.pool.ntp.org iburst
server 3.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server 1.europe.pool.ntp.org iburst

# on-boot time setup, from https://news.ycombinator.com/item?id=9796885
makestep 0.1 1

# from https://www.raspberrypi.org/forums/viewtopic.php?f=41&t=1970&start=275
refclock PPS /dev/pps0 lock GPS prefer refid PPS0
# If Beidou, there is a day offset somehow
refclock SHM 0 offset -86400 delay 0.1 refid GPS noselect

leapsectz right/Asia/Taipei

It uses NTP pool servers for initial time setup and a quick on-boot step (as we don’t want to be stuck at 1970, do we?). The time service related settings are the “refclock” parts. The first line sets the 1PPS lock-on and I tell chrony to prefer that over other sources so we’ll switch over to that when there’s good-enough data. The second refclock section uses gpsd’s NMEA data. The large offset there is due to some issues where the reference time shows up with 24h offset. That I still need to investigate. The delay time set needs some fine-tuning too (I think), though from comments 0.1-0.3 (that is seconds) seems to be suitable? I also tell chrony not to use that for reference.

I also set leap seconds timezones (with the “right/…”  TAI time zone, see e.g. the NTP support page and a discussion on the Debian mailing list). And I enable the server on the local network so other can connect to it and get the time.

I’ve enabled logging of the tracking data (so there’s offset and other information available) and refclocks (tailing /var/log/chrony/refclocks.log continuously is a handy way to see when the lock is lost).

Start chrony (or enable if you want to start it on boot)

sudo systemctl start chrony

and the results can be monitored with chronyc.

All together now

If everything worked well, gpsmon should show a good locked signal after not too much time, something like this:

From the interface above I can also see that Navspark has a 3D lock (Mode A3) to satellites number 209, 207, and 206, which are Beidou birds. And that the time offset is 86399.96 seconds (~24h), that is what the chrony configuration compensates for.

Checking the results in chrony I the system time it’s locked to PPS0 where the last sample had 410ns offset (pretty good).  Also printed the sourcestats.

And this is pretty much all that is needed in general to have a Stratum-1 (atomic clock-locked) NTP server in general.

Some extra references:

• NavSpark GPS+GLONASS Stratum-1 NTP Time Source with a Raspberry Pi
• Hackspherelab: NTP GPS Time Server
• GPSD Time Service HOWTO

Experience

Navspark has a GNSS Viewer program (available for download from their website) which can also set the module’s parameters, for example forcing it to only listen to the GPS signal or only Beidou signal. There are a lot of settings, and I think most don’t need any change, but there’s still a bit to explore, and it’s definitely a handy tool.

Running the NTP lock and looking at the tracking lock there are immediately a few lessons in the ~1 day I’ve been playing with the system:

Environment matters a lot: having wind blowing over the system and being exposed to temperature changes (e.g. outdoor setting) is quite bad for frequency/time stability. One could say duh! but it’s good to see. Even a curtain and placing it indoors helps too, though having it in a well air conditioned space is even better. Too bad that in that case the sky visibility and thus the signal strength suffers a lot, but as long as there’s enough signal to have a lock it is worth it.

PPS is lost when 3D fix is lost: this took a while to figure out why did I lose lock every now and then, especially because once chrony loses the PPS signal it won’t pick it up again.

Indoor can work: indoor lock with the default antenna can work, but the building structure matters a lot.

GPS vs Beidou

Of course the most interesting part is comparing GPS and Beidou. What follows here is not very well qualified yet, just first impressions and need more work.

One of the biggest advantage of GPS is having a lot more satellites than Beidou. I wouldn’t be surprised if they are transmitting at a higher power too, or some other ways easier to receive. When locked onto GPS I could have rock-solid indoors time lock with 4-9 satellites visible at any one time. The time precision is judged by the offset frequency distribution that chrony reports (and would love to hear if there’s a better way to do that!), and that had a standard deviation of about 1µs by experience (in an air-conditioned office).

When I’ve enabled both GPS and Beidou then Navspark seemed to have generally locked onto the Beidou signal. When i was lucky I got 5-6 satellites, but generally it’s around 3 – just on the threshold of having a 3D fix, and the source of the PPS-unlock mentioned above. The offset stability on the other hand can be really good, I had runs that had about 5x smaller time offset standard deviation than GPS had (~0.2µs), and that’s quite a bit of improvement!

The issue is, that when I set the Navspark module GPS+Beidou then when there was a low Beidou satellite count (2-3), even though it still kept a (fragile) 3D fix, the time offset stability was pretty much the same as GPS on its own, though with some intervals of improved readings….

This below is a ~35minutes run of GPS+Beidou setup in an air-conditioned office with limited sky-view and 3 more stories above, using the default 12dB antenna. Using cpgs I could see that in general the lock was provided by 2-3 Beidou satellites, and in some intervals switched over to GPS without losing PPS signal (until it did). Thus it’s not the best performance that I’ve seen but still pretty good. Offset is plotted with the reported standard deviation as error bars. The central line is the mean value of the offset readings, and the two bars are the standard deviation calculated from the offsets.

Future

The ideal setup would be a well air-conditioned roof-top location (I think it would be possible to reliably see 6-7 satellites that way), and switching to Beidou only measurements to clearly see the difference between the two satellite network.

Improved reception could probably be achieved by a different Navspark antenna, which provides 25±3dB gain (that’s an extra ~13dB above the current antenna), and should help indoor reception a lot. It’s currently on pre-sale and expected by the end of the year, so will keep an eye on it!

For more advanced improvements, the adapter board would be easy enough to redo on PCB (instead of surface mount soldering), add maybe some more voltage stabilization for the module to improve the noise figure, and probably have cleaner 1PPS signal too (compared to flying wires). Will definitely give it a try as it is so easy to get the boards printed these days, e.g. with Seeed Studio.

Finally some more setting adjustments could improve the final result, such as tweaking the 1PPS pulse length, using the surveying profile for higher precision (though have to figure out what does it mean first, but looks promision), and other little tweaks….

It looks to me that if the satellite reception issues are cleaned up, the Navspark + Beidou 1PPS signal should be interesting to try with laboratory frequency references for long term stability (such as a Rubidium Clock). And of course to run the Pi+Navspark as a local time reference and provide a NTP service, e.g. from the Tapei Hackerspace…. Plenty to do. :)

The post Dual Satellite NTP server with Navspark appeared first on ClickedyClick.