Seemingly identical configurations, different results. When two methods for setting up x86 emulation on ARM showed the exact same system configuration but behaved completely differently in Docker, I began questioning my system administration knowledge and my sanity – and briefly contemplated a new career as a blacksmith.<\/p>\n\n\n\n
This is a debugging tale for those working with containers, and a reminder that things aren’t always what they seem in Linux, all with a big pinch reminder to Read the Fine Manual, Always!<\/p>\n\n\n\n\n\n\n\n
Recently I’ve got an email from a reader of the ARM images to help the Archive Team<\/a> blogpost from years ago, asking me about refreshing that project to use again. There I was recompiling the ArchiveTeam’s Docker images to support ARM, and thus I was looking how things changed in the intervening time. I also got more Can I run the Warrior on ARM or some other unusual architecture?<\/strong><\/p>\n\n\n\n Not directly. We currently do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident that they run correctly under (among other things) different endiannesses. […]<\/p>\n\n\n\n Set up QEMU with your Docker install<\/a> and add –platform linux\/amd64 to your docker run command.<\/p>\n<\/blockquote>\n\n\n\n This actually seems like a sensible thing – if they dug that deep that they’ve seen issues in wget<\/a>, I’ve definitely been doing things naively before.<\/p>\n\n\n\n The guidance of installing QEMU seems sensible as well (we were doing a lot of those at balena<\/a>), and it goes roughly like.<\/p>\n\n\n\n For those unfamiliar, Docker advised to use a image to set things up, that is, for example for the My Raspberry Pi is running ArchLinuxARM<\/a> which installs systemd-binfmt<\/a> to load the relevant emulation settings at boot time, which seemed handy: with the docker method I had to run that every time before I could run an emulated container, with systemd I would have thing ready by every time the time Docker is ready to run (ie. keeping the Archiveteam containers always on and restarting after reboot.) So I have a strong incentive to use the systemd-based approach instead of the Now comes the kicker \ud83e\udd2f:<\/p>\n\n\n\n To see whether emulation works, the Here the Going back to the alternative, after uninstalling that emulatior I start up This shows that while the emulator is installed, Docker doesn’t find that the Well, this doesn’t work.<\/p>\n\n\n\n The binfmt_misc docs in the Linux Kernel wiki<\/a> do have plenty of info on the setup and use of the that emulation function. For example to check the configuration of the emulation setup, we can look at the contents of a file in This was the almost the same whether I the After this the output of the the runtime info in More debugging steps ensued:<\/p>\n\n\n\n I’ve read through the source code<\/strong> of I’ve tried to replicate that process<\/strong> of setting up QEMU through translating it into Python and running, still the same<\/p>\n\n\n\n I’ve recompiled the binary<\/strong> on my system, and run it outside of docker: it worked the same way as the A sort-of breakthrough<\/strong> came when I’ve tried out dbhi\/qus<\/a> Docker images, that promises “qemu-user-static (qus) and containers, non-invasive minimal working setups”, and can do the similar emulator & platform support setup with:<\/p>\n\n\n\n It was a lot slower to run (coming back to this later), but worked like the charm, just like Docker’s own recommendation. However there was a difference in the outcome when I checked the runtime config info:<\/p>\n\n\n\n It has just the apparently required How does this actually work, then?<\/p>\n\n\n\n With this above, and with a better understanding what the docs say, we have everything in place to understand the all the behaviours above, things we had pointers throughout, but not enough experience to put them together:<\/p>\n\n\n\n So, the The usual behaviour of binfmt_misc is to spawn the binary lazily when the misc format file is invoked. However, this doesn\u2019t work very well in the face of mount namespaces and changeroots, so the Because of this, if The Why does … and because it’s dynamically linked, even if the And of course, ArchLinux spells<\/a> this out:<\/p>\n\n\n\n Note:<\/strong> At present, Arch does not offer a full-system mode and statically linked variant (neither officially nor via AUR), as this is usually not needed.<\/p>\n<\/blockquote>\n\n\n\n Yes, “as this is usually not needed”. :)<\/p>\n\n\n\n Sort of lobbying ArchLinux to have static QEMU2<\/a><\/sup> what options do I have?<\/p>\n\n\n\n All three are suboptimal, potentially fragile, and the third is way too much work. Still the second one was kinda fine:<\/p>\n\n\n\n Then just like we’ve overridden the upstream Then the check again:<\/p>\n\n\n\n And the The details were all in plain sight, but not enough experience to piece these things together. The Docker-recommended image ships its own QEMU? What does that This whole process underlined my previous thoughts on Software Engineering when AI seems Everywhere<\/a>, as I did try to debug things by rubber ducking with Claude<\/a>: this time the hallucinations were through the roof (a metric tonne of non-existent systemd funcionality, non-existent command line flags), definitely got me on a wild goose chase in a few cases. So even more care’s needed, maybe a version of Hofstadter’s Law<\/a>:<\/p>\n\n\n\n Imreh’s Law3<\/a><\/sup>: LLMs are always more wrong than you expect, even when you take into account Imreh’s Law.<\/p>\n<\/blockquote>\n\n\n\n In the end, Don’t Panic, make theories and try to prove them, and talk with anyone who listens, even when they are wrong, and you are more likely to get there4<\/a><\/sup>.<\/p>\n\n\n Seemingly identical configurations, different results. When two methods for setting up x86 emulation on ARM showed the exact same system configuration but behaved completely differently in Docker, I began questioning my system administration knowledge and my sanity – and briefly contemplated a new career as a blacksmith. This is a debugging tale for those working […]<\/p>\n","protected":false},"author":1,"featured_media":2969,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"[{\"content\":\"I've download static binaries from andrew-d\/static-binaries<\/a>, recommend lazy<\/s> pragmatic since then, I was was wondering if the Archiveteam<\/a> just made some ARM or multi-arch images as I believe(d) they should. That lead me to their FAQ entry about ARM images<\/a>:<\/p>\n\n\n\n\n
\n
binfmt<\/code><\/li>\n\n\n\nF<\/code> \/ fix_binary<\/code> flag<\/li>\n<\/ol>\n\n\n\nbinfmt_misc<\/code> is a Linux kernel feature<\/a> that allows non-native binary formats to be recognized and passed to user space applications. It’s what makes it possible to run ARM binaries on x86 systems and vice versa through emulation. The various flags are how the actual behaviour of binfmt<\/code> is adjusted (F, P, C, O)<\/p>\n\n\n\nx86_64<\/code>\/amd64<\/code> platform like this:<\/p>\n\n\n\ndocker run --privileged --rm tonistiigi\/binfmt --install amd4<\/code><\/pre>\n\n\n\ndocker run<\/code> based one.<\/p>\n\n\n\n\n
docker<\/code> installed binfmt<\/code> setup worked and allowed to run linux\/amd64<\/code> containers<\/li>\n\n\n\nsystemd-binfmt<\/code> initiated binfmt<\/code> setup worked for the x86_64<\/code> binaries in the file system, but not in Docker where the binaries just failed to run<\/li>\n\n\n\n\/proc\/sys\/fs\/binfmt_misc<\/code><\/li>\n<\/ul>\n\n\n\nWhen Same’s Not the Same<\/h2>\n\n\n\n
tonistiigi\/binfmt<\/code> container can be invoked without any arguments and it shows the status. For example setting things up with docker<\/code> would show:<\/p>\n\n\n\n$ docker run --privileged --rm tonistiigi\/binfmt\n{\n \"supported\": [\n \"linux\/arm64\",\n \"linux\/amd64\",\n \"linux\/amd64\/v2\",\n \"linux\/arm\/v7\",\n \"linux\/arm\/v6\"\n ],\n \"emulators\": [\n \"qemu-x86_64\"\n ]\n}<\/code><\/pre>\n\n\n\nsupported<\/code> section shows amd64<\/code> as it should, and their test of running an amd64 image to check if the binaries are run has the expected output:<\/p>\n\n\n\n$ docker run --rm --platform linux\/amd64 -t alpine uname -m\nx86_64<\/code><\/pre>\n\n\n\nsystemd-binfmt<\/code>I can test the status again:<\/p>\n\n\n\n$ docker run --privileged --rm tonistiigi\/binfmt\n{\n \"supported\": [\n \"linux\/arm64\",\n \"linux\/arm\/v7\",\n \"linux\/arm\/v6\"\n ],\n \"emulators\": [\n[...snip...]\n \"qemu-x86_64\",\n[...snip...]\n ]\n}<\/code><\/pre>\n\n\n\nlinux\/amd64<\/code> platform is supported, and this checks out with running the alpine<\/code> image again as above:<\/p>\n\n\n\n$ docker run --rm --platform linux\/amd64 -t alpine uname -m\nexec \/bin\/uname: exec format error<\/code><\/pre>\n\n\n\n\/proc<\/code> filesystem:<\/p>\n\n\n\n$ cat \/proc\/sys\/fs\/binfmt_misc\/qemu-x86_64\nenabled\ninterpreter \/usr\/bin\/qemu-x86_64\nflags: POCF\noffset 0\nmagic 7f454c4602010100000000000000000002003e00\nmask fffffffffffefe00fffffffffffffffffeffffff<\/code><\/pre>\n\n\n\ndocker<\/code> based setup or used systemd-binfmt<\/code> with a slight difference: the flags<\/code> bit is only PF<\/code> when run with systemd-binfmt<\/code>, and POCF<\/code> when set things up with docker run<\/code>. Even if the Docker docs are asking for the F<\/code> flag, I wanted to make sure we are on equal footing, so I’ve tried to modify the QEMU setup to be the same. This means overriding the qemu-x86_64.conf<\/code> that is shipped by default:<\/p>\n\n\n\n\n
\/usr\/lib\/binfmt.d\/qemu-x86_64.conf<\/code> to \/etc\/binfmt.d\/qemu-x86_64.conf<\/code> (make sure the file has the same name to ensure this new file overrides the one from the lib folder)<\/li>\n\n\n\n:FP<\/code> to :FPOC<\/code> <\/li>\n\n\n\nsystemd-binfmt<\/code><\/li>\n<\/ul>\n\n\n\n\/proc\/sys\/fs\/binfmt_misc\/qemu-x86_64<\/code> was completely the same. Why’s the difference?<\/p>\n\n\n\nMore Debugging Ensued<\/h3>\n\n\n\n
tonistiigi\/binfmt<\/code> on GitHub<\/a> and seen that it doesn’t do anything fancy, it’s quite clear implementation of the `binfmt_misc` usage docs and the same magic values as QEMU shipped on my system. Good that no surprise, but no hints of any difference<\/p>\n\n\n\nsystemd-binfmt<\/code> setup: x86_64<\/code> static binaries1<\/a><\/sup> worked outside of Docker but not inside of it<\/p>\n\n\n\ndocker run --rm --privileged aptman\/qus -s -- -p x86_64<\/code><\/pre>\n\n\n\n$ cat \/proc\/sys\/fs\/binfmt_misc\/qemu-x86_64\nenabled\ninterpreter \/qus\/bin\/qemu-x86_64-static\nflags: F\noffset 0\nmagic 7f454c4602010100000000000000000002003e00\nmask fffffffffffefe00fffffffffffffffffeffffff<\/code><\/pre>\n\n\n\nF<\/code> flag, but the interpreter points to \/qus\/bin\/qemu-x86_64-static<\/code> … which is not in the regular file system. Nevertheless alpine<\/code> happily runs, just as my local static binaries.<\/p>\n\n\n\nEverything’s Illuminated<\/h2>\n\n\n\n
F<\/code> flag was required by the Docker docs, what does that actually<\/em> do?<\/p>\n\n\n\n\n
F<\/code> – fix binary<\/p>\n\n\n\nF<\/code> mode opens the binary as soon as the emulation is installed and uses the opened image to spawn the emulator, meaning it is always available once installed, regardless of how the environment changes.<\/p>\n<\/blockquote>\n\n\n\nF<\/code> is set, the interpreter<\/code> entry in the runtime settings doesn’t mean the path of the interpreter it will<\/em> be called, but where it was<\/em> called at the time – ie. it’s irrelevant for the actual runtime.<\/p>\n\n\n\ntonistiigi\/binfmt<\/code> image ships its own static-compiled qemu-*<\/code> binarlies, as well as aptman\/qus<\/code> container gets the right ones at runtime (hence the slowness), and the interpreter<\/code> path is the binary inside the container when the command is run<\/em>. The binary is then kept in memory, and the container can go away, the interpreter<\/code> path’s not refering anything that exists any longer.<\/p>\n\n\n\nsystemd-binfmt<\/code> fail then? Well of course because it’s a dynamically<\/code> linked binary:<\/p>\n\n\n\n$ file \/usr\/bin\/qemu-x86_64\n\/usr\/bin\/qemu-x86_64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter \/lib\/ld-linux-aarch64.so.1, BuildID[sha1]=a4b8a93a4361be61dfa34a0eab40083325853839, for GNU\/Linux 3.7.0, stripped<\/code><\/pre>\n\n\n\nF<\/code> flag makes it stay in memory, its lib dependencies aren’t, so when in run in Docker (which uses namespaces) it doesn’t have everything to run…<\/p>\n\n\n\n\n
Updated Setup and Looking Forward<\/h2>\n\n\n\n
\n
systemd<\/code> service to run the tonistiigi\/binfmt<\/code> container on startup (which is possible<\/a>)<\/li>\n\n\n\nsystemd-binfmt<\/code> uses<\/li>\n\n\n\ncd $(mktemp -d)\ndocker create --name=\"tmp_$$\" tonistiigi\/binfmt\ndocker export tmp_$$ -o tonistiigi.tar.gz\ndocker rm tmp_$$\ntar -xf tonistiigi.tar.gz --wildcards \"*\/qemu-x86_64\"\n# Copy along the binaries folder:\nsudo cp usr\/bin\/qemu-x86_64 \/usr\/bin\/qemu-x86_64-static<\/code><\/pre>\n\n\n\nqemu-x86_64.conf<\/code> we do it again:<\/p>\n\n\n\n\n
\/usr\/lib\/binfmt.d\/qemu-x86_64.conf<\/code> to \/etc\/binfmt.d\/qemu-x86_64.conf<\/code> (make sure the file has the same name to ensure this new file overrides the one from the lib folder)<\/li>\n\n\n\n:\/usr\/bin\/qemu-x86_64:FP<\/code> to :\/usr\/bin\/qemu-x86_64-static:FPOC<\/code> (that is updating the binary it points at and the flags for good measure too<\/li>\n\n\n\n:qemu-x86_64:<\/code> in the front too, say to :qemu-x86_64-static:<\/code>, to change the display name of the emulator without affecting any of the functionality, it will just rename the entrin in \/proc\/sys\/fs\/binfmt_misc<\/code> <\/li>\n\n\n\nsystemd-binfmt<\/code><\/li>\n<\/ul>\n\n\n\n$ cat \/proc\/sys\/fs\/binfmt_misc\/qemu-x86_64-static\nenabled\ninterpreter \/usr\/bin\/qemu-x86_64-static\nflags: POCF\noffset 0\nmagic 7f454c4602010100000000000000000002003e00\nmask fffffffffffefe00fffffffffffffffffeffffff<\/code><\/pre>\n\n\n\nalpine<\/code>-based checks work once more.<\/p>\n\n\n\nLessons Learned<\/h2>\n\n\n\n
F<\/code> flag actually do? Can you run binaries while you don’t have them anymore? Dynamic and static linking and the signs of their misbehaviours to provide hints… However this is coupled with confusion when expectations are broken (say the interpreter<\/code> doesn’t have to refer to an actual file path that exists now<\/em>), until I started to question my expectations. Also, just being a heavy user of Docker doesn’t mean I’m knowledgeable of the relevant kernel functionality, and probably I should be more\u2026<\/p>\n\n\n\n\n
strings<\/code> as something that’s quick and simple to use .\/strings \/bin\/sh | head<\/code> for example, allowing fast iteration. \u21a9\ufe0e<\/a><\/li>linux\/arm64<\/code>, linux\/arm\/v7<\/code>, linux\/arm\/v6<\/code> images. For ArchLinux ARM it would be a different work the other direction. If only the main Arch would support ARM, it would be a happier world (even if even more complex). \u21a9\ufe0e<\/a><\/li>strings<\/code> as something that's quick and simple to use .\/strings \/bin\/sh | head<\/code> for example, allowing fast iteration.\",\"id\":\"edde2512-e0d8-4c40-8fc7-32ce3232f660\"},{\"content\":\"ArchLinux is x86 by default, for them it would be to emulate linux\/arm64<\/code>, linux\/arm\/v7<\/code>, linux\/arm\/v6<\/code> images. For ArchLinux ARM it would be a different work the other direction. If only the main Arch would support ARM, it would be a happier world (even if even more complex).\",\"id\":\"f5c8a1a6-6844-4956-adf5-488644168f92\"},{\"content\":\"Tongue-in-cheek, of course.\",\"id\":\"f632dbc3-dd26-4b73-b6cc-0b53e878e3dc\"},{\"content\":\"And with this we just rediscovered the Feynman Algorithm<\/a>, I guess.\",\"id\":\"692118e9-bc19-43d4-8ac0-1603d91facdf\"}]"},"categories":[7],"tags":[183,217,205,17,256],"class_list":["post-2967","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comp","tag-arch-linux","tag-archiveteam","tag-docker","tag-linux","tag-qemu"],"yoast_head":"\n