At $DAYJOB I was just working on some Google Sheets-backed dashboards using Tableau, when the team hit the following error message:

It says:

Unable to complete action
Bad Connection: Tableau could not connect to the data source.
Error Code: 3834D197
Errors occurred while loading the data source.

While debugging this, there were actually two aspects to take into account, that complicated things: one is the source of the error, and the other is Tableau’s behaviour when encountering the error. Both are good to keep in mind for future development.

Searching Tableau’s support didn’t yield much useful info: not that many people seem to have hit this issue, and when someone did, people were just guessing, and throwing things at the wall, see what sticks…

While trying to debug, managed to figure out that if I made a copy of the offending Google Sheet (spreadsheet), but removed one specific sheet (tab), then the data connection worked. On the other hand, when I removed the same tab from the original spreadsheet, and kept trying to connect, the error persisted…

Scrub Those Pivot Tables

First things first: what was missing in the copy compared to the original that allowed the connection to work?

Searching Tableau’s support didn’t yield much useful info: not that many people seem to have hit this issue, and when someone did, people were just guessing, and throwing things at the wall, see what sticks… Similar throwing stuff out happened on Reddit too.

Going through the suggestions, the first suspect was a chart, as the offending sheet had a waterfall chart near the top, but after a quick trial of whipping up a chart and connecting to Tableau (which worked), that hypothesis was refuted.

Buried in some of the other comments, however, there was a mention of pivot tables:

Was breaking my head over this error and as soon as I deleted the Pivot Table the error was fixed.

Checking for that, scrolling further down on our offending sheet (the one that we had to remove in the copy to make it work), there was indeed a pivot table.

For a test, creating a simple sheet with a pivot table in it, indeed I could trigger the error on demand, so this seems to be the proximate cause of our issue!

One lesson learned: Google Sheets¹ that are connected to Tableau directly shouldn’t have pivot tables in it.

Restart Tableau to Try Again

But this was only half of the story. The nagging feeling remained, why did my modified copy of the spreadsheet work, but if I’ve fixed the original sheet (deleted the pivot table) still throw the error when I tried to reconnect?

I’ve downloaded both sheets (as xlsx) and tried to compare, but couldn’t really see any material differences… Is it in the history? Is it something more subtle that is hidden in the structure of the doc, or how Google exposes the spreadsheets over the network connection versus the downloaded files?

For simplicity, I tried things out with intentionally triggering this error with a simple sheet, then fixing it, and trying to re-add. The error was again showing even through the sheet no longer had any pivot table…

The breakthrough came when I shut down Tableau, started up again, loaded the fixed sheet — and then it worked.

Seems like Tableau caches something between the attempts of connecting to the sheet (either the whole content, which I would doubt, or just the connection results?), so when we hit an error loading the spreadsheet, that error on the same error persisted while the Tableau sesson was running. And not knowing about that caching, and because our original dashboard took a couple of minutes to click through various data connector access requests when started up, I obviously wasn’t starting fresh whenever we made a chage to the breaking sheet. So we were just banging our head on the wall, while changing stuff to see if things worked, but Tableau’s caching hid it from us that we’ve fixed stuff…

That was an hour or two well spent! The lesson will definitely remain: if you hit an error loading a connector, just restart Tableau before trying again.

And the developer experience corollary: for goodness sake make Dashboards that do not take 3-5 minutes to reconnect to all the sources whenever starting up².

1. I guess that applies to Microsoft Office Spreadsheets too as Tableau uses the similar code internally, but I haven’t tested that. ︎
2. Easier said than done, I’m aware. ︎

The post Tableau’s 3834D197 error: Pivot Tables in Google Sheets appeared first on ClickedyClick.

It was around a time when I had a new laptop, and been trying to get things right installing ArchLinux, where both the processes and the hardware changed a lot since I’ve last had done it in… 2011-ish. So that involved many queries to the Internet. So much so, that my 300 searches were done on the 3rd day or so.

So 3 days into a month I already used up my quota, and couldn’t just upgrade to unlimited (yet), since it was a gift from the Kagi. So what could I do different in the future for a better experience?

Habit Changes

What does that better experience really mean? Doing a retrospective of how was I searching, it seemed to me a really mindless, throwaway process:

• plopping in keywords, scrolling quickly around, not really clicking on any link necessarily, but changing the word and re-running the search (total shotgun approach)
• running the same query again and again across days
• relying on the search engine to get slowly changing or unchanging information

… and more. I feel like these are the habits from other search engines, where I didn’t find things, when most pages are “sponsored”, when I didn’t put in effort to check the queries I run just sent them off, and could do because it was “all you can eat”, and that made me not really pay attention to the “taste & flavour” of the results…

Bookmark more & better

One of the most obvious idea I had was that since I go to the same pages all the time, why not save those links in bookmarks?

I used to bookmark a lot, and organise those bookmarks, etc… Then I gave it all up, because … I could just search things? And because organisation wasn’t that simple either. Which folder does this link go to? Is it under Programming > Languages > Python? Or under Professional Development? Or under Useful Libraries?…

The same perfectionist “it doesn’t just need to be bookmarked, it has to be filed away correctly too!” was really not doing me any good service… It took a little while to work through this, and just settle on putting every bookmark in a flat hierarchy, and use tagging to help me find them (rather than folders). Difference being, that a single link can have multiple tags, but can exist in only one folder. I do believe that’s the only scalable way, ever since I was trying to do the other way around (and failing)¹.

So bookmark:

• the (book) library catalogues that I borrow books from
• the forums and docs pages of projects I’m using²
• the useful tools available online, from timezone wrangling to currency conversion…
• blogs, publications, web comics that I frequent…

and so on… Now whenever I visit a page, I do stop for a moment and think: could I imagine myself wanting to come back here in the future? If yes, let’s bookmark.

It’s not all perfect, but it’s more about the tools than the process: Firefox on Android doesn’t seem to handle the same “search in bookmarks” shortcuts, or make it more difficult to do. Oh, well, eventually…

Add custom search engines

Sometimes I the page I want to go to is within a large collection, such as a wiki, or a forum. I know it’s there, but unsure where exactly. Bookmarks take me to the page, but then I have to use the search as a second step. This can be made a lot more ergonomic in pretty much every current browser by adding custom search engines³.

Adding Wikipedia’s search? It might already be there. Adding ArchLinux Wiki? It takes 15 seconds to do, and I have a shortcut to so much Linux system admin knowledge that is riddiculous. Whatever site has a search, can be added just as simple.

Here the kicker is to remember that shortcut (and the fact that I’ve added that shortcut), but after that, it’s off to the races.

Change how I search

While I was auditing what sort of searches I like to do, one type that stood out was when I was asking for something like “what’s the website of this-or-that company or project or organisation?” More often than not, these are companies, projects, and organisations that are notable enough to be in Wikipedia. Here’s the new process: search Wikipedia or the company/project/organisation and use the link from their page.

This also feel more to the point than search for “<companyname> website” which is just “close enough” in meaning, and will still get me many results, even if the answers should be a single value.

This is caveated by that for programming / open source projects, the better search is probably GitHub / GitLab / CodeBerg, where they are likely hosted (in decreasing probability, currently), and switch to search engine search when that fails.

This is aling the lines that if I already know an authoritive source for the information, I should probably go there directly?

Misc

Bookmarks & custom searches brought down my search count already. One that is more of a housekeeping change is that my browser was reopening pages from my previous session whenever I started it. If I had any Kagi search results open, that just used up another in the quota, and there often there were more than one open… Setting my browser to start afresh on each time I open it helped with that — and also helped with me not being distracted every time I open my browser by whatever I was doing last time, as opposed to what I wanted to do now.

What did I Learn?

Now that I’m on a proper paid plan, I will up it to the Professional plan, where searches are not metered. It doesn’t feel like just a lazy release valve⁴, rather because I don’t believe this sorts of limit to my access information is productive. “Limited” limitations, when there’s a purpose, can indeed be “cretive limitations”.

If I believe that Kagi does a good job, then there’s no point sticking to the quota; if I don’t, then why am I using it in the first place, instead of any of the alternatives?

And if I want to use the power of creative limitations, I can always do my own quest with rules like no search, it’s within my power.

I do feel that the changes to my thinking due to this experience — being more deliberate of what am I looking for and thoughful about where might I find them; choosing and rewarding sources I find useful and reliable; using the little gray cells more — I want to keep and even cultivate. These changes also brought back a more old-school internet vibes (old as in when I bought printed magazines that came with collected links of what you can find on the World Wide Web, something more tangible and purposeful). I guess I’m getting old as well. :)

1. Tagging also got a big push from and also due to a push from the book Everything is Miscellanious. It’s also why e.g. Gmail was awesome to have tags while other clients were still just doing folders. Fastmail goes even further, lets you choose labels (tags) or folders, which is pretty awesome of them. ︎
2. Almost every good project has a good forum/docs, or maybe a good forum/docs contributes to the project being good? Here’s looking at you ArchWiki, Obsidian Help,… ︎
3. For example in Firefox. ︎
4. As in “if I’m unlimited, I don’t have to care about all the effort I’ve described so far, I can go back to my old habits”. ︎

The post How to Think Differently When Internet Searches Are Metered appeared first on ClickedyClick.

Setting up my laptop is now in its 3rd day, and the experience made me think of incremental games. Incrementals¹ are where you make some progress in gathering some resource, then have to reset your progress that gives you some small buff or gain. You then start again – but better. The cycle repeats and you might have the same experiences countless times, but after a while the game can be much faster or even unrecognisable due to the accumulated effects.

While I was installing Arch Linux this time I went through the following cycle:

• Partitioned my disk and installed the system, but then couldn’t boot into it
• Redo the partitioning, system, and bootloader install, now I can boot into it, but don’t have any network access
• Redo the system config, now I have network access, but have to figure out what desktop environment am I going to run, and what do I need for that. A hot mess ensues after trying out all main desktop environments for kicks
• Reinstall the system, cleaner, with my desktop environment of choice, now the high resolution environment makes everything either: tiny, gigantic, and/or blurred.
• Sorted out most of the sizes of things, have network, have Bluetooth, but the sound and the media buttons don’t work
• Sorted out sound, now the power management locks me out while watching a video….

… and so on. I know I have a few more rounds (few more “crunches”) to do get there, and the multilingual typing input setup will be a doozie with bopomofo, but I’m getting better and feeling better every round – that’s how a good incremental game goes. A power up there, a know-how of a semi-obscure, quality of life config here².

Of course I could have gotten an Ubuntu or Fedora image, installed everything in way less time than writing about the experience, and could already be using it — but that’s a whole different game³.

The games we choose to play show our values. On an optimistic day I feel I value increased knowledge; on more realistic days it certainly seems like procrastination. Now let’s keep this in mind when I finally have high enough level of Linux buffs that they change the game mechanics and I get to do something with my computer.

1. Keep away from Universal Paperclips and give a very wide berth to Antimatter Dimensions ︎
2. Such as enabling tapping on the touch pad for a click, without needing to actually press…. ︎
3. A game that might be won, while I’m not sure whether the one I’ve chosen has a win condition. ︎

The post Installing Arch Linux as an incremental game appeared first on ClickedyClick.

This is a debugging tale for those working with containers, and a reminder that things aren’t always what they seem in Linux, all with a big pinch reminder to Read the Fine Manual, Always!

ARM with Achiveteam v2

Recently I’ve got an email from a reader of the ARM images to help the Archive Team blogpost from years ago, asking me about refreshing that project to use again. There I was recompiling the ArchiveTeam’s Docker images to support ARM, and thus I was looking how things changed in the intervening time. I also got more lazy pragmatic since then, I was was wondering if the Archiveteam just made some ARM or multi-arch images as I believe(d) they should. That lead me to their FAQ entry about ARM images:

Can I run the Warrior on ARM or some other unusual architecture?

Not directly. We currently do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident that they run correctly under (among other things) different endiannesses. […]

Set up QEMU with your Docker install and add –platform linux/amd64 to your docker run command.

This actually seems like a sensible thing – if they dug that deep that they’ve seen issues in wget, I’ve definitely been doing things naively before.

The guidance of installing QEMU seems sensible as well (we were doing a lot of those at balena), and it goes roughly like.

1. install binfmt
2. install QEMU with statically compiled binaries
3. load those binaries to emulate the platforms you want with the F / fix_binary flag

For those unfamiliar, binfmt_misc is a Linux kernel feature that allows non-native binary formats to be recognized and passed to user space applications. It’s what makes it possible to run ARM binaries on x86 systems and vice versa through emulation. The various flags are how the actual behaviour of binfmt is adjusted (F, P, C, O)

Docker advised to use a image to set things up, that is, for example for the x86_64/amd64 platform like this:

docker run --privileged --rm tonistiigi/binfmt --install amd4

My Raspberry Pi is running ArchLinuxARM which installs systemd-binfmt to load the relevant emulation settings at boot time, which seemed handy: with the docker method I had to run that every time before I could run an emulated container, with systemd I would have thing ready by every time the time Docker is ready to run (ie. keeping the Archiveteam containers always on and restarting after reboot.) So I have a strong incentive to use the systemd-based approach instead of the docker run based one.

Now comes the kicker :

• the docker installed binfmt setup worked and allowed to run linux/amd64 containers
• systemd-binfmt initiated binfmt setup worked for the x86_64 binaries in the file system, but not in Docker where the binaries just failed to run
• both setups had identical output when looking at the config in /proc/sys/fs/binfmt_misc

When Same’s Not the Same

To see whether emulation works, the tonistiigi/binfmt container can be invoked without any arguments and it shows the status. For example setting things up with docker would show:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/amd64/v2",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-x86_64"
  ]
}

Here the supported section shows amd64 as it should, and their test of running an amd64 image to check if the binaries are run has the expected output:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
x86_64

Going back to the alternative, after uninstalling that emulatior I start up systemd-binfmtI can test the status again:

$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
[...snip...]
    "qemu-x86_64",
[...snip...]
  ]
}

This shows that while the emulator is installed, Docker doesn’t find that the linux/amd64 platform is supported, and this checks out with running the alpine image again as above:

$ docker run --rm --platform linux/amd64 -t alpine uname -m
exec /bin/uname: exec format error

Well, this doesn’t work.

The binfmt_misc docs in the Linux Kernel wiki do have plenty of info on the setup and use of the that emulation function. For example to check the configuration of the emulation setup, we can look at the contents of a file in /proc filesystem:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /usr/bin/qemu-x86_64
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

This was the almost the same whether I the docker based setup or used systemd-binfmt with a slight difference: the flags bit is only PF when run with systemd-binfmt, and POCF when set things up with docker run. Even if the Docker docs are asking for the F flag, I wanted to make sure we are on equal footing, so I’ve tried to modify the QEMU setup to be the same. This means overriding the qemu-x86_64.conf that is shipped by default:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :FP to :FPOC
• restart systemd-binfmt

After this the output of the the runtime info in /proc/sys/fs/binfmt_misc/qemu-x86_64 was completely the same. Why’s the difference?

More debugging steps ensued:

More Debugging Ensued

I’ve read through the source code of tonistiigi/binfmt on GitHub and seen that it doesn’t do anything fancy, it’s quite clear implementation of the `binfmt_misc` usage docs and the same magic values as QEMU shipped on my system. Good that no surprise, but no hints of any difference

I’ve tried to replicate that process of setting up QEMU through translating it into Python and running, still the same

I’ve recompiled the binary on my system, and run it outside of docker: it worked the same way as the systemd-binfmt setup: x86_64 static binaries¹ worked outside of Docker but not inside of it

A sort-of breakthrough came when I’ve tried out dbhi/qus Docker images, that promises “qemu-user-static (qus) and containers, non-invasive minimal working setups”, and can do the similar emulator & platform support setup with:

docker run --rm --privileged aptman/qus -s -- -p x86_64

It was a lot slower to run (coming back to this later), but worked like the charm, just like Docker’s own recommendation. However there was a difference in the outcome when I checked the runtime config info:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /qus/bin/qemu-x86_64-static
flags: F
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

It has just the apparently required F flag, but the interpreter points to /qus/bin/qemu-x86_64-static … which is not in the regular file system. Nevertheless alpine happily runs, just as my local static binaries.

How does this actually work, then?

Everything’s Illuminated

With this above, and with a better understanding what the docs say, we have everything in place to understand the all the behaviours above, things we had pointers throughout, but not enough experience to put them together:

So, the F flag was required by the Docker docs, what does that actually do?

F – fix binary

The usual behaviour of binfmt_misc is to spawn the binary lazily when the misc format file is invoked. However, this doesn’t work very well in the face of mount namespaces and changeroots, so the F mode opens the binary as soon as the emulation is installed and uses the opened image to spawn the emulator, meaning it is always available once installed, regardless of how the environment changes.

Because of this, if F is set, the interpreter entry in the runtime settings doesn’t mean the path of the interpreter it will be called, but where it was called at the time – ie. it’s irrelevant for the actual runtime.

The tonistiigi/binfmt image ships its own static-compiled qemu-* binarlies, as well as aptman/qus container gets the right ones at runtime (hence the slowness), and the interpreter path is the binary inside the container when the command is run. The binary is then kept in memory, and the container can go away, the interpreter path’s not refering anything that exists any longer.

Why does systemd-binfmt fail then? Well of course because it’s a dynamically linked binary:

$ file /usr/bin/qemu-x86_64
/usr/bin/qemu-x86_64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=a4b8a93a4361be61dfa34a0eab40083325853839, for GNU/Linux 3.7.0, stripped

… and because it’s dynamically linked, even if the F flag makes it stay in memory, its lib dependencies aren’t, so when in run in Docker (which uses namespaces) it doesn’t have everything to run…

And of course, ArchLinux spells this out:

Note: At present, Arch does not offer a full-system mode and statically linked variant (neither officially nor via AUR), as this is usually not needed.

Yes, “as this is usually not needed”. :)

Updated Setup and Looking Forward

Sort of lobbying ArchLinux to have static QEMU² what options do I have?

• set up a systemd service to run the tonistiigi/binfmt container on startup (which is possible)
• get some static QEMU binaries and override the settings that systemd-binfmt uses
• switch to anothe Linux Distro that supports the Pi, the software I run, but also ships static QEMU builds

All three are suboptimal, potentially fragile, and the third is way too much work. Still the second one was kinda fine:

cd $(mktemp -d)
docker create --name="tmp_$$"  tonistiigi/binfmt
docker export tmp_$$ -o tonistiigi.tar.gz
docker rm tmp_$$
tar -xf tonistiigi.tar.gz --wildcards "*/qemu-x86_64"
# Copy along the binaries folder:
sudo cp usr/bin/qemu-x86_64 /usr/bin/qemu-x86_64-static

Then just like we’ve overridden the upstream qemu-x86_64.conf we do it again:

• Copy the config from /usr/lib/binfmt.d/qemu-x86_64.conf to /etc/binfmt.d/qemu-x86_64.conf (make sure the file has the same name to ensure this new file overrides the one from the lib folder)
• Edit the end of the line from :/usr/bin/qemu-x86_64:FP to :/usr/bin/qemu-x86_64-static:FPOC (that is updating the binary it points at and the flags for good measure too
• As a bonus, can update the :qemu-x86_64: in the front too, say to :qemu-x86_64-static:, to change the display name of the emulator without affecting any of the functionality, it will just rename the entrin in /proc/sys/fs/binfmt_misc
• restart systemd-binfmt

Then the check again:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64-static
enabled
interpreter /usr/bin/qemu-x86_64-static
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

And the alpine-based checks work once more.

Lessons Learned

The details were all in plain sight, but not enough experience to piece these things together. The Docker-recommended image ships its own QEMU? What does that F flag actually do? Can you run binaries while you don’t have them anymore? Dynamic and static linking and the signs of their misbehaviours to provide hints… However this is coupled with confusion when expectations are broken (say the interpreter doesn’t have to refer to an actual file path that exists now), until I started to question my expectations. Also, just being a heavy user of Docker doesn’t mean I’m knowledgeable of the relevant kernel functionality, and probably I should be more…

This whole process underlined my previous thoughts on Software Engineering when AI seems Everywhere, as I did try to debug things by rubber ducking with Claude: this time the hallucinations were through the roof (a metric tonne of non-existent systemd funcionality, non-existent command line flags), definitely got me on a wild goose chase in a few cases. So even more care’s needed, maybe a version of Hofstadter’s Law:

Imreh’s Law³: LLMs are always more wrong than you expect, even when you take into account Imreh’s Law.

In the end, Don’t Panic, make theories and try to prove them, and talk with anyone who listens, even when they are wrong, and you are more likely to get there⁴.

1. I’ve download static binaries from andrew-d/static-binaries, recommend strings as something that’s quick and simple to use ./strings /bin/sh | head for example, allowing fast iteration. ︎
2. ArchLinux is x86 by default, for them it would be to emulate linux/arm64, linux/arm/v7, linux/arm/v6 images. For ArchLinux ARM it would be a different work the other direction. If only the main Arch would support ARM, it would be a happier world (even if even more complex). ︎
3. Tongue-in-cheek, of course. ︎
4. And with this we just rediscovered the Feynman Algorithm, I guess. ︎

The post The curious case of binfmt for x86 emulation for ARM Docker appeared first on ClickedyClick.

I genuinely think that putting the AI/LLM genie back into the bottle is unlikely (the same way as some people want the Internet, or smartphones, or cryptocurrencies put back into the bottle, which also not really gonna happen). That doesn’t mean that uncritical acceptance of the coding assistant tools should be the norm, au contraire, just like any tool, one needs to discover when they are fit for for the job, and when they are not. I have used GitHub CoPilot for a while, now digging into Cursor as it starts to conquer the workplace, and ChatGPT & Claude for individual coding questions. I don’t think it’s controversial to say that all these tools have their “strengths and weaknesses”, and that currently the more complex, more “production” the problem is, the further away it is from a proof-of-concept, the less likely these tools are of any help. They are help, they can be a large force multiplier, but they are big multiplier when one goes in with the least amount of input (knowledge, awailable time, reqirements for the result…)

On the day-to-day use I’ve ended up usually in one of these cases:

• the suggested results were wrong: clearly when I was lucky, and subtly when I wasn’t;
• when the results were correct I had to spend tuning the prompts, questions, and guidance until the invested effort was similar to doing the work myself;
• when the results were correct and they weren’t too much work they addressed simple stuff (such as writing structured documentation), in which case the tools indeed got me into a nice flow state;

The proportion of the above cases was about 70/20/10%, respectively².

This experience made me pause and think about what influences this status quo in the areas that I can control? For example I cannot (really) control how capable the various models are, they will be improving over time for sure. On the other hand, is there something that I could do differently to have better outcomes?

Stepping all the way back, the universal rule of thumb of garbage in, garbage out hasn’t failed me yet, so working on better quality input seems like a good place to start. And better input only happens when I’m better at the things I’m doing. Wherever I look, that getting better is coming from focused, significant, and directed effort.³

What can I do be more effortful – in the right way? What are those software engineer activities that require effort, but not toil? Looking back my past experience, so far I’ve found three main areas:

Work on debugging skills. How to read code, how to approach a brand new software system and build up model of how it works? Programming is theory building, and so the better (more correctly, faster, with more insights) you can build your theories, the better the ultimate outcomes will be.

Work on testing skills: once you know how things work, you can know better how they can go wrong, and better, you can prevent things going wrong. However not all tests are created equal: coding assistants these days can create tests very easily (tests often need a lot of boilerplate, and boilerplate is easy to generalise, exactly the coding assistants bread & butter). They seem less good figuring out what tests really move the neadle increase our certainty that things work well – and those are the tests that are really needed.

Work on software architecture skills: making something complex and complicated is less work, than making something simple and elegant. Simplicity and elegance is bot just for its own sake. Simple is easier to reason about, easier to maintain and evolve, more likely to be correct. Can coding assistants do this? So far they tend towards verbocity which breeds complexity. They are very local and thus much less holistic. They learn from the entire Internet, but that also means that they have to have extremely good attention to find what’s relevant…

These three are programming language, business context, job role agnosic, and I find it difficult to imagine an engineer role that doesn’t need them. I guess as long as humans do software engineering, these shall serve people right.

At the end of the day, instead of speeding up with coding assistants, I’ll slow down. Instead of the volume, I try focusing on quality. The assistants still have potential space there: rubber ducking, as an advanced (even if flawed) search engine, and to code the parts that have less value for a given set of goal. There machines can do the work so people have time to think⁴.

1. It might be my own bubble, being a machine learning engineer by the day. ︎
2. With better models I’d guess the cases will be the same, but the proportions will shift. ︎
3. Let this be physical such as exercise, or mental like it’s advocated in How to Read a Book. ︎
4. Rather than the other way around. ︎

The post Software Engineering when AI seems Everywhere appeared first on ClickedyClick.

The story starts with me rediscovering a DVB-T receiver USB stick, that I had for probably close to decade. It’s been “barnacled” by time spent in the Taiwanese climate, so I wasn’t sure if it still works, but it’s such a versatile tool, that it was worth trying to revive it.

When these receivers function, they can receive digital TV (that’s the DVB-T), but also FM radio, DAB, and also they can act as software defined radio (SDR). This last thing makes them able to receive all kinds of transitions that are immediately quite high on the fun level, in particular airplane (ADS-B transmission) and ship (AIS) tracking. Naturally, there are websites to do both if you just want to see it (for example Flightradar24 and MarineTraffic, respectively, are popular aggregators for that data but there are tons), but doing your own data collection opens doors to all kinds of other use cases.

So on I go, trying to find, what software tools people use these days to use these receivers. Mine is a pretty simple one (find out everything about it by following the “RTL-SDR” keywords wherever you like to do that :) and so I remembered there were many tools. However also time passed, I forgot most that I knew, and also there were new projects coming and going.

ADSBox

While I was searching, I found the adsbox project, that was interesting both kinda working straight out of box for me, while it was also last updated some 9 years ago, so it’s an old code base that tickles my “let’s maintain all the things!” drive…

The tool is written mostly in C, while it also hosts its own server for a web interface, for listing flights, and (back in the day) supporting things like Google Maps and Google Earth.

Both the Google Maps and Earth parts seem completely: Maps has changed a lot since, as I also had to update my Taiwan WWII Map Overlays project over time too (the requirement of using API keys to even load the map, changes to the JavaScript API…). Earth I haven’t tried, but I’m thinking that went the way of the dodo on the the desktop?

So in the supur of the this-is-the-weekend-and-I-have-energy-to-code moment, I started to think of the options:

• could fix up the map, either with the Google Maps changes, or bring in some other map?
• the project has barely any readme, and I mainly managed to make it work by looking at old articles from the time when adsbox waas new, could fix those up?
• during the compilation, loads of warnings happened, that seem to call for some “better quality” coding, let’s fix stuff until -Werror (making all warnings errors) passes too! This would be a learning experience
• I’m sure I can find other tasks to do as well, like an error message here, a strange behaviour there…

Here’s the kicker though: I don’t really know C. I spend most of my time in Python-land, and haven’t done a C project in anger yet. Is it worth trying to dig in, while there are other ADS-B projects that a) work better, b) are in languages that I’m more looking to learn, such as Rust?

There was an additional drive of curiosity, just like in my last post: can I use Large Language Models (LLMs) to complement me on things I lack, such as knowledge of the exact programming language at hand?

With this I thought let’s dig in, and let’s dig into the C code: that seemed immediately tractable, more limited in scope, and thus would help build up (hopefully) some successes and I’ll learn my way around the codebase better.

On the LLM side I have GitHub Copilot – though it seems somewhat crippled in my open source Code Server installation of VS Code, rather than the official VS Code, in particular the context menus and Copilot Chat seems missing, and thus it was only communicating with me through TAB-completions and me adding comments to guide or suggest. That’s not very practical, so didn’t push it too far for the relevant tasks of explanation and exploration of options that I wanted to do.

I also have Claude that I can chat with. If I wasn’t working on my 13 year old Lenovo ThinkPad X201, I’d probably set up Ollama, but that’s just excruciating with even the smallest models on this machine (until I upgrade something newer, or run the questions on my work M1 MacBook). So Claude it is for now.

Hello Fixes

I guess it’s one sign of hubris (or unlimited optimism), to jump into fixing compilation warnings, without knowing anything much of the codebase yet. This started in areas where the airplane tracking interacts with SQLite, for example had warnings about casting pointers to integers of different size while shuffling around SQLite query results:

int * t = (int *) sqlite3_value_int(argv[0]);

This was also part of a larger code section (formatting integers into hexadecimal or octal strings, for example for the ICAO codes…), and thus had to play around how much context to give to Claude to actually have something useful.

A bit of mocking around there seemed to have worked, and while I should have asked more software architecture & best practices questions, probably knew about it enough to be dangerous, and left it as it was so far.

Having said that, after this change it turned out that some part of the interface now displaying stuff differently: the 24-bit ICAO airplane registration codes had useless leading zeros for 8 hex digits, rather than the expected 6 digits – since the fix was done without this context. Here we go, manual adaptation on this regression.

Now there were cases when “sprintf may write a terminating nul past the end of the destination“, as the code seems to have written its data back into the same place as this:

sprintf(data->avr_data, "*%s", data->avr_data + 13);

This ended up being again about a much bigger context (interative reading, processing, and passing on of recorded ADS-B packets), where based on the Claude’s suggestions I couldn’t really get to anything useful. The real point was always one step further:

• instead of the line look at the nearby lines
• instead of the near by lines, look at the whole funciton
• instead of the whole function, look at the wider codebase with its configuration

These are of course no-brainers. However Claude with its chat interface cannot really do that, while Copilot without its chat interface also cannot do this digging. Catch-22? Since in the end I admitted myself (for the nth time) that I need to understand the purpose of the code better before “fixing” it. Then due to the lack of comments in the codebase + lack of natural intuition of the built in C functions’ behaviour, I’ve just left them as they were for now, since they do work.

From here I turned to other parts. The webserver was not serving some files with the correct MIME type, due to its hand-rolled file extension extraction (splitting filenames at the first . rather than the last), this was easy to fix – with a bit of StackOverflow this time, rather than asking Claude.

Then there was an issue with the tool apparently not playing back the recorded packet data, which I fixed with a combo of regular ol’ debug printouts, StackOverflow, and just thinking about how it could work (it’s the issue of explicitly filling in daylight saving data in the relevant tm struct – tm_isdst – and thus IMHO it’s doing a regular “undefined” behaviour: in this case jumped the first timestamp’s time ahead by an hour, and thus would have needed to wait an hour to pass as the playback (following the real timingof the packets) catch up and start actually replaying. Still weird, why only the first packet’s data was shifted, and could I do a more solid fix than setting it once as the code never seem to overwrite it? These are the questions that are more addressing C-knowledge or potential best practice of the code’s structure overall…

Finally I’ve started on replacing Google Maps with OpenFreeMap and got as far as displaying the map (which is the easy step:). The whole replacement would likely be a lot more, also given the amount of barely documented JavaScript code in the project – but hopefully I have more working knowledge of JS than C.

Lessons Learned

First lesson is that I likely have a “saviour complex”, trying to fix up every code I see being imperferct in some way, whether or not I am capable of doing it or not. This is something to meditate further on for sure.

When using LLMs for code work, they are just as useful as another mid-level coder without much context – almost not at all. The context of code is always relevant, so either the LLM would have to get it itself, or the person pairing with the LLM would have to provide it. Thus the work is always there, just not always possible.

It’s very nice that I can do things in programming languages that I don’t really understand, but that’s only the case if I either spend much-much time actually getting to know things so I can start to judge whether the changes even have a chance to be correct or not; or I don’t care whether they are correct or not (but is this really an option?)

Overall the LLMs need the same things as humans to do a good job, and cannot pretend that they really can do work without these (even if they might appear being able to do without these for some time):

• good comment in the code so the intention can be ascertained as well
• tests that show what the correct behaviour should be, and catch regressions or unintentional breakages
• have domain knowledge to form better mental models about what should happen

The first two wasn’t true in this project. The last point is likely where LLMs are ahead in cases like this (having been trained on “all the Internet’s data”), though wouldn’t be the same for some niche, or work internal projects.

The LLMs suggestions are still ver much localised, thus they cannot really fix up the structure of the code too much – or maybe I’m not using the right tools, of course. And this is where my future big ask would lie: don’t just tell me how to fix this line, rather tell me that the entire block is no longer needed / could be merged with another part of the code / could be broken out to its own module that would help over there… Of course, this is moving the goal post a bit of what LLM programmers’ look like, though I also think that the current “fix this line” is something I most definitely want to have enough practice with that I don’t really need to ask (though it could suggest if there are good practices I haven’t picked up yet).

Where do I go from here?

This adsbox project is mostly obsolete, as I’ve found a bunch of other tools that are better, and better supported now (adsb_deku, tar1090), but surprising it still have stuff that are better here and in other tools (the plane’s status icons, some data displayed here that is not in others, showing what sort of packets (what Downlink Format or DF numbers) were received for the aircraft, etc… So there might be still value in using it occasionally, so there might be value.

Even if I could get a kick out of it, it’s likely useful to keep things time-boxed or constrained to some topics: change the map; add comments as I find them; fix issues if they arise; package it up for ArchLinux. That’s about it, but these should be generally useful (e.g. using OpenFreeMap for other projets in the future or rewriting the aforementioned Taiwan WWII Map project to use that).

My current fixes live in my fork in GitHub imrehg/adsbox, with no guarantees. Since the project also doesn’t have a license (just a note of “free for non-commercial use”, which doesn’t cover modifications), I’m probably keeping it simple for now.

I also got the hang of software defined radio again, and there’s just so much fun to have…

What’s the most useful is seeing in practice, what does software need to be maintainable almost a decade later, and what’s missing in most projects: explanatory comments to understand what is being done and why, and tests to know whether things are running correctly or not. And maube then both my future self, my colleagues, and any potential AI pair programmer would have a better chance of succeeding at “maintain all the things!”

The post Refreshing Airplane Tracking Software With and Without AI appeared first on ClickedyClick.

It looks interesting and informative. And even though I don’t have codebases that have been around this long, there are plenty of codebases around me that are fast moving, so something like a month (or in some cases week) level cohorts could be interesting.

One way to take this challenge on is to actually sit down and write the code. Another is to take a Large Language Model, say Claude and try to get that to make it. Of course the challenge is different in nature. For this case, let’s put myself in the shoes of someone who says

I am more interested in the results than the process, and want to get to the results quicker.

See how far we can get with this attitude, and where does it break down (probably no spoiler: it breaks down very quickly.).

Note on the selection of the model: I’ve chosen Claude just because generally I have good experience with it these days, and it can share generated artefacts (like the relevant Python code) which is nice. And it’s a short afternoon. :) Otherwise anything else could work as well, though surely with varying results.

Version 1

Let’s kick it off with a quick prompt.

Prompt: How would you generate a chart from a git repository to show the age of the code? That is when the code was written and how much of it survives over time?

Claude quickly picked it up and made me a Python script, which is nice (that being my day-to-day programming language). I guess that’s generally a good assumption these days if one does data analytics anyways (asking for another language is left for another experiment).

The result is this this code. I’ve skimmed it that it doesn’t just delete all my repo or does something completely batshit, but otherwise saved in a repo that I have at hand. To make it easier on myself, added some inline metadata with the dependencies:

# /// script
# dependencies = [
#   "pandas",
#   "matplotlib",
# ]
# ///

and from there I can just run the script with uv.

First it checked too few files (my repository is a mixture of Python and SQL scripts managed by dbt), so had to go in and change those filters, expanding them.

Then the thought struck me to remove the filter altogether (since it already checks only files that are checked in git, so it should be fine – but then it broke on a step where it reads a file as if it was text to find the line counts. I guess there could be a better way of filtering (say “do not read binary files”, if there’s a way to do that), but just went with catching the problems:

# ....
    for file_path in tracked_files:
        try:
            timestamps = get_file_blame_data(file_path)
            for timestamp in timestamps:
                blame_data[timestamp] += 1
                total_lines += 1
        except UnicodeDecodeError:
            print(f"Error reading file: {file_path}")
            continue
#....

(hance I know that a favicon PNG was causting those UnicodeDecodeError hubbub in earlier runs. Now we are getting somewhere, and we have a graph like this:

This is already quite fun to see. There are the sudden accelerations of development, there are the plateaus of me working on other projects, and generally feel like “wow, productive!” (with no facts backing that feeling ). Also pretty good ROI on maybe 15 mins of effort.

Having said that, this is still fair from what I wanted.

Version 2

Promt: Could we change the code to have cohorts of time, that is configurable, say monthly, or yearly cohoorts, and colour the chart to see how long each cohort survives?

This came back with another set of code. Adding the metadata, skimming it (it has the filter on the file extensions again, never mind), and running it once more to see the output, I get this:

Because of the file extension filter in place, the numbers are obviously not aligning with the above, but it does something. The something is a bit unclear, bit it feels like progress, so let’s give it a benefit of the doubt, and just change once more.

Version 3

Promt: Now change this into a cummulative graph, please.

One more time Claude came back with this code. Adding the metadata again, same drill. Running this has failed with errors in numpy, though:

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Now this needed some debugging. It turns out a column the code is trying to plot is actually numbers as strings rather than numbers as, you know, say floats…

# my "fix"
        df['cumulative_percentage'] = df['cumulative_percentage'].astype(float)
# end

        # Plot cumulative area
        plt.fill_between(df.index, df['cumulative_percentage'],
                        alpha=0.6, color='royalblue',
                        label='Cumulative Code')

It didn’t take too many tries, but it was confusing at first – why shouldn’t be, if I didn’t actually read just skim the code…

The result is then like this:

Sort of meh, it feels like it’s not going to the right direction overall.

But while debugging the above issues, I first tried tried to ask Claude about the error (maybe it can fix it itself), but came back with “Your message exceeds the length limit. …” (for free users, that is). So I kinda stopped here for the time being.

Lessons learned

The first lesson is very much re-learned:

Garbage in, garbage out.

If I cannot express what I really want, it’s very difficult to make it happen. And my prompts were by no means expressing my wishes correctly, no wonder Claude wasn’t really hitting the mark. Whether or not a human engineer would have faired better, I don’t know. I know however, that this kind of “tell me exceedingly clearly what’s your idea” is an everyday conversation for me as an engineer (and being on both end of the convo).

The code provided by the model wasn’t really far off for some solution, so that was fun! On the other hand, when it hit any issues, I really had to have domain and language knowledge to fix things. This seems like an interesting place to be:

• the results are quick and on the surface good-enough for a non/less technical person, probably
• but they would also be the ones who couldn’t do anything if something goes wrong.

Even myself I feel that it would be hard to support the code as a software engineer if it was just generated like this. But that’s also a strange thought: so many times I have to support (debug, extend, explain, refactor) code that I haven’t had anything to do with before.

It seems to me that now that since Claude comes across as an eager junior engineer, writing decent code that always needs some adjustments, the trade-off is really in the dimension of spending time to get better at prompting vs better at coding.

If there’s a person with some amount of programming skills, mostly interested in the results not the process, and doubling down on prompting: they likely could get loads further than I did here. Good quality prompts and small amount of code adjustments being the sweet spot for them.

For others who have more programming expertise, and maybe more interested in the process, spending time on getting better at programming rather than getting really better at prompting: keeping to smaller snippets might be the sweet spot, or learning new languages, … Something as a starting point for digging in, a seed, is what this process can help with.

Future

Given the above notes on how this generated code is like a new codebase that I suddenly neet to support, here’s a different, fun exercise to actually improve engineering skills:

Take AI generated code that is “good enough” for a small problem and refactor, extent, productionise it.

I’m not sure if this would work, or would get me into wrong habits, but if I wanted do have some quick ways of doing deliberate practice – and not Exercism, LeetCode, or somilar, rather something that can be custom made, then this seems a way to get started.

Also, now that I’ve gotten even more interested in the problem, I’ll likely just dig into how to actually define that chart I was looking for and what kind of data I would need to get from git to make it happen. The example code made me pretty confident, that “all I need is Python” really, even though while prepping for this I found other useful tools like one allowing you to write SQL queries for your repo, that might be some further way to expand my understanding.

Either way, it’s just fun to mess with code on a lazy Saturday.

The post Adventures into Code Age with an LLM appeared first on ClickedyClick.

Access Options

In broad terms git repositories can be checked out either though the HTTP protocol, or through the SSH protocol. Both have pros and cons.

Having two-factor authentication (2FA) made the HTTP access more secure but also more setup (no more direct username/password usage, rather needing to create extra access keys used in place of passwords). Credentials were still in plain text (as far as I know) on the machine in some git config files.

The SSH setup was in some sense more practical one (creating keys on your own machine, and just passing in the public key portion), though there were still secrets in plain text on my machine (as I don’t think the majority of people used password-protected SSH keys, due to their user experience). This is what I’ve used for years: add a new SSH key for a new machine that I’m working on, check code out through ssh+git, and work away.

When I’ve recently came across the git-credential-manager tool that supposed to make HTTP access nicer (for various git servers and services), and get rid of plain text secrets. Of course this is not the first or only one of the tools that does git credentials, but being made by GitHub, it had some more clout. This made me re-evaulate what options do I have for SSH as well for similar security improvements.

Thus I’ve found that both 1Password and KeePassXC (the two main password managers I use) have ssh-agent integration, and thus can store SSH keys + give access to them as needed. No more plain text (or password protected) private keys on disk with these either!

Now it seems there are two good, new options to evaulate, and for the full picture I looked at how the code signing options work in this context as well.

Code Signing Options

When signing my commits to authenticate authorship, it’s possible to use PGP/GPG (the “classic way”), or now also SSH keys (as detailed, for example here or here).

The GPG setup is well established, and also links my commits to my identities used elsewhere (e.g. signed emails sent to mailing lists that care about it, with the key linked from this site’s frontpage). This of course is not always needed or desired, but it decouples the identity from the code hosting platform. There’s some serious downsides as well, though: GPG signing keys are not supposed to be numerous (just a single one), and thus if I use multiple machines to work on, I will have to take my private keys with me between machines, for example making copies of them. Or if not making copies, then have them on hardware keys (that have other problems with backups and all that, if I got it right the last time I tried to understand the process).

The SSH key commit signing is much newer (need git version at least 2.34), but it’s also simpler: add a key to my git hosting service, sign commits with that key, and thus the service can match things up and show that match. I can add as many keys as machines I’m working on if needed, no need to transfer or copy keys between machines, and I can also choose use some keys for login only or code signing only.

A third party trying to verify these signatures, though, would need to get the keys from the hosting service (I’d find it surprising if people would distribute their commit signing keys out of band the same way as they do with GPG public keys, since there are likely more of them). Hence it git hosting services will need to make the user’s keys available (as they do at the relevant username.keys URLs, e.g. mine on GitHub and GitLab).

Also can’t forget to add the relevant keys to the list of allowed signing keys locally, and all the other relevant setup (see e.g. the GitHub and GitLab docs). There are a bit too many places to update, but it’s mostly set-it-and-forget-it. After that, once started to sign commits, adding the --show-signatures flag to the commands that support it (git log, git show for example), should show the signatures.

My Winning Combo

Looking at the opions above, there’s a matrix of options that we can use, and here’s what I think about them:

	GPG signature	SSH key signature
Git Credential Helper	Extra setup	Simpler
SSH clone	The usual	Most convenience

Really, where I want to be is just SSH keys for everything, even if they are imperfect, but they have the most number of puzzle pieces to fit.

SSH Key Security

While previously SSH keys were really just held as files in your ~/.ssh folder, most likely, recently I’ve found (tada!) that the password managers I use can also store & serve SSH keys: see in particular 1Password’s SSH documentation and KeePassXC docs (scroll to SSH Agent Integration on that page), though I’m sure other password managers can do this too.

1Password

The two password managers listed above handle things quite similarly. 1Password is a bit less hands-on, though, the default settings work pretty well.

One important bit is that 1Password runs its own SSH agent, so that has to be configured in the relevant places, but it’s easy enough. The approvals are also useful, so it’s more transparent when something accesses the key.

With this things generally work, and relatively easy to reason about things. When things are less clear, it might be just a debug check-away away from seeing the keys added to this alternate agent:

$ export SSH_AUTH_SOCK=~/Library/Group\ Containers/2BUA8C4S2C.com.1password/t/agent.sock
$ ssh-add -l
256 SHA256:XfRsbxRMm+CN[...snip...]

KeePassXC

KeePassXC, being open source, is my preferred solution of the two, though unsurprisingly it’s the more awkward one to set up. The main differences from 1Password include:

• needing to generate the keys externally to the password manager (rather than having built-in ssh keygen) – this is a con on usability but a strong pro on basing security on the established tool, rather than potentially questionably reimplement it
• uses the main SSH agent, so no extra setup is necessary in most of the tools – this is a potential pro on usability for configurations, but a potential con that the worflow and config of loading keys into the agent needs a bit more understanding to be both ergonomic and safe to one’s level of paranoia
• the key use confirmation defaults to “ok” on pressing Enter on the pop-up (rather than Cancel) – this is a pro on usability, but con on “failing open” rather than closed

It’s still a pretty simple workflow, and it’s quite interesting to see how many things KeePassXC learned to do as well.

Experience

Thinking about the various threat models to my SSH crendentials, this setup adds one more layer to the defence in depth, and it does feel more relaxed already (relaxed from a point of stress I didn’t quite know I had before).

Picking the SSH key based login and signing feels like using the most appropriate tech for the moment, and there are still knobs for people to adapt it to their security levels (different SSH keys for login and signing, passwords on the keys themselves, etc…)

This setup works very well when I want to be notified whenever a tool’s using the SSH key so it would be more obvious if a stray process is trying, say exiltrate the keys. On the other hand this breaks down when git itself is running background processes, such as git-maintenance, so that’s not something that I could use here. So far out of (literally) thousands of codebases & repos I’ve used that maintenance setup exactly once, for convenience. For me it is not a major loss, then.

The one bit that feels a step backwards is that having the SSH keys in the password manager and carrying it around counteracts the “separate key for each system” arrangement. This might just be part of getting used to new processes, and not an actual downside.

Further Thoughts

In cybersecurity yesterday’s best practices might be inadecvate today and “last week’s” practices might be outright dangerous… Gonna keep revisiting this setup more broadly and in terms of details, as I learn more.

It’s a good question why even do code signing (besides having a “verified” check mark, which alone doesn’t mean much if not part of a verification process), though this needs some more space to unpack. For the time being I’ll assume that signing is better than not signing, if nothing else than as a forward looking prep for better audit processes down the line.

There’s really a question around having too many things in a single password manager: nowadays it can be the complete “royal flush” of password, TOTP, SSH key, recovery codes, passkey… and likely more bits that I might not be using yet? This does make me uneasy, and likely a scale on which usability and security will adjust over time (such as. bundling and unbundling various cybersecurity aspects).

I might also actually misunderstand various things above, if so, I’d be very keen to hear, just drop me a line!

Edit (2024-09-15): Looks like MacOS 14.6 Sonoma changes have broken the KeePassXC behaviour when confirmation is requested before using the key. I was looking at the workaround posted (some manual reconfiguring of ssh-askpass, but it seems pretty fragile… So I’d probably keep to 1Password on MacOS (sigh), while on Linux Bob continues to be your uncle…

The post Git login and commit signing with security appeared first on ClickedyClick.

When I say I have a home server, that might be too generous description of the status quo: I have a pretty banged up Raspberry Pi 3B. It’s running ArchLinux ARM, the 64-bit, AAarch64 version, looking a bit more retro on the hardware front while pushing for more modernity on the software side – a mix that I find fun.

There are a handful of services running on the device — not that many, mostly limited by it’s *gulp* 1GB memory; plenty of things I’d love to run, doesn’t well co-locate in such a tiny compartment. Besides the memory, it’s also limited by storage: the Raspberry Pi runs off an SD card, and those are both fragile, and limited in size. If one wants to run a home file server, say using a handful of other SD cards lying around, to expand the available storage, that will be awkward very soon. To make that task less awkward (or replace one kind of awkward with a more interesting one), I’ve set out to set up a ZFS storage pool, using OpenZFS.

The idea

Why ZFS? In big part, to be able to credibly answer that question.

But with a single, more concrete reason: being able to build a more solid and expandable storage unit. ZFS cancombine different storage units

• in a way that combats data errors, e.g. mirroring: this addresses SD cards fragility
• in a way that data can expand across all of them in a single file system: this addresses the SD cards size limitations

This sounds great in theory and after a bit of trial-and-error, I’ve made the following setup, relying on dynamic kernel modules for support for flexibility, and a hodgepodge of drives at hand for the storage

The file system supports needs is provided by the zfs-dkms package dynamic kernel module (DKMS), which means the kernel module required for being able to manage that file system is recompiled for each new Linux kernel version as it is updated. This is handy in theory, as I can use the main kernel packages provided by the ArchLinux ARM team.

For storage, I’ve started off with two SD cards in mirror mode (going for data integrity first). Later I’ve found — and invested in — some large capacity USB sticks that bumped the storage size quite a bit. With these, the currentl ZFS pool looks like this:

It already saved me — or rather my data — once where an SD card was acting up, though that’s par for the course. One very large benefit is that the main system card is being used less, so hopefully will last longer.

The complications

Of course, it’s never this easy… With non-mainline kernel modules and with DKMS, every update is a bit of a gamble, that can suddenly not pay off. That’s exactly what happened last year, when suddenly the module didn’t compile anymore on a new kernel version, and thus all that storage was sitting dump and inaccessible. After digging into the issue, it down to:

1. the OpenZFS project being under Common Development and Distribution License (CDDL)
2. the Linux kernel deliberately breaking non-GPL licensed code by starting to withold certain floating point capabilities, because “this is not expected to be disruptive to existing users”.

This wasn’t great, as user being between pretty much a rock & a hard place, even if this is a hobby and not strictly speaking a production use case on my side.

Nonetheless, it worked by downgrading to a working version and skipping updates to the kernel packages.

Then based on a suggestion, patching the zfs-dkms package (rewriting the license entry in the META file) to make it look like it’s a GPL-licensed module — which is fair game for one doing on their own machine. This is hacky, or let’s call it pragmatic.

--- META.prev   2024-02-28 08:42:21.526641154 +0800
+++ META        2024-02-28 08:42:36.435569959 +0800
@@ -4,7 +4,7 @@
 Version:       2.2.3
 Release:       1
 Release-Tags:  relext
-License:       CDDL
+License:       GPL
 Author:        OpenZFS
 Linux-Maximum: 6.7
 Linux-Minimum: 3.10

Now, with the current 2.2.3 version, it seems like there’s an official fix-slash-workaround for being able to get the module to compile, even if it’s not a full fix. From the linked merge request message I’m not fully convinced that this is not a fragile status quo, but it’s at least front of mind – good going for wider ARM hardware usage that brings out people’s willingness to fix things!

Future development

Some while back, while working at an IoT software deploument & management company, I had a lot of interesting hardware at hand, naturally, to build things with (or wrestle with…). Nowadays I have things I best describe as spare parts, and thus loads of thingss are more fragile than they need to be, as well as gosh-it-takes-a-long-time to compile things on a Raspberry Pi 3 – making every kernel update some half-an-hour longer!

Likely the best move would be to upgrade to a (much more powerful) Raspberry Pi 5 and use an external NVMe drive, where I’d have much less need for ZFS, at least for the original reasons. It would likely be still useful for other aspects (such as snapshotting, or sending/receiving the drive data, compression, deduplication, etc…), changing the learning path away from multi-device support to the file system features.

If I wanted to use more storage in the existing system, I could also get rid of the mirrored SD cards and just just 4 large USB sticks (maybe in a RAIDZ setup), a poor-man’s NAS, I guess. Though there I’d worry a bit about using the sticks with the same sizes for this to work (unlike pooling, which has no same-size requirements), given the differences in the supposedly same sized products from different companies (likely locking me into a having the same brand and model across the board).

I also feel like I’m not using ZFS to its full potential. If I know enough just to be dangerous… maybe that’s the generalists natural habitat?

The post ZFS on a Raspberry Pi appeared first on ClickedyClick.

As someone who enjoys (or drawn to, or sort-of obscessed with) hacking on hardware, this was a nice call of action: let’s cobble together a custom USB button that can do some kind of call control¹: say muting myself in the call, showing mute status, hanging up, etc.

This kicked off such a deep rabbit hole that I barely made it back up to the top, but one that seeded a crazy amount of future opportunities.

And as a shortcut, there’s a demo below to showcase where I got to.

Finding suitable hardware

This step was harder than I’ve expected, given that I have drawers and drawers of gadgets, but I’m likely a bit out of practice, and also out of date. What I was looking for is

• Being able to show up as a USB device (must)
• Have built in button (optional) or easy connectivity of buttons without breadboard for now
• Have built in LED (optional) or some other way of showing 1 bit of information

This doesn’t sound hard, right?

ReSpeaker

The first option that came up was Seeed Studio’s ReSpeaker Core that I had two of at hand: Arduino Leonardo compatibility, touch sensors for buttons, and an LED ring (the “Pixel Ring”). Turns out that they have been discontinued – which should be fine for now; but also my models are two different pre-release prototypes Seeed gave away for testers. Thus they are not quite like the final version, have different hardware on board here and there, so an experimental experience is expected.

The earlier prototype only has touch sensors on one side, the pixel ring lights up, but I couldn’t control it with Seeed’s ReSpeaker Arduino library. The later prototype has two sides of sensors (effectively two buttons), but the lights don’t seem to work². Regardless this

Aside: alternatives considered

It was illuminating to see how much abandoned, obsolete, discontinued, or not quite useful hardware boards do I have.

One is RFDuino, that I got from Kickstarter, I’m yet to use, and all the project’s websites have already disappeared – fortunately not the code repo. This would have been a more complex solution anyways, but wireless! Use one RFDuino to expose a USB Telephony device, and communicate wirelessly to another that operates the light and button on battery. Pretty cool. Also, it might not have worked if the chip used cannot do the cruicial “expose a USB [device]” part of the plan.

Other option that popped up was an Arduino Nano + my own made GroveHat + a Grove Button. Except, the Nano definitely cannot be a custom USB device, so there goes nothing.

Besides these, I’ve found plenty of:

• single board computers (old or obsolete),
• FPGAs (never used, and would be a whole different project to implement something on them), and
• other microcontrollers that all have interesting specialties, but don’t tick the mandatory boxes…

These boards might not be right for now, but definitely there are projects in store for them (if only thre’s time).

Back to ReSpeaker then…

Plugging in the USB

The next thing is to figure out what’s really happening when an USB device is plugged in and it shows the operating system that it can do certain things. That is, how does Meet know that there’s a compatible device to connect to?

The USB HID docs

This is answered by the USB Human Interface Devices (HID) specs — one that is pretty complicated, has a lot of legacy bits, and need a different kind of mindset. In a nutshell, though, with my current, partial understanding:

On connection the device sends a “report” to the OS that details on what can it do, including:

• what kind (or kinds!) of device it is?
• what functionality of the kind is available in this particular implementation?
• what’s the data layout to pass control information back-and-forth for this implementation?

In our example, a very minimal setup would would be:

• I’m a Telephony Device (Usage page 0x0B)
• I implement a generic “Phone” (Usage ID 0x01)
• I have capability to do a “Phone Mute” (Usage ID 0x2F)
• Here’s the 1 bit of a 1 byte payload that conveys that phone mute status

This course does not take into account other functionality, e.g.

• I can also hang up – Hook Switch, Usage ID 0x20;
• I have status LEDs – that’s a whole fun of redefining functions on the LED Page 0x08;

and so on. But for the time being this should be enough.

Device implementation

Fortunately we can stand on the shoulders of giants, that is the Arduino HID Project which implemented a bunch of different devices. And even though a “phone” like this is not among them, we can make some reasonable guesses how it would work.

Having said that, from a forum post that was also trying to do something similar (but based on the TinyUSB library):

HID report descriptor is very difficult thing to come up by oneself. You should google around, or dump report descriptor from existing device to copy/follow it.

hathach @ TinyUSB discussion 667

Okay, then do not come up with this stuff, instead let’s look for tools. The USB HID homepage links to the Microsoft HID Tools to generate HID reports from a TOML-like language. Except it needs C# and I just wasn’t ready to dive in a side-quest to install & learn a new toolchain.

So being lazy this way, a bit more sleuthing turned up someone’s example HID report for a device very close to what I’m trying to do, hurray!

I took this and started to poke around the HID project to see how other devices are implemented. Troubleshooting by using the ReSpeaker’s touch to adjust screen brightness up / down (as a “Consumer Device”) was also pretty neat! In the end I took the system buttons example and run with that one.

Having said that, the HID report is really just the interface. The devil is in how to implement actually creating the data packages that passes data according to the report definition. And this is the case when I wish I knew more C++ but copy-paste and some guesswork will have to do.

The current result lives in the “phone” branch of my HID Project fork, check for the “Phone” bits in “src/HID-APIs” and “MultiReport” folders, if interested.

Minimal viable mute

The implementation from this point on was pretty straightforward – since we cut back the scope so much…

The code to run on the ReSpeaker then just has to do the following:

• when touching one side, send a report with “Phone Mute” on
• when touching the other, send a report with “Phone Mute” off

And this is sort of simple³ :

For the full use case there would be a lot more complexity for both reading and writing data from the host, controlling multiple peripherals (LEDs and buttons) and the whole logic around it. But for now, it’s good enough for a demo:

The code repository is available on Github at imrehg/arduino-usb-phone-hid.

Notes and Future work

The specs

It’s great that stuff from 20+ years ago still works mostly the same way. The latest 1.4 version of the HID Tables is nicely formatted, has a lot more device typed defined, but has much less support text. Originally I’ve read the 1.12v2 version as that showed up in my search. Back then in 2004 they had an “examples” section (see the Telephone at Appendix 10!) which is useful to grok more of the fundamentals.

The newer version also has some devices types that looked suitable, but weren’t really: Generic Desktop Page (0x01) and 0xE0-E2 Usage IDs for Call Active LED, Call Mute Toggle, and Call Mute LED respectively. These didn’t seem to work with Meet, so it might be interesting to try implementing a device that does both and try other online call software.

I should also have read the spec more before diving into hacking on the HID implementation fork, as there’s a lot more information in the HID Device Class Definition, including how to construct the values for many of the fields (I’m looking at you “INPUT (Cnst,Var,Abs)“). RTFM is and remains a solid advice – and not just when one thinks there’s time.

Also regarding the specs: some of them I only find in the Internet Archive’s Wayback Machine. If you encounter a good source that should be kept, always add it to the Wayback Machine and preserve it for your future selves and others!

This exploration of USB HID pulled on so many threads, and left so much unfinished, that it’s a fertile ground for the future, even more than most previous projects.

More call functionality

The most obvious thing is to implement the whole setup with the buttons. I’ve tried Hook Switch to hang up a call, that works too. Could add status lights, maybe throw in some “Active Call” LEDs, or so on. This requires better understanding how data is sent over the wire for USB and how to handle incoming data. The Arduino examples rarely seem to use the “Output” fields (ie. incoming data, output from the host’s point of view, but maybe TinyUSB does ?

For this, it would be nice to find a different hardware platform that would make this more seamless (so I can concentrate on the software side more). If that platform would lend itself to be reproduced or made stand alone, that would be even nicer: imaging brining my little call control box that can be used with other computers easily as well…

Implement more USB HID devices

The Arduino HID project has a bunch of devices implemented, but there are an infinite numbers that could be added. Unfortunately for Arduino it is harder to add more device types as an add-on to this library versus the current “forked” approach⁴, so new decices should be in the main project, eventually.

So far there’s no Telephony device implemented there and it would be nice to find the right level of abstraction that works. The library doesn’t implement specific HID table pages, but specific usages or a subset of a usage. Thus like always, the hardest part would likely be setting the right interface (the right specs and “API”) for a new device to implement both the HID reports and the functions that manipulate what’s being sent and when.

On the other hand, that does sound like a fun experiment, and I’d look forward to adding 3D Game Controllers (Game Controls Page 0x05), Environmental Sensors (Sensors Page 0x20, Usage ID 0x30-3B), … or even a Submarine Simulation Device (Simulation Device page 0x02, usage id 0x05). These are stuff I go to Hackerspaces for…

WebHID for internet plus USB

While debugging this HID device behaviour, I found also WebHID that brings such devices to the web. This feature seems to be behind Meet’s and other phone systems like 3CX expanding USB support outside of the OS and into the browser. And no, Firefox does not support it, furthermore declined supporting it.

Nonetheless it’s very cool that (if I upskill a bit), I can create a web page that would help me debug such HID development:

• request devices that are filtered in various ways (vendor, product is standard, but usage page and explicit usage is the main key). This is likely what Meet does as well, “just gimme devices with Telephony usage page (or Phone usage? Need to check exactly)
• read the HID report collections sent by the device, so the results can be debugged, and
• read device input events that we can then either log for debugging or in an application react to to it

This opens a lot more mashup opportunities by the dozen.

Finally

Unlike most other projects I had where I’m focused on one specific outcome, this turned out to be more focusing on getting a new toolkit (custom USB devices) up and running, so I can think about a wider types of projects to do. In that sense, this feels a big success, even if I know how little I know about programming outside of my day-to-day environment. But ignorance is not a bliss.

And now, going on mute.

1. Many moons past I used to use a Jabra Evolve 80, that has a USB accessory controlling call features, so I did have first hand example of what sort of experience I’d like. ︎
2. I’ve tried reviewing the hardware schematics, looking into the pixel ring control functions, and given that the LEDs seems standard I’ve also attempted to use the FastLED library to drive them instead, so far nothing. I still bet on hardware differences from final schematic + my inability to debug it, but it can be faulty hardware just as well. Needs more effort – in the future. ︎
3. The Arduino code became more “simple” once I realised that things set up this way do not need debouncing for the touch sensors. In other cases that would be essential, there’s sooo much flaky signal to use those terminals as momentary switches or similar. ︎
4. At least I don’t know how nicely extend a library for C++, if that’s even possible. Keen to learn, though. ︎

The post Making a USB Mute Button for Online Meetings appeared first on ClickedyClick.