Categories
Admin Computers

ZFS on a Raspberry Pi

I have a little home server, just like mike many other geeks / nerds / programmers / technical people… It can be both useful, a learning experience, as well as a real chore; most of the time the balance is shifting between these two ends. Today I’m taking notes here on one aspect of that home server that is widely swing between those two use cases.

When I say I have a home server, that might be too generous description of the status quo: I have a pretty banged up Raspberry Pi 3B. It’s running ArchLinux ARM, the 64-bit, AAarch64 version, looking a bit more retro on the hardware front while pushing for more modernity on the software side – a mix that I find fun.

There are a handful of services running on the device — not that many, mostly limited by it’s *gulp* 1GB memory; plenty of things I’d love to run, doesn’t well co-locate in such a tiny compartment. Besides the memory, it’s also limited by storage: the Raspberry Pi runs off an SD card, and those are both fragile, and limited in size. If one wants to run a home file server, say using a handful of other SD cards lying around, to expand the available storage, that will be awkward very soon. To make that task less awkward (or replace one kind of awkward with a more interesting one), I’ve set out to set up a ZFS storage pool, using OpenZFS.

The idea

Why ZFS? In big part, to be able to credibly answer that question.

But with a single, more concrete reason: being able to build a more solid and expandable storage unit. ZFS cancombine different storage units

  • in a way that combats data errors, e.g. mirroring: this addresses SD cards fragility
  • in a way that data can expand across all of them in a single file system: this addresses the SD cards size limitations

This sounds great in theory and after a bit of trial-and-error, I’ve made the following setup, relying on dynamic kernel modules for support for flexibility, and a hodgepodge of drives at hand for the storage

The file system supports needs is provided by the zfs-dkms package dynamic kernel module (DKMS), which means the kernel module required for being able to manage that file system is recompiled for each new Linux kernel version as it is updated. This is handy in theory, as I can use the main kernel packages provided by the ArchLinux ARM team.

For storage, I’ve started off with two SD cards in mirror mode (going for data integrity first). Later I’ve found — and invested in — some large capacity USB sticks that bumped the storage size quite a bit. With these, the currentl ZFS pool looks like this:

Terminal screenshot of the 'zpool status' command.

It already saved me — or rather my data — once where an SD card was acting up, though that’s par for the course. One very large benefit is that the main system card is being used less, so hopefully will last longer.

The complications

Of course, it’s never this easy… With non-mainline kernel modules and with DKMS, every update is a bit of a gamble, that can suddenly not pay off. That’s exactly what happened last year, when suddenly the module didn’t compile anymore on a new kernel version, and thus all that storage was sitting dump and inaccessible. After digging into the issue, it down to:

  1. the OpenZFS project being under Common Development and Distribution License (CDDL)
  2. the Linux kernel deliberately breaking non-GPL licensed code by starting to withold certain floating point capabilities, because “this is not expected to be disruptive to existing users”.

This wasn’t great, as user being between pretty much a rock & a hard place, even if this is a hobby and not strictly speaking a production use case on my side.

Nonetheless, it worked by downgrading to a working version and skipping updates to the kernel packages.

Then based on a suggestion, patching the zfs-dkms package (rewriting the license entry in the META file) to make it look like it’s a GPL-licensed module — which is fair game for one doing on their own machine. This is hacky, or let’s call it pragmatic.

--- META.prev   2024-02-28 08:42:21.526641154 +0800
+++ META        2024-02-28 08:42:36.435569959 +0800
@@ -4,7 +4,7 @@
 Version:       2.2.3
 Release:       1
 Release-Tags:  relext
-License:       CDDL
+License:       GPL
 Author:        OpenZFS
 Linux-Maximum: 6.7
 Linux-Minimum: 3.10

Now, with the current 2.2.3 version, it seems like there’s an official fix-slash-workaround for being able to get the module to compile, even if it’s not a full fix. From the linked merge request message I’m not fully convinced that this is not a fragile status quo, but it’s at least front of mind – good going for wider ARM hardware usage that brings out people’s willingness to fix things!

Future development

Some while back, while working at an IoT software deploument & management company, I had a lot of interesting hardware at hand, naturally, to build things with (or wrestle with…). Nowadays I have things I best describe as spare parts, and thus loads of thingss are more fragile than they need to be, as well as gosh-it-takes-a-long-time to compile things on a Raspberry Pi 3 – making every kernel update some half-an-hour longer!

Likely the best move would be to upgrade to a (much more powerful) Raspberry Pi 5 and use an external NVMe drive, where I’d have much less need for ZFS, at least for the original reasons. It would likely be still useful for other aspects (such as snapshotting, or sending/receiving the drive data, compression, deduplication, etc…), changing the learning path away from multi-device support to the file system features.

If I wanted to use more storage in the existing system, I could also get rid of the mirrored SD cards and just just 4 large USB sticks (maybe in a RAIDZ setup), a poor-man’s NAS, I guess. Though there I’d worry a bit about using the sticks with the same sizes for this to work (unlike pooling, which has no same-size requirements), given the differences in the supposedly same sized products from different companies (likely locking me into a having the same brand and model across the board).

I also feel like I’m not using ZFS to its full potential. If I know enough just to be dangerous… maybe that’s the generalists natural habitat?

Categories
Maker Programming

Making a USB Mute Button for Online Meetings

I use Google Meet every day for (potentially hours of) online meetings at work, so it’s very easy to notice when things change and for example new features are available. Recently I’ve found a new “Call Control” section in the settings that promised a lot of fun, connecting USB devices to control my calls.

Screenshot of the Google Meet Settings menu during calls, showing the call control menu and a call-out to connect my USB device.
Google Meet Settings menu during a call, witht the Call control section

As someone who enjoys (or drawn to, or sort-of obscessed with) hacking on hardware, this was a nice call of action: let’s cobble together a custom USB button that can do some kind of call control1: say muting myself in the call, showing mute status, hanging up, etc.

This kicked off such a deep rabbit hole that I barely made it back up to the top, but one that seeded a crazy amount of future opportunities.

Categories
Programming

Doing the Easy Problems on Leetcode

Over the last decade I seem to have been working in environments, where many engineers and engineering minded people spend time with programming puzzles and coding challenges. Let it be Advent of Code, Project Euler, Exercism, TopCoder, or Leetcode. I’ve tried all of these before (and probably a few more that I no longer remember), though with various amount of time spent all fired up, and then fizzled out. Recently I’ve picked up Leetcode, since from the above list that’s why I’ve spent the least amount of time with and others mentioned using it a way to relax and learn on weekends (suspend judgement on the wisdom of that for now).

Thus in the last two weeks I was solving problems, though not just any problems, but went in mostly for the Easy ones. These few dozen problems and short amount of time doesn’t give me a deep impression, but from past experiences I can still distill some lessons that help shaping future experiments.

The purpose of using the Easy problems is different from e.g. going all in for puzzle-solving fun, which is likely in the Hard ones. Rather than that, I think easy problems can be used for learning some new techniques, looking for common patterns, and becoming more polygot.

Categories
Admin

Mixing GitLab personal and work accounts: Enterprise Users

TL;DR: if you are about to become a GitLab enterprise user, time to split your work from passion.

I’m often asked by other team members just starting on their version control journey, when using the likes of GitHub and GitLab, whether to have separate accounts for work and personal projects, or have a single one for both?

So far my advice has been pretty much along the lines of: “use a single one“, for many reasons, like every service seems to handle email aliases, git+ssh is pain enough with a single account not even multiple, and people generally seem to build their professional and open source contributions under a single persona anyways.

This advice no longer stands, at least for GitLab. I received this email recently, and how their use of Enterprise Users (and SSO Login + domain verification) makes it absolutely necessary to separate work and personal accounts:

I’ve been looking for a blogpost or other announcement, but couldn’t find one, hence the reposting of it here. I definitely gonna scramble a bit to create some new accounts (and keep my preferred username for the personal one).

Categories
Admin

When WordPress caching is not what it seems

When parts of a system are strongly interconnected, one can discover latent issues while debugging something completely different. This is what happened with this blog’s caching and integrating with the Fediverse.

Fediverse adventures

I was part of The Great Twitter Exodus of 2022, and like many I’ve landed on Mastodon (hey, hello, https://fosstodon.org/@imrehg). Mastodon and the whole Fediverse and its build around the ActivityPub protocol is technically very interesting and brings back a bit of retro-joy to me (which needs some reflections on why and how is retro joyful, but another time). This current blog is running WordPress, and soon found that there’s a plugin to turn a WordPress blog into a my own ActivityPub node. That seemed some excellent way to connect up tools and make a more inter-connected Internet (besides nerding out, if I’m fully honest).