The story starts with me rediscovering a DVB-T receiver USB stick, that I had for probably close to decade. It’s been “barnacled” by time spent in the Taiwanese climate, so I wasn’t sure if it still works, but it’s such a versatile tool, that it was worth trying to revive it.

When these receivers function, they can receive digital TV (that’s the DVB-T), but also FM radio, DAB, and also they can act as software defined radio (SDR). This last thing makes them able to receive all kinds of transitions that are immediately quite high on the fun level, in particular airplane (ADS-B transmission) and ship (AIS) tracking. Naturally, there are websites to do both if you just want to see it (for example Flightradar24 and MarineTraffic, respectively, are popular aggregators for that data but there are tons), but doing your own data collection opens doors to all kinds of other use cases.

So on I go, trying to find, what software tools people use these days to use these receivers. Mine is a pretty simple one (find out everything about it by following the “RTL-SDR” keywords wherever you like to do that :) and so I remembered there were many tools. However also time passed, I forgot most that I knew, and also there were new projects coming and going.

ADSBox

While I was searching, I found the adsbox project, that was interesting both kinda working straight out of box for me, while it was also last updated some 9 years ago, so it’s an old code base that tickles my “let’s maintain all the things!” drive…

The tool is written mostly in C, while it also hosts its own server for a web interface, for listing flights, and (back in the day) supporting things like Google Maps and Google Earth.

Both the Google Maps and Earth parts seem completely: Maps has changed a lot since, as I also had to update my Taiwan WWII Map Overlays project over time too (the requirement of using API keys to even load the map, changes to the JavaScript API…). Earth I haven’t tried, but I’m thinking that went the way of the dodo on the the desktop?

So in the supur of the this-is-the-weekend-and-I-have-energy-to-code moment, I started to think of the options:

• could fix up the map, either with the Google Maps changes, or bring in some other map?
• the project has barely any readme, and I mainly managed to make it work by looking at old articles from the time when adsbox waas new, could fix those up?
• during the compilation, loads of warnings happened, that seem to call for some “better quality” coding, let’s fix stuff until -Werror (making all warnings errors) passes too! This would be a learning experience
• I’m sure I can find other tasks to do as well, like an error message here, a strange behaviour there…

Here’s the kicker though: I don’t really know C. I spend most of my time in Python-land, and haven’t done a C project in anger yet. Is it worth trying to dig in, while there are other ADS-B projects that a) work better, b) are in languages that I’m more looking to learn, such as Rust?

There was an additional drive of curiosity, just like in my last post: can I use Large Language Models (LLMs) to complement me on things I lack, such as knowledge of the exact programming language at hand?

With this I thought let’s dig in, and let’s dig into the C code: that seemed immediately tractable, more limited in scope, and thus would help build up (hopefully) some successes and I’ll learn my way around the codebase better.

On the LLM side I have GitHub Copilot – though it seems somewhat crippled in my open source Code Server installation of VS Code, rather than the official VS Code, in particular the context menus and Copilot Chat seems missing, and thus it was only communicating with me through TAB-completions and me adding comments to guide or suggest. That’s not very practical, so didn’t push it too far for the relevant tasks of explanation and exploration of options that I wanted to do.

I also have Claude that I can chat with. If I wasn’t working on my 13 year old Lenovo ThinkPad X201, I’d probably set up Ollama, but that’s just excruciating with even the smallest models on this machine (until I upgrade something newer, or run the questions on my work M1 MacBook). So Claude it is for now.

Hello Fixes

I guess it’s one sign of hubris (or unlimited optimism), to jump into fixing compilation warnings, without knowing anything much of the codebase yet. This started in areas where the airplane tracking interacts with SQLite, for example had warnings about casting pointers to integers of different size while shuffling around SQLite query results:

int * t = (int *) sqlite3_value_int(argv[0]);

This was also part of a larger code section (formatting integers into hexadecimal or octal strings, for example for the ICAO codes…), and thus had to play around how much context to give to Claude to actually have something useful.

A bit of mocking around there seemed to have worked, and while I should have asked more software architecture & best practices questions, probably knew about it enough to be dangerous, and left it as it was so far.

Having said that, after this change it turned out that some part of the interface now displaying stuff differently: the 24-bit ICAO airplane registration codes had useless leading zeros for 8 hex digits, rather than the expected 6 digits – since the fix was done without this context. Here we go, manual adaptation on this regression.

Now there were cases when “sprintf may write a terminating nul past the end of the destination“, as the code seems to have written its data back into the same place as this:

sprintf(data->avr_data, "*%s", data->avr_data + 13);

This ended up being again about a much bigger context (interative reading, processing, and passing on of recorded ADS-B packets), where based on the Claude’s suggestions I couldn’t really get to anything useful. The real point was always one step further:

• instead of the line look at the nearby lines
• instead of the near by lines, look at the whole funciton
• instead of the whole function, look at the wider codebase with its configuration

These are of course no-brainers. However Claude with its chat interface cannot really do that, while Copilot without its chat interface also cannot do this digging. Catch-22? Since in the end I admitted myself (for the nth time) that I need to understand the purpose of the code better before “fixing” it. Then due to the lack of comments in the codebase + lack of natural intuition of the built in C functions’ behaviour, I’ve just left them as they were for now, since they do work.

From here I turned to other parts. The webserver was not serving some files with the correct MIME type, due to its hand-rolled file extension extraction (splitting filenames at the first . rather than the last), this was easy to fix – with a bit of StackOverflow this time, rather than asking Claude.

Then there was an issue with the tool apparently not playing back the recorded packet data, which I fixed with a combo of regular ol’ debug printouts, StackOverflow, and just thinking about how it could work (it’s the issue of explicitly filling in daylight saving data in the relevant tm struct – tm_isdst – and thus IMHO it’s doing a regular “undefined” behaviour: in this case jumped the first timestamp’s time ahead by an hour, and thus would have needed to wait an hour to pass as the playback (following the real timingof the packets) catch up and start actually replaying. Still weird, why only the first packet’s data was shifted, and could I do a more solid fix than setting it once as the code never seem to overwrite it? These are the questions that are more addressing C-knowledge or potential best practice of the code’s structure overall…

Finally I’ve started on replacing Google Maps with OpenFreeMap and got as far as displaying the map (which is the easy step:). The whole replacement would likely be a lot more, also given the amount of barely documented JavaScript code in the project – but hopefully I have more working knowledge of JS than C.

Lessons Learned

First lesson is that I likely have a “saviour complex”, trying to fix up every code I see being imperferct in some way, whether or not I am capable of doing it or not. This is something to meditate further on for sure.

When using LLMs for code work, they are just as useful as another mid-level coder without much context – almost not at all. The context of code is always relevant, so either the LLM would have to get it itself, or the person pairing with the LLM would have to provide it. Thus the work is always there, just not always possible.

It’s very nice that I can do things in programming languages that I don’t really understand, but that’s only the case if I either spend much-much time actually getting to know things so I can start to judge whether the changes even have a chance to be correct or not; or I don’t care whether they are correct or not (but is this really an option?)

Overall the LLMs need the same things as humans to do a good job, and cannot pretend that they really can do work without these (even if they might appear being able to do without these for some time):

• good comment in the code so the intention can be ascertained as well
• tests that show what the correct behaviour should be, and catch regressions or unintentional breakages
• have domain knowledge to form better mental models about what should happen

The first two wasn’t true in this project. The last point is likely where LLMs are ahead in cases like this (having been trained on “all the Internet’s data”), though wouldn’t be the same for some niche, or work internal projects.

The LLMs suggestions are still ver much localised, thus they cannot really fix up the structure of the code too much – or maybe I’m not using the right tools, of course. And this is where my future big ask would lie: don’t just tell me how to fix this line, rather tell me that the entire block is no longer needed / could be merged with another part of the code / could be broken out to its own module that would help over there… Of course, this is moving the goal post a bit of what LLM programmers’ look like, though I also think that the current “fix this line” is something I most definitely want to have enough practice with that I don’t really need to ask (though it could suggest if there are good practices I haven’t picked up yet).

Where do I go from here?

This adsbox project is mostly obsolete, as I’ve found a bunch of other tools that are better, and better supported now (adsb_deku, tar1090), but surprising it still have stuff that are better here and in other tools (the plane’s status icons, some data displayed here that is not in others, showing what sort of packets (what Downlink Format or DF numbers) were received for the aircraft, etc… So there might be still value in using it occasionally, so there might be value.

Even if I could get a kick out of it, it’s likely useful to keep things time-boxed or constrained to some topics: change the map; add comments as I find them; fix issues if they arise; package it up for ArchLinux. That’s about it, but these should be generally useful (e.g. using OpenFreeMap for other projets in the future or rewriting the aforementioned Taiwan WWII Map project to use that).

My current fixes live in my fork in GitHub imrehg/adsbox, with no guarantees. Since the project also doesn’t have a license (just a note of “free for non-commercial use”, which doesn’t cover modifications), I’m probably keeping it simple for now.

I also got the hang of software defined radio again, and there’s just so much fun to have…

What’s the most useful is seeing in practice, what does software need to be maintainable almost a decade later, and what’s missing in most projects: explanatory comments to understand what is being done and why, and tests to know whether things are running correctly or not. And maube then both my future self, my colleagues, and any potential AI pair programmer would have a better chance of succeeding at “maintain all the things!”

The post Refreshing Airplane Tracking Software With and Without AI appeared first on ClickedyClick.

It looks interesting and informative. And even though I don’t have codebases that have been around this long, there are plenty of codebases around me that are fast moving, so something like a month (or in some cases week) level cohorts could be interesting.

One way to take this challenge on is to actually sit down and write the code. Another is to take a Large Language Model, say Claude and try to get that to make it. Of course the challenge is different in nature. For this case, let’s put myself in the shoes of someone who says

I am more interested in the results than the process, and want to get to the results quicker.

See how far we can get with this attitude, and where does it break down (probably no spoiler: it breaks down very quickly.).

Note on the selection of the model: I’ve chosen Claude just because generally I have good experience with it these days, and it can share generated artefacts (like the relevant Python code) which is nice. And it’s a short afternoon. :) Otherwise anything else could work as well, though surely with varying results.

Version 1

Let’s kick it off with a quick prompt.

Prompt: How would you generate a chart from a git repository to show the age of the code? That is when the code was written and how much of it survives over time?

Claude quickly picked it up and made me a Python script, which is nice (that being my day-to-day programming language). I guess that’s generally a good assumption these days if one does data analytics anyways (asking for another language is left for another experiment).

The result is this this code. I’ve skimmed it that it doesn’t just delete all my repo or does something completely batshit, but otherwise saved in a repo that I have at hand. To make it easier on myself, added some inline metadata with the dependencies:

# /// script
# dependencies = [
#   "pandas",
#   "matplotlib",
# ]
# ///

and from there I can just run the script with uv.

First it checked too few files (my repository is a mixture of Python and SQL scripts managed by dbt), so had to go in and change those filters, expanding them.

Then the thought struck me to remove the filter altogether (since it already checks only files that are checked in git, so it should be fine – but then it broke on a step where it reads a file as if it was text to find the line counts. I guess there could be a better way of filtering (say “do not read binary files”, if there’s a way to do that), but just went with catching the problems:

# ....
    for file_path in tracked_files:
        try:
            timestamps = get_file_blame_data(file_path)
            for timestamp in timestamps:
                blame_data[timestamp] += 1
                total_lines += 1
        except UnicodeDecodeError:
            print(f"Error reading file: {file_path}")
            continue
#....

(hance I know that a favicon PNG was causting those UnicodeDecodeError hubbub in earlier runs. Now we are getting somewhere, and we have a graph like this:

This is already quite fun to see. There are the sudden accelerations of development, there are the plateaus of me working on other projects, and generally feel like “wow, productive!” (with no facts backing that feeling ). Also pretty good ROI on maybe 15 mins of effort.

Having said that, this is still fair from what I wanted.

Version 2

Promt: Could we change the code to have cohorts of time, that is configurable, say monthly, or yearly cohoorts, and colour the chart to see how long each cohort survives?

This came back with another set of code. Adding the metadata, skimming it (it has the filter on the file extensions again, never mind), and running it once more to see the output, I get this:

Because of the file extension filter in place, the numbers are obviously not aligning with the above, but it does something. The something is a bit unclear, bit it feels like progress, so let’s give it a benefit of the doubt, and just change once more.

Version 3

Promt: Now change this into a cummulative graph, please.

One more time Claude came back with this code. Adding the metadata again, same drill. Running this has failed with errors in numpy, though:

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Now this needed some debugging. It turns out a column the code is trying to plot is actually numbers as strings rather than numbers as, you know, say floats…

# my "fix"
        df['cumulative_percentage'] = df['cumulative_percentage'].astype(float)
# end

        # Plot cumulative area
        plt.fill_between(df.index, df['cumulative_percentage'],
                        alpha=0.6, color='royalblue',
                        label='Cumulative Code')

It didn’t take too many tries, but it was confusing at first – why shouldn’t be, if I didn’t actually read just skim the code…

The result is then like this:

Sort of meh, it feels like it’s not going to the right direction overall.

But while debugging the above issues, I first tried tried to ask Claude about the error (maybe it can fix it itself), but came back with “Your message exceeds the length limit. …” (for free users, that is). So I kinda stopped here for the time being.

Lessons learned

The first lesson is very much re-learned:

Garbage in, garbage out.

If I cannot express what I really want, it’s very difficult to make it happen. And my prompts were by no means expressing my wishes correctly, no wonder Claude wasn’t really hitting the mark. Whether or not a human engineer would have faired better, I don’t know. I know however, that this kind of “tell me exceedingly clearly what’s your idea” is an everyday conversation for me as an engineer (and being on both end of the convo).

The code provided by the model wasn’t really far off for some solution, so that was fun! On the other hand, when it hit any issues, I really had to have domain and language knowledge to fix things. This seems like an interesting place to be:

• the results are quick and on the surface good-enough for a non/less technical person, probably
• but they would also be the ones who couldn’t do anything if something goes wrong.

Even myself I feel that it would be hard to support the code as a software engineer if it was just generated like this. But that’s also a strange thought: so many times I have to support (debug, extend, explain, refactor) code that I haven’t had anything to do with before.

It seems to me that now that since Claude comes across as an eager junior engineer, writing decent code that always needs some adjustments, the trade-off is really in the dimension of spending time to get better at prompting vs better at coding.

If there’s a person with some amount of programming skills, mostly interested in the results not the process, and doubling down on prompting: they likely could get loads further than I did here. Good quality prompts and small amount of code adjustments being the sweet spot for them.

For others who have more programming expertise, and maybe more interested in the process, spending time on getting better at programming rather than getting really better at prompting: keeping to smaller snippets might be the sweet spot, or learning new languages, … Something as a starting point for digging in, a seed, is what this process can help with.

Future

Given the above notes on how this generated code is like a new codebase that I suddenly neet to support, here’s a different, fun exercise to actually improve engineering skills:

Take AI generated code that is “good enough” for a small problem and refactor, extent, productionise it.

I’m not sure if this would work, or would get me into wrong habits, but if I wanted do have some quick ways of doing deliberate practice – and not Exercism, LeetCode, or somilar, rather something that can be custom made, then this seems a way to get started.

Also, now that I’ve gotten even more interested in the problem, I’ll likely just dig into how to actually define that chart I was looking for and what kind of data I would need to get from git to make it happen. The example code made me pretty confident, that “all I need is Python” really, even though while prepping for this I found other useful tools like one allowing you to write SQL queries for your repo, that might be some further way to expand my understanding.

Either way, it’s just fun to mess with code on a lazy Saturday.

The post Adventures into Code Age with an LLM appeared first on ClickedyClick.

As someone who enjoys (or drawn to, or sort-of obscessed with) hacking on hardware, this was a nice call of action: let’s cobble together a custom USB button that can do some kind of call control¹: say muting myself in the call, showing mute status, hanging up, etc.

This kicked off such a deep rabbit hole that I barely made it back up to the top, but one that seeded a crazy amount of future opportunities.

And as a shortcut, there’s a demo below to showcase where I got to.

Finding suitable hardware

This step was harder than I’ve expected, given that I have drawers and drawers of gadgets, but I’m likely a bit out of practice, and also out of date. What I was looking for is

• Being able to show up as a USB device (must)
• Have built in button (optional) or easy connectivity of buttons without breadboard for now
• Have built in LED (optional) or some other way of showing 1 bit of information

This doesn’t sound hard, right?

ReSpeaker

The first option that came up was Seeed Studio’s ReSpeaker Core that I had two of at hand: Arduino Leonardo compatibility, touch sensors for buttons, and an LED ring (the “Pixel Ring”). Turns out that they have been discontinued – which should be fine for now; but also my models are two different pre-release prototypes Seeed gave away for testers. Thus they are not quite like the final version, have different hardware on board here and there, so an experimental experience is expected.

The earlier prototype only has touch sensors on one side, the pixel ring lights up, but I couldn’t control it with Seeed’s ReSpeaker Arduino library. The later prototype has two sides of sensors (effectively two buttons), but the lights don’t seem to work². Regardless this

Aside: alternatives considered

It was illuminating to see how much abandoned, obsolete, discontinued, or not quite useful hardware boards do I have.

One is RFDuino, that I got from Kickstarter, I’m yet to use, and all the project’s websites have already disappeared – fortunately not the code repo. This would have been a more complex solution anyways, but wireless! Use one RFDuino to expose a USB Telephony device, and communicate wirelessly to another that operates the light and button on battery. Pretty cool. Also, it might not have worked if the chip used cannot do the cruicial “expose a USB [device]” part of the plan.

Other option that popped up was an Arduino Nano + my own made GroveHat + a Grove Button. Except, the Nano definitely cannot be a custom USB device, so there goes nothing.

Besides these, I’ve found plenty of:

• single board computers (old or obsolete),
• FPGAs (never used, and would be a whole different project to implement something on them), and
• other microcontrollers that all have interesting specialties, but don’t tick the mandatory boxes…

These boards might not be right for now, but definitely there are projects in store for them (if only thre’s time).

Back to ReSpeaker then…

Plugging in the USB

The next thing is to figure out what’s really happening when an USB device is plugged in and it shows the operating system that it can do certain things. That is, how does Meet know that there’s a compatible device to connect to?

The USB HID docs

This is answered by the USB Human Interface Devices (HID) specs — one that is pretty complicated, has a lot of legacy bits, and need a different kind of mindset. In a nutshell, though, with my current, partial understanding:

On connection the device sends a “report” to the OS that details on what can it do, including:

• what kind (or kinds!) of device it is?
• what functionality of the kind is available in this particular implementation?
• what’s the data layout to pass control information back-and-forth for this implementation?

In our example, a very minimal setup would would be:

• I’m a Telephony Device (Usage page 0x0B)
• I implement a generic “Phone” (Usage ID 0x01)
• I have capability to do a “Phone Mute” (Usage ID 0x2F)
• Here’s the 1 bit of a 1 byte payload that conveys that phone mute status

This course does not take into account other functionality, e.g.

• I can also hang up – Hook Switch, Usage ID 0x20;
• I have status LEDs – that’s a whole fun of redefining functions on the LED Page 0x08;

and so on. But for the time being this should be enough.

Device implementation

Fortunately we can stand on the shoulders of giants, that is the Arduino HID Project which implemented a bunch of different devices. And even though a “phone” like this is not among them, we can make some reasonable guesses how it would work.

Having said that, from a forum post that was also trying to do something similar (but based on the TinyUSB library):

HID report descriptor is very difficult thing to come up by oneself. You should google around, or dump report descriptor from existing device to copy/follow it.

hathach @ TinyUSB discussion 667

Okay, then do not come up with this stuff, instead let’s look for tools. The USB HID homepage links to the Microsoft HID Tools to generate HID reports from a TOML-like language. Except it needs C# and I just wasn’t ready to dive in a side-quest to install & learn a new toolchain.

So being lazy this way, a bit more sleuthing turned up someone’s example HID report for a device very close to what I’m trying to do, hurray!

I took this and started to poke around the HID project to see how other devices are implemented. Troubleshooting by using the ReSpeaker’s touch to adjust screen brightness up / down (as a “Consumer Device”) was also pretty neat! In the end I took the system buttons example and run with that one.

Having said that, the HID report is really just the interface. The devil is in how to implement actually creating the data packages that passes data according to the report definition. And this is the case when I wish I knew more C++ but copy-paste and some guesswork will have to do.

The current result lives in the “phone” branch of my HID Project fork, check for the “Phone” bits in “src/HID-APIs” and “MultiReport” folders, if interested.

Minimal viable mute

The implementation from this point on was pretty straightforward – since we cut back the scope so much…

The code to run on the ReSpeaker then just has to do the following:

• when touching one side, send a report with “Phone Mute” on
• when touching the other, send a report with “Phone Mute” off

And this is sort of simple³ :

For the full use case there would be a lot more complexity for both reading and writing data from the host, controlling multiple peripherals (LEDs and buttons) and the whole logic around it. But for now, it’s good enough for a demo:

The code repository is available on Github at imrehg/arduino-usb-phone-hid.

Notes and Future work

The specs

It’s great that stuff from 20+ years ago still works mostly the same way. The latest 1.4 version of the HID Tables is nicely formatted, has a lot more device typed defined, but has much less support text. Originally I’ve read the 1.12v2 version as that showed up in my search. Back then in 2004 they had an “examples” section (see the Telephone at Appendix 10!) which is useful to grok more of the fundamentals.

The newer version also has some devices types that looked suitable, but weren’t really: Generic Desktop Page (0x01) and 0xE0-E2 Usage IDs for Call Active LED, Call Mute Toggle, and Call Mute LED respectively. These didn’t seem to work with Meet, so it might be interesting to try implementing a device that does both and try other online call software.

I should also have read the spec more before diving into hacking on the HID implementation fork, as there’s a lot more information in the HID Device Class Definition, including how to construct the values for many of the fields (I’m looking at you “INPUT (Cnst,Var,Abs)“). RTFM is and remains a solid advice – and not just when one thinks there’s time.

Also regarding the specs: some of them I only find in the Internet Archive’s Wayback Machine. If you encounter a good source that should be kept, always add it to the Wayback Machine and preserve it for your future selves and others!

This exploration of USB HID pulled on so many threads, and left so much unfinished, that it’s a fertile ground for the future, even more than most previous projects.

More call functionality

The most obvious thing is to implement the whole setup with the buttons. I’ve tried Hook Switch to hang up a call, that works too. Could add status lights, maybe throw in some “Active Call” LEDs, or so on. This requires better understanding how data is sent over the wire for USB and how to handle incoming data. The Arduino examples rarely seem to use the “Output” fields (ie. incoming data, output from the host’s point of view, but maybe TinyUSB does ?

For this, it would be nice to find a different hardware platform that would make this more seamless (so I can concentrate on the software side more). If that platform would lend itself to be reproduced or made stand alone, that would be even nicer: imaging brining my little call control box that can be used with other computers easily as well…

Implement more USB HID devices

The Arduino HID project has a bunch of devices implemented, but there are an infinite numbers that could be added. Unfortunately for Arduino it is harder to add more device types as an add-on to this library versus the current “forked” approach⁴, so new decices should be in the main project, eventually.

So far there’s no Telephony device implemented there and it would be nice to find the right level of abstraction that works. The library doesn’t implement specific HID table pages, but specific usages or a subset of a usage. Thus like always, the hardest part would likely be setting the right interface (the right specs and “API”) for a new device to implement both the HID reports and the functions that manipulate what’s being sent and when.

On the other hand, that does sound like a fun experiment, and I’d look forward to adding 3D Game Controllers (Game Controls Page 0x05), Environmental Sensors (Sensors Page 0x20, Usage ID 0x30-3B), … or even a Submarine Simulation Device (Simulation Device page 0x02, usage id 0x05). These are stuff I go to Hackerspaces for…

WebHID for internet plus USB

While debugging this HID device behaviour, I found also WebHID that brings such devices to the web. This feature seems to be behind Meet’s and other phone systems like 3CX expanding USB support outside of the OS and into the browser. And no, Firefox does not support it, furthermore declined supporting it.

Nonetheless it’s very cool that (if I upskill a bit), I can create a web page that would help me debug such HID development:

• request devices that are filtered in various ways (vendor, product is standard, but usage page and explicit usage is the main key). This is likely what Meet does as well, “just gimme devices with Telephony usage page (or Phone usage? Need to check exactly)
• read the HID report collections sent by the device, so the results can be debugged, and
• read device input events that we can then either log for debugging or in an application react to to it

This opens a lot more mashup opportunities by the dozen.

Finally

Unlike most other projects I had where I’m focused on one specific outcome, this turned out to be more focusing on getting a new toolkit (custom USB devices) up and running, so I can think about a wider types of projects to do. In that sense, this feels a big success, even if I know how little I know about programming outside of my day-to-day environment. But ignorance is not a bliss.

And now, going on mute.

1. Many moons past I used to use a Jabra Evolve 80, that has a USB accessory controlling call features, so I did have first hand example of what sort of experience I’d like. ︎
2. I’ve tried reviewing the hardware schematics, looking into the pixel ring control functions, and given that the LEDs seems standard I’ve also attempted to use the FastLED library to drive them instead, so far nothing. I still bet on hardware differences from final schematic + my inability to debug it, but it can be faulty hardware just as well. Needs more effort – in the future. ︎
3. The Arduino code became more “simple” once I realised that things set up this way do not need debouncing for the touch sensors. In other cases that would be essential, there’s sooo much flaky signal to use those terminals as momentary switches or similar. ︎
4. At least I don’t know how nicely extend a library for C++, if that’s even possible. Keen to learn, though. ︎

The post Making a USB Mute Button for Online Meetings appeared first on ClickedyClick.

Thus in the last two weeks I was solving problems, though not just any problems, but went in mostly for the Easy ones. These few dozen problems and short amount of time doesn’t give me a deep impression, but from past experiences I can still distill some lessons that help shaping future experiments.

The purpose of using the Easy problems is different from e.g. going all in for puzzle-solving fun, which is likely in the Hard ones. Rather than that, I think easy problems can be used for learning some new techniques, looking for common patterns, and becoming more polygot.

New Techniques & Common Patterns

It’s 100% that I’ll be able to solve the Easy problems by myself (otherwise I should give back my nerd card). On the other hand, there are always more than one way of doing things.

For example, many Pandas problems on Leetcode could be done with various “merge” or “join” application, or oneliners versus intermediate variables. It’s fine to know one, but then checking the solutions by others, and finding the alternate ways, trying them out, and seeing their tradeoffs.

Another kind I’ve encoungtered, that’s maybe only new to me, but illustrates things, is solving certain problems while iterating one way (incrementing counters, from left to right, etc…), while in reverse (decrementing, right to left, etc…) the solution might become neater or more obvious.

After solving the problem one way, just going through others examples, looking for options that I haven’t considered, and pitching them against what I did originally definitely expands my toolkit. In addition since these are easy problems, it affects more common situations, code that I would encounter day to day.

Skipping through the other shared solutions I can also get to see common patterns. It’s a bit like statistical mechanics that while among the solutions there’s loads of crap¹ but on average the patterns of solutions appears that is relevant for a given language.

Polygot Programming

Given that most Leetcode problems are allowed to be solved in multiple languages, that supports learning across languages very well. For example Pandas problems paired with solutions in MySQL: sometimes the pattern of solutions were very similar in the two, other times needed very different thiking to get to a solution that felt “right”-ish. Similar case in the other combination of Python and Rust that I’m trying to dive into.

The trick is to try to solve things in any way at first, and then do the previous “look for techniques and patterns” to build up the experience.

Using Easy problems in this case gives a lot of different, simple use cases where the basics can be compared easier, focusing on the language similarities and differences. The problem difficulty then doesn’t muddy the waters.

Build Up Programming Muscles

The Easy problems have smaller scope (that’s why they are easy), and seems to allow improving pattern recognition, on which other, more difficult problems can build on. It’s all about building shortcuts based on encountering similar stuff and making the connection, and building chunks that can be reused later.

Most of the day-to-day programming likely encounters these sorts of easy problems anyways, thus my hypothesies that they can be good bang-for-buck for improving baseline effectiveness. Of course if someone always works on big-hairy-problems, this would stand. But how common is that?

Flipside: Why Not Do This?

The incentives at Leetcode doesn’t seem to align towards quality overall. When developing for efficiency, the run result timing seems to support that but the variance is way too high to be really useful. The solution sharing and feedback also seems to support that, but there’s way too much noise and broken feedback looks to be effective at that. Thus the quality of the above mentioned steps really depends on how much effort I put in there. It’s very easy to grind (get a lot of problems done)², but learn almost nothing, or learn the wrong patterns.

Wrong patterns is indeed the main culprit. The problems are not set up to be “production quality” that would make one a good engineer. Rather than doing things in a clever way, or going in for the oneliners at the expense of any readability, or using a solution that is more hard-coded & tailored to a narrow case rather than with a bit nicer way that generalises, etc…

Most of the time I also cannot even really remember the solution from even an hour ago, probably the side effect of going for quantity (easy) instead of quality (hard problems).

Next Steps

Leetcode is definitely looks like a reasonable tool to have in one’s kit, though not as the main one. I think I’ll do look around more among the Easy problems to see more patterns, but for more interesting ones I’d go gradually up, otherwise I’d just drop it soon enough.

On the other hand, the more useful step might be picking up my learning paths on Exercism, where a few things are diametrically opposite: more quality focused, direct feedback from knowledgeable people, and empracing iterative improvement. It seems less polygot (there are many languages, but they are learning paths independent from each other, even if some problems do repeat between them), but that’s not a showstopper.

The main thing is, though to continue a habit that is created by this experiment: deliberately practicing programming, seeking out alternatives, and not taking them at face value.

1. Given the gamified upvoting system, it’s no surprise that it creates so much spam. ︎
2. I’m still surprised that with 36 Easy problems, I’ve already soved more of them than almost 70% of people. ︎

The post Doing the Easy Problems on Leetcode appeared first on ClickedyClick.

Definitely wanted to tackle it using Python as that’s my current utility language that I want to know most about. Since the aim of Protohackers, I think, is to go from scratch, I set to use only the standard library. With some poking around documentation I ended up choosing SocketServer as the basis of the work. It seemed suitable, but there was a severe dearth of non-dummy code and deeper explanation. In a couple of hours I did make some progress, though, that already felt exciting:

• Figured out (to some extent) the purpose of the server / handler parts in practice
• Made things multi-user with data shared across connections
• Grokked a bit the lifecycle of the requests, but definitely not fully, especially not how disconnections happen.

Still it was working to some extent, I could make a server that functioned for a certain definition of “functioned”, as the logs attest:

On the other hand, ended up in a relative dead-end, as some message ordering issues kicked in, and reliably failed the test here, not knowing much what to try next just yet:

Since it’s a learning exercise and definitely not a competition on my part, I started to procrastinate. Not long before I’ve looked at the status of the leaderboard. Funnily enough, looking at the top entries, they were linking to the repositories where their solutions were!

Shoulders of Giants

Here’s my surprise and delight started, though. Within the first 7 entries there were 3 with Python implementations that included code! Even better, they actually covered 3 completely different ways of solving the task. Jackpot, really!

1. The first solution used pure sockets, which is quite versatile if I’d want to go all-in on low-level networking in the future. It had quite a lot of helper code, though which makes it look like a pretty decent effort to duplicate.
2. The second solution went with SocketServer just like I’ve tried, and that is nice to dig in a bit more, given how small the whole code is. The main thing here was that I should have understood from the problem description this being a Streaming TCP connection case. Looks like streaming is the part that takes care of a lot of details, including the connection/disconnection that plagued me. Bam!
3. The third solution then used asyncio, to take it in a different direction again. It’s amazing how simple it all is when the relevant components and abstractions are understood.

Which one is the most tempting solution to follow (and/or learn from)? Pure sockets are likely just a fallback option when there’s nothing else. On the SocketServer vs asyncio front however there was some useful StackOverflow discussion, even if a bit dated, coming from 2016. It pointed at the different use of threading and event loops. I guess this would make this answer a bit unsatisfying, but quite realistic: learn both and know when either is applicable for your use case.

What did we learn?

In the end I haven’t finished my code yet. Reading the existing solutions influences me and just adapting what others did and submit would feel like cheating (to myself). The way to resolve this is setting your own goals on top of the original challenge. Here I picked the following, and achieving these would complete things for me:

• Use proper project structure and try out PDM
• Figure out how to set up the project & code to be testable with pytest (basically grok testing of programs that run servers)

The combination of these focuses on something akin to “going to production”, besides obviously writing the actual code, which is very much relevant to my interests.

So far I haven’t seen many examples of testing SocketServer, though there’s Python’s own test suit that could be a starting place. It has a lot of super useful helper functions (such as finding an unused port to run the server on), but overall seems a lot of boilerplate too. For asyncio I haven’t looked around yet. It being “cooler” there might be more discussion around it, but it’s by no means a given. Would be interesting to combine this with a Basic Chat client as well.

Another impression from today’s effort is that Python modules are documented to very varying levels. Their complexity definitely jumps when I try to go from dummy stuff to anything useful. For example here understanding the proper role and interaction of the Server and Handler parts of this multi-user environment.

I’m also acutely aware that my networking knowledge is very patchy regardless of doing networking-adjacent stuff for decades. It’s a very useful frontier to tackle when I have a chance.

Finally, ngrok is still very cool tool, nice to be able to sit in a cafe and safely exposing a server to the internet.

The post Programming challenge: Protohackers 3 appeared first on ClickedyClick.

1. A banking website with CAPTCHAs and no programmatic access
2. An email received each day with an password-protected PDF containing the last day’s transactions in a table

Neither of these are fully appetizing to tackle, but both are similar to bits that I do at #dayjob, but 2. was a bit closer to what I’ve been doing recently, so that’s where I landed. That is:

• Forward the received email (the email provider does it)
• Receive it in some compute environment
• Decrypt the PDF
• Extract the transaction data table
• Clean and process the tabular data
• Put raw in some data warehouse
• Transform data to get the right aggregation
• …
• Literally profit?

I was surprised how quick this actually worked out in the end (if “half a weekend” is quick), and indeed this can be a first piece of a “personal finance data warehouse”.

Technical implementation

I wanted to have the final setup run in “The Cloud”, as that’s one less thing to worry about. The most obvious arrangement, based on past experiences was combing AWS Simple Email Service (SES) to receive an email, and a Lambda to run serverless processing. On the data warehouse side the real obvious choice is GCP’s BigQuery, however, so I looked into what would be a similar arrangement for the processing pieces if I want to put everything into a single cloud provider.

After some docs diving the most natural arrangement on GCP seemed to be quite different: an App Engine deployment with Mail API enabled. This gives a receiving domain name (@[Cloud-Project-ID].appspotmail.com) , and every email sent there is just passed to the server that is running in App Engine. This seemed pretty simple! App Engine also has a free tier, though that comes with pretty small memory limits, which features in this story too.

The final result of the server part is shared on GitHub, and should be easy to reuse or extend.

PDF processing

Getting the attachments out of the email was pretty straightforward with the Mail API, so the first heavier task was opening the encrypted PDF and getting the table out of it. Opening PDFs are quite common, but the table extraction was a bit of a journey.

False try

First I was searching around (as anyone else does) for someone else’s rundown of the options, as an example. From there I honed in on pikepdf to open the password-protected files, an tabula-py which seemed handy to extract tables right into Pandas DataFrames. One subtlety was that tabula-py is just a wrapper around tabula-java to do the extraction, and needs a Java environment installed. The free tier of App Engine uses their standard environment where all I have is my code and “requirements.txt” to install my python dependencies, so it’s obvious how would I get Java into the deployment correctly.

Enter the scene install-jdk which can install the Java environment at runtime. That was sufficiently crazy hack to actually work, and it did work. Or so it seemed, since the data was processed and showing up in BigQuery, when I’ve sent test emails into the system.

Upon closer inspection, though, there were loads of duplicate lines. Between signing off in the evening, and checking it in the morning, I had bunches of them, and were still coming in…

I should have checked the logs earlier, because once dig in, there were bunches of “server errors” listed that didn’t connect to any programming errors that I might have made, rather than (here comes the epiphany) instances being killed for being out of memory / blowing their memory budget (of 256MB for the free tier). Thus what happened is:

• the Java run of tabula was just using too much memory while processing the PDFs
• it finished processing and like loaded the data but it takes a bit of time
• GCP catches up and kills the instance while that is still going on, and reports to the Mail API that the email hasn’t been properly handled (server error during that process)
• Whatever is handling the incoming email queue in GCP will just just keep the data and retries later
• The cycle repeats…

This didn’t seem very helpful and the repeat emails were piling up in whatever (opaque, to me) system GCP has, so needed a quick replace of tabulate with something lighter…

Worse is better and actually good

Going down the list of recommended libraries, next I looked at camelot-py which looks great, but needs OpenCV on the machine to do its work, so back to the “how to install OS packages on Standard AppEngine?” question. For some extra inspiration I was looking at camelot’s comparison with other similar tools page and it was a bit disappointing (though not surprising) that pretty much every other library is “worse” on various PDFs compared to camelot. Just for kicks I did try some out, and pdfplumber actually delivered:

• it does actually work on the example PDFs I had from previous bank emails
• nothing else beside pip install
• it can actually handle decrypting the PDF as well, so helper libraries can be dropped
• the extracted data is in Python tables, but it’s just an extra line to get DataFrames, so no sweat
• The extracted data was actually better quality than tabula’s, so had to do fewer cleanup steps!

This was a pure win, and indeed it’s worth looking stuff that works with the data at hand, not ignoring the edge cases, but also not overly emphasizing being able to do “everything” when there’s a clear target of what “thing” needs to work. (Potential technical debt considered too).

Data transformations and visibility

Now the data sits in BigQuery properly:

The raw transaction data loaded into BigQuery was the first step, but still need to answer the question: in this billing period, how much have I spent?

Not being a data analyst (or not yet?:), this took a bit of figuring out. As other novices share their bit of “clever code” when it’s actually trivial to the experts, I’m sharing here the bit of SQL queries in a similar “that was fun to figure out, wasn’t it?” way. I’m sure it can be much improved, but it’s a good reminder for myself as well.

Given that my billing period starts on the 23rd of the month, get the aggregated value of transactions for each billing period:

WITH
  Aggregated AS (
  SELECT
    DATE(TransactionDate, 'Asia/Taipei') AS day,
    TransactionAmountNTD
  FROM
    `personal-data-warehouse.finance.huanan` ),
  calendar AS (
  SELECT
    day,
    -- Find the last day before the new interval
    DATE_SUB(
      DATE_ADD(
        day,
        INTERVAL 1 Month),
      INTERVAL 1 DAY
    ) AS endday
  FROM
    UNNEST (
      GENERATE_DATE_ARRAY(
        -- Start date in the past before any data,
        -- on the right day of the month for
        -- the billing cycle.
        '2022-05-23', 
        CURRENT_DATE('Asia/Taipei'),
        INTERVAL 1 Month
      )
    ) AS day
) SELECT
  SUM(TransactionAmountNTD) AS `MonthlyTransactions`,
  COUNT(*) AS `TransactionCount`,
  EXTRACT(Year FROM c.day) AS `Year`,
  EXTRACT(Month FROM c.day) AS `Month`,
  FORMAT('%d-%02d', EXTRACT(Year
    FROM
      c.day), EXTRACT(Month
    FROM
      c.day)
  ) AS `Interval`
FROM
  calendar AS c
JOIN
  Aggregated AS a
ON
  a.day BETWEEN c.day AND c.endday
GROUP BY
  c.day

Good stuff on the date array and joining with a “between” statement, those are the main TIL. They also already came up at #dayjob, which was very satisfying.

From here the data I surface in a connected Google Sheet which is pretty practical, though leaves the “being notified when I approach/reach X” out, but that’s fine for now.

Testing and getting to “production”

One good thing about personal projects is that I can make them as “good” as I want to (or as “bad”, of course), which usually results in an unhealthy amount of tweaking, trying out various best practices to see if they work, and so on. Here I really wanted to get the system well tested, for example, which turned out to take loads more time than actually writing the original service. Actually, there’s nothing surprising about that for software engineering professionals, but still can catch people off-guard.

Here the tricky parts came from two areas: FastAPI settings and cloud service integrations.

The former is always a bit of an issue, depending on how the code uses the settings (whether things can be patched well at testing time), but here I also used a trick for the server to pull the PDF decryption key from Secret Manager, so I don’t have to deploy environment files, nor keep settings like that in version control, etc… But this meant a trickier flow of getting the FastAPI testing client up in a way that it worked without it talking to the cloud backends (and stalling, and failing…). Nothing that some good mocking cannot solve (says the person with hindsight).

For the cloud services part it meant mocking BigQuery connections, so that the test can actually pretend to “receive” an email all the way looking at the “database” and see the right information being there. Under the hood I’m using pandas-gbq, and thus it was interesting to look under the hood for their tests, borrowing some of them. Took a bit more time, but that’s working pretty well now. Still need to do some extra bits and pieces to do cover more of the workflow, but I’m already more confident about things working. Also, all this will be very useful on other projects that are interacting with BigQuery in any way (not just through Pandas).

Big evergreen lesson on testing: you have to write your code to be testable. Lots of code out there is not even not tested, but it’s even extremely difficult to actually test. This needs remembering in every development. Also, test writing never really stops, there’s always more thing to test for. And finally, can always try more advanced testing, such as using automated test case generation (e.g with hypothesis), and fuzz testing (e.g. with pythonfuzz). The next frontier, right after I’ve implemented the currently skipped tests. And finally, remember that code coverage is not case coverage, so the goals should be maximizing the latter, while the former is just a potential proxy for it.

Future outlook

It would be nice to take this idea of financial data analysis further and add some actual dashboard (say deploying Superset somewhere which is excellent for this). It would help to get more information into the system as well, though, currently it’s very sparse. That would mean adding other financial sources, maybe if finding an API in the end, or doing a bit of “pragmatic execution” and do a CAPTCHA bypass (since I quickly checked that my credit card provider’s CAPTCHA is completely readable by Tesseract, for example, so I could likely scrape things there if I really wanted.

I’m not holding my breath for having something like UK’s Open Banking here which enables apps like Emma so all this is accessible for people who don’t want to code. But where’s the fun in that (for me)? :) (In fact there’s a lot of fun in open access APIs, so this would be the real way of doing it…)

Finally, it’s good to remember how easy it is to corrupt “production” data sets, but also with the right tools (like snapshots), some of that pressure can be less. There are always bugs, the question is how to mitigate their effect.

The post A personal finance data pipeline project appeared first on ClickedyClick.

On the other hand, I’m one for meta-games, especially when I have doubts whether I stand a chance in the game itself, thus came the idea of do something around 100DaysToOffload that might also result in a blogpost. Hence came the “Hundred Days to Offload” WordPress Plugin idea: get a bit of coding in, make something useful to see if the game has been “won”, an also get one (or more) write-ups out of it.

Spoiler alert: it’s working now, very barebones, but to the point… that there’s a long way to go.

In the process, that took a couple of days over the weekend, I’ve revisited PHP, that I used to “play” with for projects before, though haven’t done anything serious, nor made it part of my Language of the Month series, so far. It was still quite interesting to revisit with more mature eyes of e.g. how good projects look like in the Python ecosystem (where I spend most of my time), and whether lessons learned there are applicable here.

The Making of a WordPress Plugin

The plugin itself is pretty straightforward: query WordPress for all the post that were published in the last 1 year, and count them. It could be slightly more complicated, if the goal would be more closely aligned to the “100 days to offload” name, that is if one would have to count the number of unique days on which posts were made, but let’s not move the goal-posts here.

Starting from the relevant WordPress Handbook page I could have an initial stab at laying out a plugin project structure, though I have to say, that the examples are not going too far in helping with that. This was augmented by a GitHub wordpress-plugin topic search, where I could work from examples a lot better, and see some (maybe too many) best (or at least reasonable) practices.

This lead me to various ecosystem tooling such as

• Composer, the PHP dependency manager that I didn’t know before, and could very much relate to (with it’s prod/dev dependencies, scripts definitions, etc, reminding me of Poetry for Python a bit in spirit)
• PHPStan static analysis tool, and its “fun times” with WordPress code (which can be fixed by various stub projects, etc),
• all the “code standards” tools that do various fixes (while not being quite a linter and code formatted, but still being a little), for example wpcs. This latter had a fun side effect that have to work from its cutting edge “develop” branch to work with PHP 8.1 that I had at hand, so definitely learned through that a few Composer quirks.

In the end I had my little snippet of code, run through all these codes, fixed up tabs vs spaces, docs strings, types for return values, dependencies, actual bugs of wrong input types to various PHP functions, and also some best practices (which might or might not apply to my little code). It’s a rabbit hole that could go a lot deeper, though, so cutting a release version at a point. The rest goes into an issue tracker to pick up one day.

And now, with an MVP (minimum viable plugin;) at hand, I’ve submitted for review. That’s just the cherry on the top, or start of new problems with a project to support. Either way I can use the plugin checked out with git just fine.

I’ve definitely picked up a tiny bit of PHP for now, even if mostly accidentally, and likely full of bugs and bad practices. It’s still interesting, and a learning experience. For example data handling is a source of many, various pains in all languages I’ve across so far. The PHP way (or WordPress way?) of object orientation is also just scratched on the surface and accepted as it goes, but not from a point of solid understanding.

Looking to the future

From the plugin’s side it would be good to add some style and looks, as it’s super dummy and bare bones. But how should it look? Time to learn a bit of design and/or run some user testing? The mixing of HTML and code is also particularly mind-bending (or awkward, or intriguing, or…), I would need to look where to set the right boundaries for that, as there could be many.

Adding CI/CD as well, so definitely various PHP versions can be tested (seems easy), and potentially various WordPress versions as well (seems a lot more hairy) could be a practice and baseline.

Adding some settings to change would help me to learn more plugin development, though this particular plugin might not benefit a lot from it.

Work out how to do proper translations / i18n would be probably pretty solid benefit as well, and that needs a lot more digging (variable texts are not straightforward to translate).

Finally, all of these improvements might also just make a few extra blogposts to get towards my 100 Days to Offload.

The post 100 Days to Offload WordPress Plugin appeared first on ClickedyClick.

The router I have is a less-common variety, an Alcatel Linkhub HH441 (can’t even properly link to on the manufacturer’s site, just on online retail stores). At least as it should, it does have a web front-end, that one can poke around in, and gather metrics from – of course in an automatic way.

Looking at the router’s web interface, and checking the network activity through the browsers’ network monitor (Firefox, Chrome), the frontend’s API calls showed up, so I could collect a few that requested different things like radio receiving metrics, bandwidth usage, uptime, and so on… From here we are off to the races setting up our monitoring infrastructure, along these pretty standard lines:

• Set up a Prometheus metrics exporter, pulling data from the router’s internal API (the same way the web interface does it)
• Spin up a Prometheus + Grafana interface to actually monitor, alert on, and debug any metrics

Metrics Exporter

Given that I’m mostly working with Python, using the existing Prometheus Python client was an easy choice, in particular using their internal HTTP exporter to get started quickly. It was relatively straightforward to turn many of the metrics into various gauges (radio reception metrics, bandwidth used, &c.), though some were naturally info fields, such as mobile network name and cell ID. This latter would be very useful as my hunch was cell hopping by the router is what’s mainly affecting my network quality.

After some poking around I’ve also realised, that the API exposed is just JSON-RPC (although the router’s backend doesn’t seem implement everything in there, e.g. there’s no batch), which made a lot of things clearer, and potentially easier to use.

In the end, I’ve ended up with one class to do all the metrics gathering from a couple of JSON-RPC methods, working relatively robustly. The authentication was simplified very much: most need an auth token that can be extracted by manually observing some requests (more on this later) and some need a referrer header for the request to pretend to be coming from the admin console.

The resulting code is on GitHub: imrehg / linkhub_prometheus_exporter, and should be a full-featured server with most (though probably not all) the metrics available in the admin console as well.

Monitoring

With the metrics exporter running, I used a Docker Compose-based Prometheus + Grafana stack locally to have everything together, just adding an extra “linkhub” task in Prometheus to pull the data periodically, and a new dashboard in Grafana to have a quick overview.

I also went a bit overboard and added some extra bits and pieces, like coloured regions for the signal metrics to show what’s bad / acceptable / good / excellent or so, based on some scouting, making it clearer when things are good or not good.

I also tried to use a bit more of Grafana’s tooling (not a lot, but a bit more), so added some different sections for signal quality and network metrics, as well as a running average on some of the noisy metrics.

Lessons learned

Learned a bunch of things as this was the first time I used, from scratch, many of the tools here. The very first one being: how to choose the right Prometheus metrics for various data streams? Now I see how does it look like in practice, planning for a metric that needs to be monitored from the very early stage. There are fewer varieties of metrics that I’ve expected, and while there’s a lot of derivative stuff to make it a lot more useful, it’s not that everything that one imagines can be made to work.

Used Poetry here more than previously, and set up poetry-dynamic-versioning plugin (as a candidate competitor to setuptools-scm). That meant also using poetry plugins and a beta Poetry release at this stage. It’s not bad, but sooo many gotchas in the process, and still have to figure out what would be a good reusable template for projects using these. (including __version__ variables, etc).

Figured out how to do good CI Docker image builds with libraries that rely on git for versioning, fortunately setuptools_scm did the work for us: bind mount of .git in the specific build step. I think in CI/CD all this reliance of repo data being available can still make things a bit trickier, but something’s gotta give, and it’s not much extra compared to the rest of the things.

Learned a bit about JSON-RPC (and how the router might or might not be fully compliant). Not sure if I’d go with that for any future project myself, but good to be aware of it, and potentially looking at its presence in other routers or interfaces’ communications channels.

Chance to use some Python 3.10-based features (match) and hit/fix some of GitHub actions related issues with 3.10:way to go libraries that convert 3.10 to “3.1″ because it’s a number so let’s round it, right? Or actually way to go libraries/YAML to allow both ‘x.y’ and x.y forms (quoted and unquoted), and the former would have been the correct form all the time, but people generally go with the latter to save a few keypresses. It’s subtle, but experience is expecting the subtleties and the reasons for them arising.

Seen how mypy can actually benefit the coda quality: while trying to fix all the reported issues actually found stuff that was clear benefit and it’s coming not from adding all the type hints (that’s good, but baseline), rather than being smart where it complains and think about what’s the underlying issue (e.g. patterns of getting values out of dicts where there might not be result, exhaustive matching of match and return values of functions, etc…)

The resulting Docker image is north of 1GB due to Python, and that’s not great considering that it doesn’t do that much work. Writing/rewriting this whole thing in Go could be interesting and would be useful learning experience (or another compiled language, I guess, but Prometheus itself is written in Go, so there’s a connection). One step at a time, projects written in Python are useful proof-of-concept to compare other stuff against later, so it was well.

Having said that, I’ve seen the best practices listed when writing Prometheus exporters, and given the current environment, I couldn’t apply all the best practices. For example: “Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers.” The official Prometheus Python Exporter on the other hand seems to need to use exactly that sort of “while True” loop to keep getting/storing metrics, instead if running on demand. There might be a more subtle pattern to do on-demand work (which I see to be more correct), but I need to find it.

So what have I learned about the actual network issues? Most of the instability seemed to be correlated with switching to specific cell towers (based on cell IDs). Certain cell towers would pretty stable, and on some of the days the router was switching between towers, and that’s when most of my online calls were pretty futile.

Finally, I did think a lot about the adage that “something that isn’t worth doing isn’t worth doing well.” On the other hand there’s no kill like overkill, so here we are…

Future development

Compared to other projects I’ve done, this might be lighter maintenance, given that it’s sorta done for the moment (except if others start to use it and need other metrics, for example). Otherwise the Docker-based deployment and poetry.lock’d dependencies make bit-rot a bit slower, hopefully. In the meantime, I’ve switched to a wired connection, so unlikely to need this project much, but it could be that much of this will be repurposed for other monitoring projects.

The post Creating a Prometheus metrics exporter for a 4G router appeared first on ClickedyClick.

This is the story of how I’ve tried one particular project that seemed easy enough, but leading me to go back a few (a bunch of) steps, and rethink my whole approach.

I bet almost everyone in tech (and a lot of people beyond) heard of AlphaGo, Deepmind’s program to play the game of Go beyond what humans can do. That has evolved, and the current state of the art is Alpha Zero, which takes the approach of starting from scratch, just the rules of the game, and applying self-play, can master games like Go to an even higher level than the previous programmatic champion after relatively brief training (and beating AlphaGo and it’s successor AlphaGo Zero), but also apply to other games (such as chess and shogi). AlphaZero’s self-learning (and unsupervised learning in general) fascinates me, and I was excited to see that someone published their open source AlphaZero implementation: alpha-zero-general. That project applies a smaller version of AlphaZero to a number of games, such as Othello, Tic-tac-toe, Connect4, Gobang. My plan was to learn by adding some features and training some models for some of the games (learn by doing). That sounds much easier to say than to do, and unravelled pretty quickly (but probably not as quickly as it should have been).

I’ve picked the game Connect4, because a) I used to play that long time ago, b) feels like a relatively simple game, while still interesting, c) the repository didn’t have a pre-trained model for the PyTorch platform, that I wanted to try.

PyTorch was the choice, as it works both on GPUs (as the preferred workhorses of machine learning projects) and also on my laptop that only has a CPU to use.

The unraveling

Getting started was easy enough. The neural network setup seemed to be pretty much the same between the different games, so I’ve just copied and adapted another PyTorch setup to Connect 4. Run it briefly on my laptop, and it was doing things, it was playing games. The training takes place in 3 phases: first generating some example games (from random valid moves, or existing model, I think); next train the model on those example games to recognize better what it takes to win; finally the new model is pitted against the previous one, and either accepted or rejected as the basis for further training, based on the win-draw-lose percentages.

I’ve trained for a while, and then played against the model to see how it feels. Seemed sensible enough, but of course, since I don’t play the game too well, so didn’t know what to expect.

I’ve started to look for some better Connect4-player software that I can pit the trained model against, when realized that it is an already solved game, with a perfect strategy! You can see either Expert Play in Connect-Four by James D. Allen, or this entire masters thesis, A Knowledge-based Approach of Connect-Four by Victor Allis. I … am yet to read either, but since I found an online implementation of the Connect 4 Solver, I knew that will help me to test how well the trained model work. Since the game is solved, and the first player will always win, that should help simplify and the training a bit, right? For example, I would know when the perfect strategy is achieved: when in the post-training it consistently gets 50% wins, 50% loss, and 0% draw, then it plays perfectly.

Trying to validate this hunch, I was training it on my laptop, but the training times went up tremendously as the number example games are increased in each iteration. My laptop generated example games fast enough, but then the training was getting slower and slower. Needed to search for an alternative. Fortunately, I had access to an idle NVIDIA Jetson TX2 machine, which is more-or-less a 64-bit ARM machine with some pretty decent GPU with plenty of CUDA power attached. Was designed for machine learning, and should be perfect (or rather overkill?) for my application.

Setting up wasn’t too difficult, though I think the pytorch module has built for a whiiiiiile, so I saved it as a Python wheel to make it quicker to install next time. Then started the whole training process afresh the TX2. The second, training step was indeed a lot quicker (30-60 minutes instead of 4-8 hours), though now since the ARM core of the device is much slower than my X86 laptop, the first, game generation step and the last, tournament phase were a lot slower (an hour instead 10-15 minutes if I recall). Overall it’s still a win, so could run a handful of iterations in the process each day. It was the winter holidays, so there were plenty of days for the machine to train.

What I ended up seeing is a kinda static phase. Almost all draws in the tournament phase, with very few wins. There were many iterations with nothing really changing, new models being rejected for the lack of wins. Then pitting the model against the solver, it starts off well enough, for about 4-5 moves, and after that, the algorithm makes a move that is impossible to recover from…

I was trying to change the code a bit, based on some quite interesting issues opened on the original repo. For example, there’s one asking why the Monte Carlo Tree Search is set up as it is, and whether another setup would help it train faster? (Faster is good, faster to see both successes and failures in the whole training process. Can’t waste time, when everything takes hours!). Then there’s another question regarding the models ending up in draws all the time (that’s familiar?). Or this very straightforward “Connect4 Not Learing“, which is obviously relevant to my interests. All are open and unsolved questions, so I went to look a bit deeper, that prompted even more questions.

Why the implementation uses its neural network setup as it does, and does it work since it is much simpler than the original AlphaZero, as mentioned in the code author’s blog? What do the different neural network designs mean, and how can I see quickly if I’ve set up one that has the potential to learn correctly or it’s a dummy? How does the AlphaZero algorithm actually work? Is it all tied down properly, or some parameters/implementations are not clearly defined in the papers, as the issues I’ve mentioned above hint? How does PyTorch work and how do I use properly (and optimize it, much further down the line)?

You might say these questions are very basic, fundamental, and that’s indeed the case. One misses the fundamentals in the hubris of jumping into a project that just because I have a general birds-eye view of the problem and some code of an unknown quality that someone’s written. It’s not all set and garden path to some sweet easy wins. I am not writing off that two weeks of experimentations as waste, but do need to change the approach.

Now what?

At this point going back to the basics sounds like the right way, and with more humble, step by step learning build up the skills to make more sense of the whole project.

• Get to know PyTorch, and make some working examples building on the tutorials (or get inspired by them). Looking at some online courses like the one by IBM shared on edX. Work until the tools are getting comfortable.
• Read the AlphaZero papers (the original, and the follow-up linked from their website), as well as explore around the topic following their references and articles referring to them
• Properly review the source code of the open source implementation I was using, and the accompanying tutorial. Not just skim and pull code blindly.
• Bonus, find some higher level inspiration for neural network design (as getting the network right is the big part of the whole thing, that’s how it look). Sources like The Neural Network Zoo by the Asimov Institute, which shows different networks for different kinds of tasks.

This should get me to a point where the original plan has some chance of succeeding. And if it sounds like a common-sense plan, it is because yeah, maybe that’s what a lot of occasions need, a bit more common sense… I wonder if it’s a radical thought, but it took me a while to get to that.

In the future should also probably compare the different machine learning platforms as well, TensorFlow, Keras, MXNet, maybe there are other ones as well, and run some sweet-sweet learning in the cloud (have some spare credits here and there, I wonder if they will expire by the time I’m done with my “common sense plan”).

I wonder though if others also run into a wall like me, and if did, what did they do? Is there any other lesson in human learning that this machine learning story can provide?

The post How not to start with machine learning appeared first on ClickedyClick.

The target solution: light up an LED if a particular person is at home location. Sort of a basic alarm system, though notice that the location of the LED was not mentioned – it can actually be anywhere in the world, as long as there’s Internet connectivity.

I’ve used the Wio Link, a Grove LED light, an Olimex OLinuXino Lime2 board running ArchLinux for the server, and a Buffalo router with DD-WRT system installed.

Wio Link

Wio Link was introduced in Seeed’s Kickstarter campaign, where they have raised more than 8x of their original target. It looks like a neat little board, and was happy to try out when I got my hands on one.

Their wiki page has quite a bit of information, so it was easy to get started. Connect to power, hold down the configure button till the LED lights up in a “breathing” pattern, connect through their Wio Link app, set up the wireless network settings and so on. Once connected, can define what kind of devices are attached to the board, and it looks like most of Grove devices are represented there. I only had a Grove LED at hand, so added it (“Generic Digital Output”), updated it, which created a new firmware and pushed onto the device.

The first update took a couple of minutes, but it’s pretty straightforward. The device then also has an API link, which brings up a web page with all the options to query, control, and reset the attached accessories (in my case that’s the one digital output).

The API is pretty simple and usable, though definitely not perfect (will come back to this).

Since manual switching the light on and off worked like the charm, the next step was to enable automatic switching by the presence detection within the home automation system.

Home Assistant

Home Assistant is an open source home automation system written in Python. There’s a lot to like about it, and a lot of people are checking it out since a number of other, closed system providers are shutting down their services. On the other hand, figuring out the configuration to implement the automation steps I came up with is far from trivial…

Setting up Home Assistant is pretty simple. Create a new directory, add a new virtualenv environment, and install home assistant (I’ve added some bonus modules that the logs recommended):

$ virtualenv env
Using base prefix '/usr'
New python executable in /tmp/env/bin/python3
Also creating executable in /tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ source env/bin/activate
(env) $ pip install homeassistant
...
(env) $ pip install python-Levenshtein
(env) $ pip install colorlog
(env) $ hass --open-ui

Next step is adding the actual components (the pieces of the home automation).  I’ve implemented the Wio Link light as a Command Line Switch. It has two commands for “on” and “off”, plus an extra for checking the on/off status (this latter is a very nice touch, and works well checking whether the switch took effect). In the configuration.yaml  file in the ~/.homeassistant  directory:

switch:
  platform: command_line
  switches:
    presence_light:
      name: Home Presence Light
      oncmd: "/usr/bin/curl -s -X POST https://iot.seeed.cc/v1/node/GenericDOutD0/onoff/1?access_token=<token>"
      offcmd: "/usr/bin/curl -s -X POST https://iot.seeed.cc/v1/node/GenericDOutD0/onoff/0?access_token=<token>"
      statecmd: "/usr/bin/curl -s https://iot.seeed.cc/v1/node/GenericDOutD0/onoff_status?access_token=<token>"
      value_template: '{{ value_json.onoff == 1 }}'

group:
  remote:
    - switch.home_presence_light

Here the URLs are coming straight from Wio Link API page, and you have to add your token value and the correct channels instead of GenericDOutD0  if you plugged your light into a different connector than me. I’ve also removed their -k  command line parameter (allow insecure SSL connection), and added  -s  (silent mode). The last section starting with “group” is not really necessary, but could group devices to be controlled together.

Adding the presence detection through DD-WRT is just the same as in their example configuration, obviously add your own parameters.

device_tracker:
  platform: ddwrt
  host: 192.168.x.y
  username: <user>
  password: <password>
  interval: 10

I don’t like that username/password is in clear text, so better protect your Home Assistant device and this configuration file!

By the default configuration, after restarting Home Assistant this presence detection module will create a known_devices.yaml  file in the ~/.homeassistant/  directory and add all new devices (see the device tracker component page for some more info). Here I had my smartphone, after connecting it to the home network.

xxxxxxxxxxxx:
  name: HTC One M9
  mac: xx:xx:xx:xx:xx:xx
  picture: http://www.gravatar.com/avatar/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  track: yes
  hide_if_away: no

That file then can be edited, for example  adding device name, disabling tracking for some discovered devices (eg. the small server running the Home Assistant server itself), and adding an image. I’ve added an image using a Gravatar link, where the xxxx section is just the md5 hash of your email, get it by.

echo -n "my@email.com" | md5sum -

Alternatively, this week’s update added local file support too, as it was discussed in the Home Assistant blog, so files in the ~/.homeassistant/www/  can be linked through the http://127.0.0.1/local/  path, so no online link is necessary (but I think Gravatar is a good unified approach).

After all this, the control interface will look something like this (the Sun is added as I had my home’s longitude/latitude information in the configuration). My icon will show Home/Away status, and clicking the switch on the presence light will trigger the light on the Wio Link.

The final step is the automation. This was the hardest step, as the Home Assistant automation documentation and their cookbook is pretty shallow, and often feel confusing. In the end, one working result is this pair of rules, which turn the light on when status change to home , and turn it off when changes to not_home  for that particular device.

automation 1:
  trigger:
    platform: state
    entity_id: device_tracker.xxxxxxxxxxxx
    to: 'home'
  action:
    service: homeassistant.turn_on
    entity_id: group.remote

automation 2:
  trigger:
    platform: state
    entity_id: device_tracker.xxxxxxxxxxxx
    to: 'not_home'
  action:
    service: homeassistant.turn_off
    entity_id: group.remote

I’m sure it can be done differently, but I’m also just glad it works, as it took quite a few hours of checking documentation, trying things, and reading the Home Assistant source code to come up with a working solution.

It functions pretty well, though presence (arriving) is detected quicker than absence (going away), I guess that’s part of detection method and certain timeouts within DD-WRT.

Lessons

I quite like how it all turned out, and I’ve learned a lot along the way, in the about 5-6 hours it took to pull everything together for the first time.

There will always be bugs. When trying to read the Wio Link LED’s status, first I was trying to implement it as a RESTful Binary Sensor. Took a while to find that that component had a breaking bug. I’ve made a patch, but haven’t sent it upstream yet, because Home Assistant makes me fill out so much information before a pull request can be even considered. To make it worthwhile, I’ll review things and send the fix later.

Not all the Wio Link API design is great. In particular, turning the light on/off are empty POST request to two different API endpoints! That’s not really REST at all. I think the way it should have been is using the same API endpoint (one single URL), and the setting sent over the POST data. Because of peculiarity, I couldn’t implement the Wio Link control over the RESTful Switch component, which requires a single endpoint.

Since the Wio Link can be placed anywhere where there’s internet connection, this home automation system can incorporate data from anywhere, sending or receiving. Can have an LCD screen miles away that displays metrics from your home. Can incorporate external sensors, actuators, the whole shebang. This makes more of a “local area” or even “wide area” automation network, not just “home”. Let your imagination run amok!

The Home Assistant documentation is really lacking for me, and I say that while every component has code samples and there are a lot of cookbook samples around. The problem is that there’s very little consistency, there’s no overview, and the options are not documented extensively. There’s no list of “these are all the options and possible values”, and there are no “these are all the ways you can do this thing”. In the end, one needs a lot of trial and error and monumental amount of head scratching! (Use the Source, Luke!) All this basically means that Home Assistant is the “worst documented project that I’m the most excited about“. :) Fortunately the Home Assistant website’s source is also on Github and can submit updates if I figure out how to explain things better.

Creating automation through triggers, conditions, and actions is probably pretty straightforward at first. I guess it works very well for simple systems, though I wonder how it could be made to scale better. In this configuration based system, even just a couple of automation scripts can balloon to pages and pages of code, and possible very little operator understanding how rules possible cross-interact with each other. I don’t know what model would be better (some kind of state machine definition maybe?), but it would worth thinking about it for the long term of home automation.

There are more components defined than I could reasonably ever try in my life, just check the component’s page!

This abundance also means that the number of possible combinations are astronomical. I would wager, though, that the value of those combinations follow a power-low: most of the combinations are not interesting at all, while a relatively small number of combinations are the most useful in general! I guess the Cookbook would be a good place to highlight some more “common use cases”, not just covering the bases and filling in for the lack of documentation. There are some obviously more useful components, IFTTT component on its own can add a ton of functionality, or the RESTful components.

Future

I hope, that more device manufacturers will create components and generously add it to this lineup. I can’t see any downside and there’s just so much upside working with an open source system! Any system integrator that figures out Home Assistant will also have a big competitive advantage in some niches for sure.

I hope, that the documentation can be improved in the future and things will be easier to figure out (e.g. how would I update this project above to light up the LED if any one of the tracked devices is present and turn the LED off when all of them are absent?).

The huge number of possible component combinations to mix-and-match is also pretty paralyzing too (see the classic Paradox of Choice), I have no good “production-level” project in mind yet, even though I’m super excited about many components I’ve found so far! This particular project worked, because it’s already something I was thinking about, and could possibly replace an existing system in the Taipei Hackerspace!

Have you built something interesting? Would love to hear about it! :)

The post Home Automation Mix-and-Match appeared first on ClickedyClick.