Category: Programming

Categories
Programming Taiwan

A personal finance data pipeline project

Screenshot of a credit card statement PDF showing the tables that contain the transaction data.
A daily credit card transaction statement

I had received a (family) project brief recently. In Taiwan many credit/debit cards have various promotions and deal, and many of them depend on one’s monthly spending, for example “below X NTD spending each month, get Y% cashback”. People also have a lot of different cards, so playing these off each other can be nice pocket change, but have to keep an eye on whether where one is compared to the max limit (X). So the project comes from here: easy/easier tracking of where one specific card’s spending is within the monthly period. That doesn’t sound too difficult, right? Except the options for these are:

  1. A banking website with CAPTCHAs and no programmatic access
  2. An email received each day with an password-protected PDF containing the last day’s transactions in a table

Neither of these are fully appetizing to tackle, but both are similar to bits that I do at #dayjob, but 2. was a bit closer to what I’ve been doing recently, so that’s where I landed. That is:

  • Forward the received email (the email provider does it)
  • Receive it in some compute environment
  • Decrypt the PDF
  • Extract the transaction data table
  • Clean and process the tabular data
  • Put raw in some data warehouse
  • Transform data to get the right aggregation
  • …
  • Literally profit?

I was surprised how quick this actually worked out in the end (if “half a weekend” is quick), and indeed this can be a first piece of a “personal finance data warehouse”.

Continue reading “A personal finance data pipeline project”
Categories
Programming

100 Days to Offload WordPress Plugin

Hundred Days to Offload code view
VSCode view of the Hundred Days to Offload Wordpress Plugin

In the course of pushing myself to write more on this blog, I’ve come across the #100DaysToOffload project. It’s super simple: write a 100 blogposts in a year in your personal blog to unlock the achievement. It seems like gamification done to the right level, as it’s not to strenuous (“write every day” would likely fail before lift-off), and not too lax (100 blogpost are still quite a stretch to go!). Thus it looked like the right too to trick myself into doing the thing I already wanted.

On the other hand, I’m one for meta-games, especially when I have doubts whether I stand a chance in the game itself, thus came the idea of do something around 100DaysToOffload that might also result in a blogpost. Hence came the “Hundred Days to Offload” WordPress Plugin idea: get a bit of coding in, make something useful to see if the game has been “won”, an also get one (or more) write-ups out of it.

Spoiler alert: it’s working now, very barebones, but to the point… that there’s a long way to go.

Screenshot of the Hundred Days to Offload plugin in the WordPress admin interface.
How the Hundred Days to Offload plugin looks in practice (as of now).

In the process, that took a couple of days over the weekend, I’ve revisited PHP, that I used to “play” with for projects before, though haven’t done anything serious, nor made it part of my Language of the Month series, so far. It was still quite interesting to revisit with more mature eyes of e.g. how good projects look like in the Python ecosystem (where I spend most of my time), and whether lessons learned there are applicable here.

Continue reading “100 Days to Offload WordPress Plugin”
Categories
Programming

Creating a Prometheus metrics exporter for a 4G router

Grafana view of some of the reception metrics
Grafana view of some of the reception metrics

Recently I begun fully remote working from home, with the main network connectivity provided by a 4G mobile router. Very soon I experienced patchy connectivity, not the greatest thing when you are on video calls for half of each day. What does one do then (if not just straight replacing the whole setup with a wired ISP, if possible), other than monitor what’s going on and try to debug the issues?

The router I have is a less-common variety, an Alcatel Linkhub HH441 (can’t even properly link to on the manufacturer’s site, just on online retail stores). At least as it should, it does have a web front-end, that one can poke around in, and gather metrics from – of course in an automatic way.

The Alcatel HH41 LinkHub router that I had at hand to use

Looking at the router’s web interface, and checking the network activity through the browsers’ network monitor (Firefox, Chrome), the frontend’s API calls showed up, so I could collect a few that requested different things like radio receiving metrics, bandwidth usage, uptime, and so on… From here we are off to the races setting up our monitoring infrastructure, along these pretty standard lines:

Continue reading “Creating a Prometheus metrics exporter for a 4G router”
Categories
Programming

How not to start with machine learning

I’m a technical and scientific person. I’ve done some online courses on machine learning, read enough articles about different machine learning projects, I go through the discussions of those projects on Hacker News, and kept a bunch of ideas what would be cool for the machines to actually learn. I’m in the right place to actually do some project, right? Right? 🚨 Wrong, the Universe says no…

This is the story of how I’ve tried one particular project that seemed easy enough, but leading me to go back a few (a bunch of) steps, and rethink my whole approach.

I bet almost everyone in tech (and a lot of people beyond) heard of AlphaGo, Deepmind’s program to play the game of Go beyond what humans can do. That has evolved, and the current state of the art is Alpha Zero, which takes the approach of starting from scratch, just the rules of the game, and applying self-play, can master games like Go to an even higher level than the previous programmatic champion after relatively brief training (and beating AlphaGo and it’s successor AlphaGo Zero), but also apply to other games (such as chess and shogi). AlphaZero’s self-learning (and unsupervised learning in general) fascinates me, and I was excited to see that someone published their open source AlphaZero implementation: alpha-zero-general. That project applies a smaller version of AlphaZero to a number of games, such as Othello, Tic-tac-toe, Connect4, Gobang. My plan was to learn by adding some features and training some models for some of the games (learn by doing). That sounds much easier to say than to do, and unravelled pretty quickly (but probably not as quickly as it should have been).

Continue reading “How not to start with machine learning”
Categories
Maker Programming

Home Automation Mix-and-Match

This week I got a Wio Link prototype from a friend at Seeed Studio. It is an ESP8266-based little Internet of Things board with 6 Grove connectors for easy device connectivity, wifi networking, and controlled over an app & the Internet. For a quick project I wanted to hook it up with Home Assistant, an open source home automation platform that I read a lot about lately. The main focus was to have a first impression of both parts, and build up some experience for future, more serious projects.

The target solution: light up an LED if a particular person is at home location. Sort of a basic alarm system, though notice that the location of the LED was not mentioned – it can actually be anywhere in the world, as long as there’s Internet connectivity.

I’ve used the Wio Link, a Grove LED light, an Olimex OLinuXino Lime2 board running ArchLinux for the server, and a Buffalo router with DD-WRT system installed.

Wio Link

Wio Link was introduced in Seeed’s Kickstarter campaign, where they have raised more than 8x of their original target. It looks like a neat little board, and was happy to try out when I got my hands on one.

Their wiki page has quite a bit of information, so it was easy to get started. Connect to power, hold down the configure button till the LED lights up in a “breathing” pattern, connect through their Wio Link app, set up the wireless network settings and so on. Once connected, can define what kind of devices are attached to the board, and it looks like most of Grove devices are represented there. I only had a Grove LED at hand, so added it (“Generic Digital Output”), updated it, which created a new firmware and pushed onto the device.

Wio Link setup process (left to right): add device, update firmware, check status
Wio Link setup process (left to right): add device, update firmware, check status

The first update took a couple of minutes, but it’s pretty straightforward. The device then also has an API link, which brings up a web page with all the options to query, control, and reset the attached accessories (in my case that’s the one digital output).

Continue reading “Home Automation Mix-and-Match”