Books

Readingwise this is a year with maybe the slimmest picking since I’ve started to record stuff. It’s really not fully about quantity but quality, just also I cannot recall too many of the things read. Also the balance is shifting from almost totally literature to partially non-fiction, as the effect of learning-at-work kicks in. So in this sense, maybe the most memorable read was Crucial Conversations that probably saved my bacon multiple times, so would totally recommend for reading and re-reading.

Science

Interestingly a sort-of mid-life-crisis seems to have kicked in for me recently, and it’s not about getting a motorcycle or whatnot, rather than digging into physics as I’ve learned but forgotten, or never fully learned at all. So here it goes, can we learn some quantum field theory, just for the fun of it? Starting all the way back at maths, classical, then quantum physics further than I’ve been back in uni and grad school.

This whole thing was kicked off by PBS Spacetime, a Youtube channel that I can only recommend. Enough questions raised from there to dig deeper into books, course material online, etc… It’s a bit tougher since I won’t be using it day to day, but why not? Interdisciplinary stuff are really my cup of tea, so there it goes.

Working with leverage

This year at work, the theme really was for me to work through others rather than just an individual contributor. This software engineering leadership, engineering management, software architect needs were definitely quite tough, and there’s no end to the learning either. Building the right kind of technical knowledge to help other people grow, rather than just do what I’m myself are interested continues to be quite a frontier to explore, and makes me uncomfortable probably in a way that’s needed.

Cats

This year we took in a little cat, which was quite a change, not in least as I’m allergic to cats. But so far it seems to be working excellently, and he’s really a charmer. It does feel like too much responsibility quite often, but one has to just deal with it.

He’s still very tiny (4 months and counting, and been with us for 2), so it’s very interesting to see how the personality and skills are developing. And no need to hurry it, let kids be kids.

Gym

Been to the gym much more than any time before, that’s the advantage of having one right nearby, and enough peer pressure to actually show up. The first half of the year was better, though. After catching Covid in the autumn, I feel like I slacked a bit. It was interesting see how much change exercise can effect in the mid-term, or even on the short term, when one has the habit. This is something to definitely explore more with more curiosity.

Mental health

The whole pandemic brought out the topic of mental health quite a bit for a lot of people who might not have thought about it much or enough. And I was one of those people, just going with the flow. It’s now 3 years in, and definitely feels like it’s hard to overstate the the importance of taking care of the one thing that people actually have (while they have it), their mind. So far it’s a lot of hit and miss, and it feels like a puzzle when one has all the pieces “just” need to be able to fit them together. Taking one’s own advice, or doing the things that one knows one should be doing is 99% of the work and effort. So just keep beginning again, when inevitable fail, and continue making progress.

There’s a lot of little things, of course, that happened this year, and this above seems to be both summarising the important themes to a very large extent, as well as filtering things through the lens of availability bias, either happening more recently or more top of mind at this year end. Still, a bite at a time, a step at a time.

Happy New Year, everyone!

The post A year in review: 2023 appeared first on ClickedyClick.

1. A banking website with CAPTCHAs and no programmatic access
2. An email received each day with an password-protected PDF containing the last day’s transactions in a table

Neither of these are fully appetizing to tackle, but both are similar to bits that I do at #dayjob, but 2. was a bit closer to what I’ve been doing recently, so that’s where I landed. That is:

• Forward the received email (the email provider does it)
• Receive it in some compute environment
• Decrypt the PDF
• Extract the transaction data table
• Clean and process the tabular data
• Put raw in some data warehouse
• Transform data to get the right aggregation
• …
• Literally profit?

I was surprised how quick this actually worked out in the end (if “half a weekend” is quick), and indeed this can be a first piece of a “personal finance data warehouse”.

Technical implementation

I wanted to have the final setup run in “The Cloud”, as that’s one less thing to worry about. The most obvious arrangement, based on past experiences was combing AWS Simple Email Service (SES) to receive an email, and a Lambda to run serverless processing. On the data warehouse side the real obvious choice is GCP’s BigQuery, however, so I looked into what would be a similar arrangement for the processing pieces if I want to put everything into a single cloud provider.

After some docs diving the most natural arrangement on GCP seemed to be quite different: an App Engine deployment with Mail API enabled. This gives a receiving domain name (@[Cloud-Project-ID].appspotmail.com) , and every email sent there is just passed to the server that is running in App Engine. This seemed pretty simple! App Engine also has a free tier, though that comes with pretty small memory limits, which features in this story too.

The final result of the server part is shared on GitHub, and should be easy to reuse or extend.

PDF processing

Getting the attachments out of the email was pretty straightforward with the Mail API, so the first heavier task was opening the encrypted PDF and getting the table out of it. Opening PDFs are quite common, but the table extraction was a bit of a journey.

False try

First I was searching around (as anyone else does) for someone else’s rundown of the options, as an example. From there I honed in on pikepdf to open the password-protected files, an tabula-py which seemed handy to extract tables right into Pandas DataFrames. One subtlety was that tabula-py is just a wrapper around tabula-java to do the extraction, and needs a Java environment installed. The free tier of App Engine uses their standard environment where all I have is my code and “requirements.txt” to install my python dependencies, so it’s obvious how would I get Java into the deployment correctly.

Enter the scene install-jdk which can install the Java environment at runtime. That was sufficiently crazy hack to actually work, and it did work. Or so it seemed, since the data was processed and showing up in BigQuery, when I’ve sent test emails into the system.

Upon closer inspection, though, there were loads of duplicate lines. Between signing off in the evening, and checking it in the morning, I had bunches of them, and were still coming in…

I should have checked the logs earlier, because once dig in, there were bunches of “server errors” listed that didn’t connect to any programming errors that I might have made, rather than (here comes the epiphany) instances being killed for being out of memory / blowing their memory budget (of 256MB for the free tier). Thus what happened is:

• the Java run of tabula was just using too much memory while processing the PDFs
• it finished processing and like loaded the data but it takes a bit of time
• GCP catches up and kills the instance while that is still going on, and reports to the Mail API that the email hasn’t been properly handled (server error during that process)
• Whatever is handling the incoming email queue in GCP will just just keep the data and retries later
• The cycle repeats…

This didn’t seem very helpful and the repeat emails were piling up in whatever (opaque, to me) system GCP has, so needed a quick replace of tabulate with something lighter…

Worse is better and actually good

Going down the list of recommended libraries, next I looked at camelot-py which looks great, but needs OpenCV on the machine to do its work, so back to the “how to install OS packages on Standard AppEngine?” question. For some extra inspiration I was looking at camelot’s comparison with other similar tools page and it was a bit disappointing (though not surprising) that pretty much every other library is “worse” on various PDFs compared to camelot. Just for kicks I did try some out, and pdfplumber actually delivered:

• it does actually work on the example PDFs I had from previous bank emails
• nothing else beside pip install
• it can actually handle decrypting the PDF as well, so helper libraries can be dropped
• the extracted data is in Python tables, but it’s just an extra line to get DataFrames, so no sweat
• The extracted data was actually better quality than tabula’s, so had to do fewer cleanup steps!

This was a pure win, and indeed it’s worth looking stuff that works with the data at hand, not ignoring the edge cases, but also not overly emphasizing being able to do “everything” when there’s a clear target of what “thing” needs to work. (Potential technical debt considered too).

Data transformations and visibility

Now the data sits in BigQuery properly:

The raw transaction data loaded into BigQuery was the first step, but still need to answer the question: in this billing period, how much have I spent?

Not being a data analyst (or not yet?:), this took a bit of figuring out. As other novices share their bit of “clever code” when it’s actually trivial to the experts, I’m sharing here the bit of SQL queries in a similar “that was fun to figure out, wasn’t it?” way. I’m sure it can be much improved, but it’s a good reminder for myself as well.

Given that my billing period starts on the 23rd of the month, get the aggregated value of transactions for each billing period:

WITH
  Aggregated AS (
  SELECT
    DATE(TransactionDate, 'Asia/Taipei') AS day,
    TransactionAmountNTD
  FROM
    `personal-data-warehouse.finance.huanan` ),
  calendar AS (
  SELECT
    day,
    -- Find the last day before the new interval
    DATE_SUB(
      DATE_ADD(
        day,
        INTERVAL 1 Month),
      INTERVAL 1 DAY
    ) AS endday
  FROM
    UNNEST (
      GENERATE_DATE_ARRAY(
        -- Start date in the past before any data,
        -- on the right day of the month for
        -- the billing cycle.
        '2022-05-23', 
        CURRENT_DATE('Asia/Taipei'),
        INTERVAL 1 Month
      )
    ) AS day
) SELECT
  SUM(TransactionAmountNTD) AS `MonthlyTransactions`,
  COUNT(*) AS `TransactionCount`,
  EXTRACT(Year FROM c.day) AS `Year`,
  EXTRACT(Month FROM c.day) AS `Month`,
  FORMAT('%d-%02d', EXTRACT(Year
    FROM
      c.day), EXTRACT(Month
    FROM
      c.day)
  ) AS `Interval`
FROM
  calendar AS c
JOIN
  Aggregated AS a
ON
  a.day BETWEEN c.day AND c.endday
GROUP BY
  c.day

Good stuff on the date array and joining with a “between” statement, those are the main TIL. They also already came up at #dayjob, which was very satisfying.

From here the data I surface in a connected Google Sheet which is pretty practical, though leaves the “being notified when I approach/reach X” out, but that’s fine for now.

Testing and getting to “production”

One good thing about personal projects is that I can make them as “good” as I want to (or as “bad”, of course), which usually results in an unhealthy amount of tweaking, trying out various best practices to see if they work, and so on. Here I really wanted to get the system well tested, for example, which turned out to take loads more time than actually writing the original service. Actually, there’s nothing surprising about that for software engineering professionals, but still can catch people off-guard.

Here the tricky parts came from two areas: FastAPI settings and cloud service integrations.

The former is always a bit of an issue, depending on how the code uses the settings (whether things can be patched well at testing time), but here I also used a trick for the server to pull the PDF decryption key from Secret Manager, so I don’t have to deploy environment files, nor keep settings like that in version control, etc… But this meant a trickier flow of getting the FastAPI testing client up in a way that it worked without it talking to the cloud backends (and stalling, and failing…). Nothing that some good mocking cannot solve (says the person with hindsight).

For the cloud services part it meant mocking BigQuery connections, so that the test can actually pretend to “receive” an email all the way looking at the “database” and see the right information being there. Under the hood I’m using pandas-gbq, and thus it was interesting to look under the hood for their tests, borrowing some of them. Took a bit more time, but that’s working pretty well now. Still need to do some extra bits and pieces to do cover more of the workflow, but I’m already more confident about things working. Also, all this will be very useful on other projects that are interacting with BigQuery in any way (not just through Pandas).

Big evergreen lesson on testing: you have to write your code to be testable. Lots of code out there is not even not tested, but it’s even extremely difficult to actually test. This needs remembering in every development. Also, test writing never really stops, there’s always more thing to test for. And finally, can always try more advanced testing, such as using automated test case generation (e.g with hypothesis), and fuzz testing (e.g. with pythonfuzz). The next frontier, right after I’ve implemented the currently skipped tests. And finally, remember that code coverage is not case coverage, so the goals should be maximizing the latter, while the former is just a potential proxy for it.

Future outlook

It would be nice to take this idea of financial data analysis further and add some actual dashboard (say deploying Superset somewhere which is excellent for this). It would help to get more information into the system as well, though, currently it’s very sparse. That would mean adding other financial sources, maybe if finding an API in the end, or doing a bit of “pragmatic execution” and do a CAPTCHA bypass (since I quickly checked that my credit card provider’s CAPTCHA is completely readable by Tesseract, for example, so I could likely scrape things there if I really wanted.

I’m not holding my breath for having something like UK’s Open Banking here which enables apps like Emma so all this is accessible for people who don’t want to code. But where’s the fun in that (for me)? :) (In fact there’s a lot of fun in open access APIs, so this would be the real way of doing it…)

Finally, it’s good to remember how easy it is to corrupt “production” data sets, but also with the right tools (like snapshots), some of that pressure can be less. There are always bugs, the question is how to mitigate their effect.

The post A personal finance data pipeline project appeared first on ClickedyClick.

I’m glad I have managed go all five weeks without caffeine – no coffee, no tea, no dark chocolate to be on the “safe” side… That’s all nice, but looking at the purpose of the decaf period, did it make any material difference? And was it any different compared to the previous two times when I did this? The result is not totally clear cut to me, it definitely was much clearer in previous times. No experiment running for five long weeks in real life can really just change one parameter (me with/without caffeine in this case), while keeping everything else the same. Thus changes or lack of change is harder to interpret.

One main target was reducing “jumpiness”, as it seemed to be pretty well correlated to coffee consumption. This didn’t seem to happen this time very much, or rather it feels like the “jumpiness-cycle” is more connected to: a) I’m like that by nature, b) how stressful work was at any given point, c) how much I’ve meditated. Thus points at just being more aware of being jumpy in the future, and deploy a variety of coping mechanisms, instead of a substantial tool like this.

Another target was improving on my sleep. That did happen, definitely crashing by 23:00-23:30 these days, instead of staying up later. Also waking up better in the morning, though not much earlier, if at all, compared to my benchmark 8 hours of sleep. I guess there’s a lot to improve on my sleep quality, but there was some positive effect at least.

This time the substitutes I had were different. The main one was switching to infusions (fruit teas), and those seemed to have worked out well, I think I keep some of that up later. The other substitute was decaf coffee. That felt a bit stranger, drinking decaf knowingly made me feel a bit empty inside (why would someone actively choose a “faux” version of something like this?), but had the right effect of me being able to use the cafés’ social aspect while not breaking the fast. Definitely good choice when getting a “coffee” late in the afternoon and in the evening, and not want to be influenced. The taboo is gone, so I might use this again, but likely not without shaking my head at myself inwardly. Still learned some new things, such as when getting a drink one day, I’ve thought why not go one step further and try replacing the normal milk with something else. Even if asking “decaf cappuccino with almond milk” made me almost laugh out loud because this is one of the most hipster things I’ve done, the result was pretty good, I’m a fan of almond otherwise and now I know that it is indeed good with a hot drink. Also an affirmation to keep experimenting.

I was surprised also, that when I had any cravings, it was for a good strong English breakfast tea, instead of a coffee. Since cravings point at weaknesses, now I’m more wary of the breakfast tea…

There are other health effects that I can’t really measure myself, but would have been nice to know, such as my iron levels. I get some good feedback on that when I go donating blood (and the last few times my level was lower than I’m used to), but there’s still a few weeks till next time, not sure how clear it will be any difference from these weeks.

All in all, I’m glad I went ahead with this decaf period, more mindful than last times. While the results are mild at best, that’s a signal too. Now let’s see how the old habits will come back or some things have changed more permanently.

The post Five weeks of no-caffeine appeared first on ClickedyClick.

The first time it was a very special experience, many aspects really stayed with me. The most important memory I carry over is the expected way the caffeine withdrawal should play out, based on this first time. The first week is kinda easy. The second is harder. The third was really rough, I reverted to pretty much to be a manchild, (figuratively) banging on the table and shouting “I am a grownup, I can have coffee whenever I want!” Then the last week was very good, much better mood, much more balanced physical functions too. It was bliss. Was almost strange to stay in that state for only a week, and resuming having coffee .

The second time was less memorable but had its own lessons. The rough phase happened much earlier and was less rough. Also, I think I had a hard time working around this no-caffeine limitation when going out. Seemed like every drink that I could have either had caffeine in it, or was full of sugar, or was just not worth the price, such as a single tea bag in a cup of water the same price as cappuccino in many cafés? Still survived, and was worth it, even if this second time was less transformative than the first.

This third time I started again because of the warning signs listed earlier were unmistakable. Today’s day 6 and so far it went down as such :

• Day 1: really big coffee craving, feeling almost useless the whole day, crashing at 10pm (hours before previous sleep time).
• Day 2: quite refreshed after long sleep, almost no craving, this should be eeeeeasy.
• Day 4: cravings slowly creep back, for example when others have coffee around me, just by smelling them. Starting to have enough of the teas I’m having.
• Day 6: the “child tantrums” are lurking, I catch myself with similar thoughts already. On the other hand, waking up better in the morning, and not feeling especially tired through the day, much more even energy. I, however, catch myself thinking “this goes well so far, I wonder what should be the first drink to have when I’m finished, to make it special?” This is really my brain subverting the effort.

I’ve also tried to do a little bit more thinking and checking when starting. For the hot beverage cravings, I choose more rooibos (or red bush) teas or flower/fruit teas if need to. I have to say, I couldn’t really stand the flavour of rooibos tea before, but now I’m acclimatized to it. Don’t think I would ever choose over all the wonderful (but caffeinated) teas of Taiwan if I had a choice, but I think it will carry me over. There are other drinks like MILO that I can fall back to. Curiously enough, decaf coffee still has caffeine in it (just much less), I’m still on the fence whether having that is acceptable within the rules of this game, but the jury is still out on that one…

Another change is cutting down on chocolate as well (or actually skipping it as much as possible), as that has caffeine as well. I guess that’s a win on the sugar side too, but have to watch snacks in general. Having said that, I do feel that when the craving moods hit, a piece of dark chocolate would sort me out – have not tried (as playing by my own rules), but I’m pretty certain it would work.

This time to match some plans that I already had in place, I choose not “1 month”, but a 5-week interval for this detox. There’s a clear ending (with a trip at the end of the 5 weeks) that makes the whole experiment self-contained, which feels nice. Also, when I get to the bliss-phase, hopefully, I should have a bit more time to enjoy it.

In the meantime, hope everyone else can enjoy their choice of drinks , regardless of what silly things I myself am trying to do…

The post Starting on a no-caffeine month appeared first on ClickedyClick.

No Man’s Land

(2016 December) I was reading a few volumes of Pinter’s collected plays, and was feeling very envious of New York, that they had Patrick Stewart and Ian McKellen putting a dual production of Waiting for Godot and No Man’s Land on stage there. But then I was lucky enough to catch  No Man’s Land in London, and that was a really intense kick-off for my theatre season.

Theatre of the absurd is maybe my favorite, and this one makes a really good watch. The play is not perfect, though, the long monologue towards the end made me switch off, regardless how good the delivery was. But it was superb setup, superb acting (though Sir Ian had the better part), and very memorable. I think it’s a pretty good introduction to British culture as well, I’m definitely drinking much more since (alcoholism is one of the central elements).

Art

(2016 December) For a long time I wanted to check out the Old Vic. The second play of the season was Art by Yasmina Reza (whose God of Carnage I’ve also seen many-many years ago in London), a French playwright who seem to be extremely playful and have really interesting insights into human nature.

The characters were great, Rufus Sewell and and Paul Ritter are really bringing out the worst (and in some way the best) of each other. Great stage and visual design, that will become the theme of this year too, subtle things that add up to a lot more in their overall effect. I was really amazed how much they could make simple colours, simple spaces work.

The play is also very enjoyable, examining the ways friendships work. I’d highly recommend to anyone who’s wondering about adult friendships, I took home some interesting thoughts about dealing with the (perceived) shortcomings of friends. (That’s how plays should be, making us reflect about our world, right?) It seem the play’s also online in PDF form, which is pretty fortunate…

Woolf Works

(2017 January) After drama, the new year brought a bit of exploration and expanding my comfort zone by ballet and dance at the Royal Opera House. The first one was Woolf Works, a three-set piece by Wayne McGregor inspired by the writing and life of Virginia Woolf.

Not sure how much I’ve missed by not having read Virginia Woolf myself (shame), but this performance was pretty intense and creative on its own. Again amazing stage and visual design, how sets, lighting (including lasers), projectors were used. Great choreography (saying it as someone who’s not familiar with dance theatre in general), and creative use of dancers that created powerful emotional effects – for example in the last part that deals with her drowning, how the all the other dancers turn out to represent the waves of the sea in subtle but intense way that one realizes when it’s pretty much “too late”.

Sleeping Beauty

(2017 February) After modern theatre and dance, switched back to classic ballet. It was quite a switch, watching Sleeping Beauty, again at the Royal Opera House.

The feeling of watching what was actually the popular taste of the 19th century – I feel like I was taken back quite a bit. It’s grandiose, it’s different, it’s somewhat interesting, but also a lot of “filler” and “what’s the point” sections for my 20th/21st century taste. But when I could get out of that mindset it was nice… They still managed to pull of some very 21st century visual effects, the way they represented the forest in Act II, and how they created an immersive lake & willows environment was a sight to behold…

Hamlet

(2017 March) Already had tickets for the following play, Rosencrantz and Guildenstern are Dead, which is a twisted view on Hamlet, so I was very happy to find that there was a Hamlet playing in town, a very new one. Except, there were no tickets, since everyone wanted to check out Andrew Scott. Thus as a first, I was queuing for a day ticket at the Almeida Theatre on a grey Thursday morning, the one day when I could try to get a ticket before the following play, and managed, by a hair….

It was totally worth queuing for, a superb modern take, where most of the shock and awe is still coming from the original play, but plenty of opportunities for the creators and actors to shine through. And of course, Hamlet will leave you thinking as well, a lot to discuss about the world after one leave the theatre. This production was very well received in London, and continued on in another, larger theatre afterwards for quite a while.

Rosencrantz and Guildenstern are Dead

(2017 March) Following Hamlet, went to see the characters who follow Hamlet around, Rosencrantz and Guildenstern are Dead by Tom Stoppard, again in the Old Vic.

It’s a really sharp, witty, and in it’s sarcasm very insightful play indeed. Daniel Radcliffe was very good (and was interesting to see him in person), as was Joshua McGuire who just never stopped, but somehow quite a bit of the attention was stolen by David Haig as the Player. It was definitely a highlight and a play that I’d love to see a few more time…. In the meantime it seems like it’s also available on the (grey) net to read.

The Human Seasons / After the Rain / Flight Pattern

(2017 March) Next back to the Royal Opera House for a mixed ballet program of three independent pieces. The style probably couldn’t be more different between them, which made it a great eye opener evening.

The highlight for me was Flight Pattern by Crystal Pite, which I think was premiered as part of this programme. It deals with the plight of refugees, and it felt like the most developed, most thoughtful choreography I’ve seen so far. A lot of dancer but they were superbly working together, and could create amazing atmospheres just by motion (I can’t even describe…) One piece this year that left me quite literally breathless and aching.

The Glass Menagerie

(2017 Apr) Back to drama with The Glass Menagerie by Tennessee Williams. If the previous plays were superbly English and French, this is so clearly American.

Maybe falling on the side of the melodramatic a bit more than I’ve expected (since I didn’t expect too much, haven’t really read anything from Tennessee Williams yet). The characters and their execution was pretty good, and here too they managed to bring in some great visual design, how the flat level of the stage could act convincingly as a top-story apartment with all the space (and stars, and Moon) around it.

Travesties

(2017 April) Another Tom Stoppard play, Travesties, that worked out just as well, or in some sense even better than Rosencrantz… I kinda wish I’ve checked out the play beforehand, since as Rosencrantz is a swipe at Hamlet, Travesties incorporates the Importance of Being Earnest, which, I’m ashamed to say, haven’t read before. I guess I would have gotten more of the jokes and references if I had.

As opposed to Hamlet, which I’ve caught before it went big, Travesties was in its extended run after being moved from its original, smaller theatre. Looking at the pictures from the earlier run, there were some changes, and it feels like that was for the better. So I guess there’s no universal lesson whether it’s better to catch a play in its early or in its extended stage (I wish I could say “well, see it in both stages”, which would be my ideal world:).

It’s a smart and entertaining take on what (and who) is real, similar to the Oscar Wilde play that it works with. Not as deep as Rosencrantz, probably, but maybe has more breadth. Would be great to see the actors from this production in other plays too on one hand; as well as learning a bit more about the early 20th century expat intellectual world. It seems mighty interesting.

Obsession

(2017 May) Obsession was a theatre adaptation of Luchino Visconti’s 1943 movie, starring among others Jude Law and Halina Reijn. Barbican is quite a different theatre compared to the other ones so far, a lot more spacious, lot more experimental.

There was a lot of play around making certain parts very cinematic in a theatrical environment, with sound effects, lighting, a treadmill for a very intense run scene and so on, and it works well, though also makes the play somehow a collection of views with quite sharp transitions (not sure if that’s how the film was). The personal drama is in center, and the Italian style complex (or rather complicated) relationships do come to life to a very painful extent… Relationships and the mistakes made, but which one of us doesn’t?

The Mentor

(2017 July) The last of the season was again a new nationality, a peek into the German psyche in The Mentor by Daniel Kehlmann. F. Murray Abraham is the star of the show, and this is also an extended run, originally put on stage in Bath.

Very modern, and playful while does provoke some thoughts self-examination. It’s fun to see that the characters thinking are clearly different from the British, French, American, and Italian approach to the world as in the other plays. It was a pretty short one, though, maybe it was more of a sketch, though I’m guessing it could be made more impactful. At the moment it’s more entertaining though with a pretty big dose of cringe.

Epilogue

I’m really happy to have caught up a bit with theatre, because unlike movies, once you miss a performance, there’s not going to be a repeat (though I’m surprised that theatres are not recording more of the plays and releasing them, I wonder why, copyright?) There’s never enough, though, there are venues to check (looking forward going to the National Theatre again, haven’t been to the Globe before, and know very little about the Off-West end theatres…), great actors to see on stage, and experience plays that I’ve read & admire.

From tips and tricks this year: the back-of-theatre seats are usually pretty good already, no need to really fight (pay an arm-and-a-leg) for more forward seats – though I can see how it can make a decent difference in the experience;  I do not read the reviews before the play, and very rarely afterwards, that usually do not add much value and I just end up second-guessing my choices of plays; for the best productions by the time you hear about them they are already nearly or entirely sold out, so prepare for some disappointments, but also when it’s worth it try to hustle for a day ticket;

For the next season I have a few things booked, hopefully the review next year will be just as rave, or hopefully even more.

The post My London Theatre Season Highlights, 2016-17 Edition appeared first on ClickedyClick.

In this rather graphic post I try to summarize the process of getting a MOICA card as a foreigner in Taiwan, setting it up, and some (opinionated) lessons to learn from it.

The Process

The process of getting and setting up the card is outlined in this leaflet that I’ve picked up.

Though – not surprisingly – in practice everything is a lot more complicated than these 8 bulletpoints.

There’s also a Chinese version of this leaflet, just for the record.

Registration

As it is included in the National Immigration Agency’s (NIA) own announcement, the process is kicked off by visiting one of the service centers (here’s a list, but people generally know which center they belong to, they have to), and requesting the card. Even though I was applying (in the Taipei center) on one of the first days when it was available, there was no surprise or difficulty. Picked up a number, the at the counter forms were filled out for me all digitally, and then got a receipt like this (sensitive info blacked out here):

It included my application number, name, email, phone number, and a registration code (the last blacked out line) which was needed for the online payment. This is because, for some reason, the bill cannot be settled at the time of the application, but needed to be done online.

Payment

The card is only sent out to me after the online payment, and if no cash is received in 14 days, the application is cancelled. Thus I wanted to settle things very quickly. Originally I thought, based on the info I’ve received at the NIA that I will receive an email or other electronic communication with further info on how to pay. Instead, for the next 2-3 days I got a daily nudge message that I need to pay, but no information on how.

Visiting the MOICA website for foreigners (“aliens”) does not make me smarter, it looks like this below, with a couple of options on the side, but no “PAY HERE” or similar.

After one and a half weeks the time was running out for the application, so gave in and called the 0800-080-117 number from the message for help. They did help: see the funny looking Chinese-only banner with two cartoon figures? Have to click that, and then go onto this Chinese-only interface to continue.

Using the code from the receipt and logging in, there was an online payment interface. I really should have taken a screenshot, but so taken aback by it, that I forgot. It’s a HTTPS page (for a change), with big (Chinese-only) announcement that “because the site uses SSL your data is completely safe and nothing to worry about”, plus Chrome giving me a big security warning that the website’s SSL settings are vulnerable. Who do you trust in this situation? Rhetorical question: of course Chrome but still pay the government…

At least their messaging service works a lot better than their website, within seconds I got dual language notification that the credit card payment went through fine.

There’s not much to do after this, but wait till the postman arrives with the card.

Card Activation

The card arrived with two page of instructions, sticked in the middle, highlighting the security features of the card.

On a side note, not sure how all these visual security features (which likely serve paper bills well) help with computer security, since the only important part of that card is the chip inside, but I digress.

The minimum system requirements are Windows XP SP3 and IE 6.0. I’m using Linux, but fortunately have an XP in VirtualBox, and Internet Explorer on it, so might just work. The instructions said to go to the MOICA website again, and activate the card there. Except, there was no activation-related menu there at all, and the instructions remain tight-lipped. I was hoping that the back-side of the instructions, that are in Chinese, will contain some clues, though it turns out that they are even less useful, if that’s possible.

It’s time for another phone call to the 0800 number, and turns out that the “helper kids” are the ones we need again. Clicking that Chinese-only banner on the otherwise dual-language website brings up the page that I used earlier, but now instead of logging in, have to click the last menu point on the left, saying 憑證IC卡開卡. That takes me back to a page that looks very similar to the original MOICA page – except the menu is full. Indeed, the first page only contains “Commonly Used Functions” in the menu and have to do this extra trip to get a complete list of available “Certificate Procedure“[sic].

Here the activation can begin in earnest. First steps are:

• the site prompting for the install of half-a-dozen plugins,
• there’s an external program for the card reader smart card functions called HiCOS (get it from the download page)
• besides the card reader’s own drivers (I have an EasyATM Pocket Reader 111, seem to work well both in Linux and Windows)
• and a “gcms-plugin” linked from the activation page, what this does is unclear to me.

Most of these are also outlined (in Chinese) on one of the helper pages.

I did all this, loaded the card data (取得資料), filled out the page’s form, and submitted (the 開卡 button, “enable card”), but the process just hung there for about 5 minutes without any result. Tried more times that I probably should have…

One difficulty is that the page asks to “enter” subscriber code, and “set” PIN code: I didn’t know any subscriber code (probably meaning “username”), and “enter” is not “set” but who knows how the translations work on these sites?

On the page there was a link to help if things don’t work, which takes us to the MOICA blog, lo and behold:

This doesn’t strike me with a lot of confidence. Not just that it’s all in Chinese, the post is more than a year old, the color scheme is going crazy, and basically doesn’t contain any new info. Mainly it’s about using an external site (together with advertisements) for an important information like this.

Further down the page there was another link to even more help on this blog to how to set things up for the MOICA site to work with the smart card reader.

The info is in Chinese, but more or less says this:

• Enable compatibility mode in IE for moica.nat.gov.tw and  61.60.9.42,
• In Internet Options / Security / Trusted Sites / Sites disable “require server verification (https:) for all sites in this zone”, and add moica.nat.gov.tw and  61.60.9.42 to the trusted sites
• Internet Options / Security / Trusted Sites / Custom Level enable everything releated to ActiveX (in my opinion, set to “Prompt” instead of “Enable” whenever possible)
• Clear cookies and browser data and try afresh

This is pretty horrifying again, but what to do, have to make that card work somehow. The IP in that list is actually the IP where the form is served, within an iframe on the MOICA site… No security, open everything if you want to use the card.

The final comment on the page says, that if the activation form still hangs, then something is still not set up correctly. Of course my form was still hanging, but I was quite confident, that I haven’t missed any part of the setup. It’s time change strategy: to put myself into the shoes of the developer of a system like this. Imagining it as someone is making a site to technical specs only but not for usability, I was questioning what functions I might need. Looking through the menu, there is a “How to change subscriber code” page – and it does not ask for existing subscriber code, only other information! So it is more like resetting a forgotten code/password, this might be what I need!

On that page I’ve added my Alien Residency Certificate number, date of birth, and my MOICA card’s number as requested, and got a brand new subscriber code.

Putting this new subscriber code into the activation page, setting a PIN and finished in a second like a charm: Success! The activation was also verified by going on the Certificate Query page (using the 64-digit certificate serial number from the Activation page that can load the card data):

The lesson seem to be, unless I’ve missed something, that a subscriber code needs to be created first (not mentioned in the help), and the activation page hangs on all kinds of issues such as missing/wrong subscriber number as well, not just when the technical setup is incorrect.

Now what can I use this card for? I’ve seen some articles saying that there are 3000 different services to be used with, but I guess that’s only sort of correct. The FAQ says:

[…] electronic highway drivers services, tax filing, internet application system of the Labor Insurance Bureau, electronic gateway of the Land Administration, household registration internet applications, ChungHwa Telecom mobile phone (citywide) query, company online application system of the Ministry of Economic Affairs, public safety inspection system for buildings, internet customs clearance system for air cargoes, 1988 tax information instant messaging, and so on.

Foreigners cannot do tax filing for the 2015 tax year (just being filed now till the end of this month), but should be good from next year (I’m using my stock trading file-based certificate so I’m good to file online anyways). Other uses I guess need some exploration.

Endnotes

Tech

I haven’t really used smart cards for anything so far, the only other one is the National Health Insurance card earlier this month, that was a tiny bit more straightforward, and sort of immediately useful.

Under Linux I can use pcsc-tools and the CCID hardware driver for my reader to get some info about the card’s hardware, for what it’s worth:

Thu May 26 15:23:24 2016
Reader 0: Generic USB2.0-CRW [Smart Card Reader Interface] (20070818000000000) 00 00
  Card state: Card inserted, 
  ATR: 3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5

ATR: 3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5
+ TS = 3B --> Direct Convention
+ T0 = B8, Y(1): 1011, K: 8 (historical bytes)
  TA(1) = 13 --> Fi=372, Di=4, 93 cycles/ETU
    43010 bits/s at 4 MHz, fMax for Fi = 5 MHz => 53763 bits/s
  TB(1) = 00 --> VPP is not electrically connected
  TD(1) = 81 --> Y(i+1) = 1000, Protocol T = 1 
-----
  TD(2) = 31 --> Y(i+1) = 0011, Protocol T = 1 
-----
  TA(3) = FA --> IFSC: 250
  TB(3) = 52 --> Block Waiting Integer: 5 - Character Waiting Integer: 2
+ Historical bytes: 43 48 54 4D 4F 49 43 41
  Category indicator byte: 43 (proprietary format)
+ TCK = A5 (correct checksum)

Possibly identified card (using /home/user/.cache/smartcard_list.txt):
3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5
	citizen digital certificate (PKI)
	http://moica.nat.gov.tw/

Regarding the websites and web services in general I see more issues than good sides:

Chinese and English are used in a very strange way. English translation is often uninformative (at best, misleading at worst), or missing. Sometimes resetting a form changes all the labels into Chinese, the relevant FAQs are often in Chinese…. All this wouldn’t be a problem, if it wasn’t advertised to be usable in English too. So far it’s about a 20% solution. Getting people to translate, doing consistent writing, not having text in images (so copy-paste translation can work as a last resort), and just thinking about the users in general would go a long way.

Paper documentation and blog that contains help and advice is out of date and incomplete. The website has changed since the screenshots, IE has changed since the screenshots, not all steps are described, some technical info is not presented in enough detail to make it work easily. This is again I think an issue of low manpower for maintenance and people not putting themselves into the user’s shoes when writing documentation (that’s a common issue in all tech).

The site’s front page showing Commonly Used Functions only, and no apparent way to see all functions. That’s just a big usability issue (people needing to call to use the website), that should come up if they check what people call them about. So I guess there’s no monitoring and review of support calls or no feedback to the site design.

A big no-no is not having SSL enabled everywhere, it would be so easy to do. Chunghwa Telecom that they used for some of the smart card software development is the local Certificate Authority and the government has its own too, can’t imagine why there isn’t security set up. There’s even Let’s Encrypt for last resort now, so no excuses. Those pages that have SSL setup seems to be done by people who are not familiar with it, or not maintaining it properly. It’s a big misconception that “if HTTPS is in the URL, then the communication is secure”. In the public sector good sysadmins don’t fall for this, so the government and public sector need to step up their efforts. Staying with this immature security thinking will just lead to more issues (such as the just hacked Chungwa Post).

Using IP address-only designation and iframes for the smart card functions is also baffles my mind. I cannot imagine legitimate technical reasons for that, would be happy to hear if anyone knows one. Right now it looks like an alpha-quality software.

Relying on ActiveX for all these functions obviously is the result of the age of the project but it is definitely not future proof. Win 10 and the latest IE drops ActiveX as well, so there are no up-to-date systems that could use this smart card functionality. How many people will keep an XP around just to use the government websites? (The answer is probably not many and still more than should be).

Alternatives

With all this technical background, I really feel that this is a dead-end solution, which might work for a while still but has no future. To have a future, it needs to break from ActiveX, and go cross-platform too. Probably would need to break from the smart card format as well (at least as it is now). I’m in no way an expert in this area, these are some ideas based on my experience so far.

If trying to keep such certificate-based services around, then there has to be some serious research put into either how to use smart cards better (maybe based on OpenSC?), or change hardware. I’m thinking something like Yubikey could be an interesting model, especially with it’s Personal Identity Verification PIV support. Probably PIV and the related standards (that I have very tiny idea about so far) would be a viable path to look at.

Another alternative comes to my mind thinking about how bank logins work in some countries, using a Two-Factor Authentication (2FA) device that generates a one-time password for logging in. I see the model of this good in the sense of not having any hardware attached to the computer so it’s by default cross-platform and much less setup. Of course 2FA alone is surely not enough (eg. if electronic signature functions are used with MOICA), but the current model also just seem to be a 2FA (the factors are the card and the pin) login for the services. Maybe a mixed model, with simplicity in mind would be a good path.

Future

The idea of MOICA is fundamentally good, and pretty much inevitable in one form or another. The main thing would be thinking over the use cases and user experience a lot better. It is an interesting project, to a level that I almost wish I could work on this: technology development, QA, user & system testing, project development consulting, security enhancement, evaluation… It is unlikely to happen, so I really-really hope the Taiwan government (ie. the relevant ministries) have a decent sized and quality team working on this. Otherwise…. disaster and/or becoming irrelevant is also pretty much inevitable, only question is time.

Would love to hear if anyone else goes through this process and what do they use your cards for?

The post Taiwan Citizen Digital Certificate appeared first on ClickedyClick.

2015 – Results

This year have seen a long trip to Europe in February (Hungary, Prague, Vienna) which definitely opened my eyes to a lot of things that I probably should have understood back in high school (especially regarding history). On the other hand, I was a visitor back in my home country Hungary too, and tried a bunch of things that I have never tried when I lived there before, been there with different eyes (and would love to spend more time home again too).

The rest of the year was spent here in Taiwan, on small trips all around the north and east coast, and around Taipei City.  A lot can fit into day trips, and it is fun to mix popular places with off the beaten track.

I have finished a bunch of projects, probably more than any time recently. These include the Taiwan Bank SSL Status Monitoring service which is going strong, and looks like it had a real effect; then there’s the Formosa City Plan Overlays page which gave me all kinds of warm and fuzzy feelings; there are also a lot more hardware design projects than before (but still fewer than I wanted), including the PCIeDuino that still has a lot of unrealized potential; for writing there’s the Atomic Physics Apparatus Advent Calendar which I didn’t think I’ll be able to finish, but I’m really glad I did.

On the reading front, since I haven’t met my goals in the last couple of years (usually 52 books each year), I’ve set a lower goal at 30 this time. I reached that goal (here’s the list of books on Goodreads), which means I should increase the target to be more challenging. There are a lot of good books this year, though I think not one book dominated the year’s impressions. Progressing with the Culture series is good, Invitation to a Beheading, V. are both brilliant works. There were a lot of business intelligence books on the menu, and a couple of plays, which I’m very happy getting back into.

Watched  38 movies (here’s the list on IMDb), and they are overwhelmingly good, and added quite a bit to my world view. I’m being picky about them, and that does a lot of good. There are a lot more in reserves to watch some day than how many I actually see, and that’s a good imbalance, in Eco’s “Antilibrary” sense, and same for books too.

My hardware experiment with Tindie was sort of okay, though my store doesn’t have much traffic or sales at all, but was a good source of a few lessons (and not just on that I need a lot better execution and follow through).

On the side-projects front everything was quiet, probably too quiet. Many big things from before are kind of on ice, and even the ones that continue, such as working on the Taipei Hackerspace are on the back burner.

2016 – Outlook

An experiment that I’m going to try is that instead of new year resolutions, let’s have a theme for the year. Instead of setting priorities now, under the current circumstances, let’s have a tool in the shape of that theme, which helps me make choices along the way. I haven’t quite made up my mind on the theme yet (still have about 24 hours to do that), but my current candidate is giving back. Doesn’t matter that it is very much a cliché in general when one’s being asked about their priorities or values, and the answer has very little bearing on the real world. Here it will be in the background, with an active role, guiding a year’s worth of experimentation and exploration.

Other than that, I’ll try to keep my /now page up to date, as good tool to focus.

How your year’s been and how’s the next one looking up?

The post Year in Review and Preview 2015/2016 appeared first on ClickedyClick.

There are 26 maps in the collection, made by the US Army by flying over different parts of the island, and mostly I guess stitching together aerial photographs. The maps themselves were not that easy check in an image viewer, since there’s no context, zoom is clumsy, and have no idea where about half the places should be located. Instead, I thought it would be great to have them as an overlay on top of current maps and satellite imagery on Google Maps.

The result is Taiwan City Maps overlays, which does exactly that. Feel free to click the link and explore right now! In the rest of this post, I try to first show how that page was made, and also some history lessons I gained by making it.

How it was made

The Google Maps Ground Overlay API does exactly what I needed, so can’t claim much (almost any) innovation in the page, though I stumbled on it after a few other “Overlays”, as Google Maps has a couple, all working slightly differently.

Ground Overlay basically takes the following parameters:

• the map’s center,
• zoom level,
• an image,
• the image’s N/E/W/S bounds in geographical coordinates.
• whether to show satellite or maps view underneath

Even cooler, the opacity of the image can be adjusted, so the overlay and the base layer can be viewed together much easier!

In that list above, the trickiest part is of course figuring out the boundaries of the image – basically the way to overlay them – for multiple reasons. First, all the maps have borders with information, titles, legend, and so on. Here below is the Taipei city plan from 1944. I can’t really tell where the corner of the image should be geographically, if there’s nothing to compare to.

Then there’s also the problem of things have changed in the last ~70 years, so a lot of things are different on the maps, if I want to match them up. Lastly, the maps are not perfectly accurate either, they are stitched from multiple photographs of unknown quality.

Been trying to match the maps by hand for a while, and codings some tools to make that process easier – but it didn’t really get easier. I needed to get to get around all these issues, and have a reasonable, objectively good fit. Objectivity is a key, and math helps with that. What I ended up with doing a linear fit of geographical and image coordinates.

• look for landmarks that are reasonably the same back then and now
• note their geographical coordinates, and their pixel coordinates on the image
• note the image size
• finally: run a linear fit that gives back the geographical coordinates of the image edges that would best match the landmark data in both sources

In really ugly Python it would be something like this:

import numpy as np

# # For example the contents of "taoyuanfit.csv":
# # Landmark data: image vertical/horizontal (px), latitude/longitude
# 1677,2861,24.991962,121.322634
# 1597,524,24.992282,121.304856
# 2385,1348,24.986873,121.311122
# 764,1342,24.998116,121.311411
datafile = "taoyuanfit.csv"
points = np.loadtxt(datafile, delimiter=",", comments="#")
imgwidth, imgheight = 3975, 3209

# Points on the image, scaled to range 0 to 1
px, py = points[:, 1]/imgwidth, (imgheight-points[:, 0])/imgheight
# Points on the map
gx, gy = points[:, 3], points[:, 2]
# Do the fitting
lx, ly = np.polyfit(px, gx, 1), np.polyfit(py, gy, 1)

# Fitting parameters give map boundaries, results in this case
west = lx[1]          # 121.300983
east = lx[1] + lx[0]  # 121.331141
north = ly[1]+ly[0]   #  25.003429
south = ly[1]         #  24.981204

This is now pretty objective, and then quality of the overlay depends on the quality of the landmarks I find, and the maps that they are on.

The best landmarks I could turn up were notable roundabouts like below…

… and recognizable streets and intersections, like below…

… but not too small and not too large streets. Large streets have fewer recognizable locations (unless some notable mid-sized street is intersecting with it), and small streets are usually not placed too carefully on the map, so can’t really trust their positions.

What I couldn’t trust, though: rivers (they changed a lot), bridges (usually can’t tell if they are the same bridge in both era, or a new one built close to the no longer there old one, as it is a common practice), coastlines, lakes, large structures.

The minimal number of landmarks needed for a fit (ideally) is just two, and surprisingly, there were a couple maps where I could get along with about three. In most cases, though, I needed at least 4-6 landmarks, and there were some I needed to discard for overall better quality of fit. Some maps show pretty good accuracy, while others (such as e.g. Kaohsiung) are quite poor – compared to today’s satellite pictures, of course. It’s all enjoyable, though.

Then added a few more functions to the page, such as changing the opacity, being able to link to the exact location and zoom level I’m looking at, and some page-load animation (the maps are in the range of 4-10MB, not that quick to load).

The source code for the site is on Github, and pull requests are accepted, naturally.

History lessons learned

I was very interested to note the things that have changed, and things that haven’t in the decades passed.

Things that mostly stayed the same

Schools are usually at the same location, even stayed the same type (e.g. girls’ high school staying that in Changhua).

Airports are usually at the same location (duh!), though usually greatly expanded, while the old runways can be matched to the new ones (like in Hsinchu).

Train stations, of course, although tracks might change, or entire stations disappear (like in Beigang or in Hualien).

Military installations, which surprised me, can stay the same, looks like the KMT troops just took over the Japanese posts and continued to use them. Some turned into schools, or could disappear too.

Many roundabouts, outlines of city blocks (like the CKS Memorial Hall now, or Yilan’s city outline), mountains, also stayed mostly the same.

Things that changed a lot

City areas changed enormously (duh!), just look at the capital, Taipei, the west side stayed mostly the same but built in about twice/three times as big area on the east. Most other cities grown just like that, as it’s very clear for example for Taitung. It’s no wonder, 5.8 million people on the island in 1940, while above 23 million by now (according to Wikipedia). This also brings the changes of a lot of streets and roads.

Water is definitely very changeable. Rivers got diverted a lot, most notably on the east side of Taipei (near Neihu and Songshan districts) the Keelung river was straightened up, but the roads can still mirror the old path. Other rivers (or channels?) got covered up, and now only live on as roads – though I guess they might still flow under our feet (just like Fleet Street)? Some more dramatic change is an entire missing island in Danshui, I wonder what happened to it? Lakes change too, mostly I guess due to development around them (like in Zuoying). The sea shore is different for the many cities, as probably they were developed and also the shallows were naturally filled by the rivers (like in Hsinchu and in Tainan). Harbors don’t stay the same either (like in Penghu).

Some landmarks such as shrines have changed too. The most notable I could find so far is in Taipei / Yuanshan, where the current Grand Hotel / 圓山大飯店 replaced a shrine in 1952 (yellow rooftop on the left below), while the Taipei Martyrs’ Shrine seems to be built on the site of another previous shrine in 1969 (yellow rooftop and cross-shaped grounds on the right below). On the map there’s another temple-complex looking structure in between them, that’s now the Taipei American Club and the International Community Radio Taipei (ICRT).

Race courses disappeared too – together with the cavalry, I guess (like in Tainan).

I’m sure there are a bunch of other things too, it will take some time to discover and understand things…

Other notes

The legends and information on the maps was pretty interesting. For example here’s the coverage map for Taipei / Taihoku, showing what aerial maps they stitched together, and when were the flights that took the photos:

The Internet also taught me a few things, such as this very insightful comment on Reddit that traced back the Chinese / Japanese city names of these maps.

I was also amazed by how detailed information the Americans had about the functions of the buildings – down to the type of factories, the type schools the buildings were. I would not be surprised if the maps were supplemented by some real intelligence from the ground. Especially likely as the big cities had so much more such details than the smaller locations (where there might not be more than a few likely military installations noted). I wonder if there’s any good source to learn about this – about American intelligence behind enemy lines in the Pacific theatre of WWII.

To the Future

Now that everything kinda works, I think I’ll spend more time looking at the maps instead of working with them, try to learn more.

Hosting this project on Github is definitely pretty easy and convenient, though I start to see that their speed is not necessarily very good for big assets like the map images. Could add Cloudfront very easily to speed things up, just I don’t necessarily want to pay for that extra speed just yet. :P

To be able to show the loading animation I currently need to use jQuery to preload the image, then rely on the browser caching to not to download the image for the second time when the image is attached to the map object to create the overlay. It works pretty well… except in the Facebook Android app’s built in browser – that happily ignores caching and downloads the image twice (ouch). Would be awesome to figure out how can I get directly from the Google Map API when it finished downloading the image.

Might revisit a couple of the maps to see if I can improve the fit, though really just a couple of them, most are totally fine for their quality. Should also add the Japanese writing of the city names for more authenticity and discoverability.

And in the meantime, keep checking the map, and any comment or feedback is appreciated! A big thanks to the University of Texas Libraries and the Perry-Castañeda Library Map Collection for the public domain maps!

The post Taiwan WWII Map Overlays appeared first on ClickedyClick.

Many of the organizations improved their setup since then, and it became quite troublesome to manually check each bank and each change, update the table and so on. It’s also good to have not just a snapshot in time, but a continuous record of how they were doing.

Thus I’ve hacked together some monitoring scripts, put the results online, and here’s the Taiwan Financial Institute SSL Status page.

Page features include:

• Automatically run once a day
• Highlighting issues, showing grade evolution
• RSS feed of grade changes
• Automatic tweeting of daily status and changes as @twbankssl

This is quite a bit more than “minimal features”, but wanted to make something that is actually useful.

Notes on the tech

In a nutshell:

• to run the SSL Test queries, I’m using ssllabs-scan, a official program written in Go to do just that.
• scrapting is parallelized and managed by a Python script.
• the results are parsed and output JSON, RSS, tweets are generated by another Python script.
• the site displays the results as a single-page app, pulling in the results through the generated JSON file.
• the grade sparklines are using an external library.

All the code is open source on Github. Tried to make it reusable for people wanting to monitor any other sets of sites (eg. other countries’ banks, government institutions, own sites), though I did not completely succeed. There are hard-wired parts that could use a rewrite, but can be a starting point for any other project for sure. By the way, patches / pull requests are welcome if you see something could be improved!

Future

I hope to leave the scripts and page running for a while to see whether it actually works autonomously, whether it’s useful for anyone, and as a tool to push for change for these financial institutions. Will also try to connect to people here in Taiwan who can make that change happen faster.

Some code rewrite and expanding the documentation is also inevitable – once my current coffee high has been metabolized. :)

What do you think? How would you use this data, or how you’d improve on the service?

The post Taiwan Bank SSL Continuous Monitoring appeared first on ClickedyClick.

SSL Test Overview

I had a list of 43 banks, and for a quick overview I took into account a few key features only. The first is whether there are any active vulnerabilities against the site according to the test (these were mostly CRIME, FREAK, and POODLE attacks). The second is whether RC4 encryption was enabled, as it is now prohibited, and having it on is an automatic Payment Card Industry Data Security (PCI) compliance failure, according to one of the commenters. Other various warnings are mentioned when something really stands out, they are not crucial but more nice to have (though I’d contend that Forward Secrecy and HTTP Strict Transport Security is more than “nice” for anything financial).

Edit: Since publishing this post, there’s a brand new password recovery attack against RC4, so it’s even more urgent to switch it off.

Test Outcome

The only A rating was for Deutsche Bank, which uses its global site for all regional sections too, so not that surprising that it was better than average. The rest of it in a nutshell:

• 22 out 43 banks (51%) got an F,
• 2 of them did not have SSL (though no online banking either, fortunately),
• 1 SSL enabled site couldn’t be tested somehow, possibly because of redirections
• Only 4 of them (9%) disabled RC4, the rest of it (91%) is vulnerable to password recovery attacks,
• 24 of them (56%) are vulnerable to common attacks, 5 (11%) to multiple different attacks (besides RC4), POODLE being the most common
• There was one site where the general corporate website’s HTTPS was better configured (B) than the e-banking (F)…
• Handling redirects and subdomains is generally very confusing for banks
• Most banks have multiple subdomains for different services (“bank site”, “e-banking”, and “web ATM” are the three most common)
• Of the 4 banks I have account at, there’s one B and three F-rated (not naming names)

The raw overall data and test links follow (with an outlook afterwards). Usually I’ve linked to the e-bank pages of banks, unless I couldn’t see it or was the same domain as their corporate site.

Update (2015-03-19): Some banks started to fix their systems, noticed changes will be added to the bottom of this post!

Update (2015-03-25): Instead of keeping up with the changes manually, made a page for automatic status tests. It’s still in development but you can check it here: Taiwan Financial Institutes SSL Tests.

Taiwanese Bank SSL Test Results

What’s next?

Financial fraud in Taiwan is pretty prevalent, though it’s usually the “old fashioned” phone scam type. On the other hand, people here seem to be very lucrative target of groups based in China and Philippines. While I don’t know about the latter, the former likely has black hat teams with a lot more computer savvy than most users here (or anywhere else). It would be very-very good to fix up these systems.

Since these banks don’t seem to be on Twitter in general (except maybe the international ones), would be good to look up the local tech contacts, and bring the problem to their attention. One thing that makes me optimistic about it on the long term, is that Taiwan has a lot of computer savvy and outspoken activists. Will try to reach out to them, and use the local talent (and local ways) to approach this. Also, local administration (ie. city government) seems to be more powerful here, and we have a quite techy and driven new mayor in Taipei city. Putting pressure on the banks through them is not inconceivable.

I like it a lot where technology is going over here (many days I feel like being in a science fiction), and hope to make it work out well by keeping an eye on the practical implementations such as this.

If you find any problems in the dataset above (or any updates as sites are fixed:), please let me know!

Updates

• 2015-03-19: Chinatrust fixed their POODLE vulnerability, improving their score from F to B! Hope they’ll fix RC4 next.
• 2015-03-19: Bank of Taiwan went from F to A-, the first Taiwan-based bank to achieve an A!
• 2015-03-23: The ANZ Credit Card site went from F to B as well, which is pretty good, even if their corporate site remains F.
• 2015-03-24: Mega Bank went from B to A- by disabling RC4.

The post SSL status of Taiwanese banks: a sad affair appeared first on ClickedyClick.