1. A banking website with CAPTCHAs and no programmatic access
2. An email received each day with an password-protected PDF containing the last day’s transactions in a table

Neither of these are fully appetizing to tackle, but both are similar to bits that I do at #dayjob, but 2. was a bit closer to what I’ve been doing recently, so that’s where I landed. That is:

• Forward the received email (the email provider does it)
• Receive it in some compute environment
• Decrypt the PDF
• Extract the transaction data table
• Clean and process the tabular data
• Put raw in some data warehouse
• Transform data to get the right aggregation
• …
• Literally profit?

I was surprised how quick this actually worked out in the end (if “half a weekend” is quick), and indeed this can be a first piece of a “personal finance data warehouse”.

Technical implementation

I wanted to have the final setup run in “The Cloud”, as that’s one less thing to worry about. The most obvious arrangement, based on past experiences was combing AWS Simple Email Service (SES) to receive an email, and a Lambda to run serverless processing. On the data warehouse side the real obvious choice is GCP’s BigQuery, however, so I looked into what would be a similar arrangement for the processing pieces if I want to put everything into a single cloud provider.

After some docs diving the most natural arrangement on GCP seemed to be quite different: an App Engine deployment with Mail API enabled. This gives a receiving domain name (@[Cloud-Project-ID].appspotmail.com) , and every email sent there is just passed to the server that is running in App Engine. This seemed pretty simple! App Engine also has a free tier, though that comes with pretty small memory limits, which features in this story too.

The final result of the server part is shared on GitHub, and should be easy to reuse or extend.

PDF processing

Getting the attachments out of the email was pretty straightforward with the Mail API, so the first heavier task was opening the encrypted PDF and getting the table out of it. Opening PDFs are quite common, but the table extraction was a bit of a journey.

False try

First I was searching around (as anyone else does) for someone else’s rundown of the options, as an example. From there I honed in on pikepdf to open the password-protected files, an tabula-py which seemed handy to extract tables right into Pandas DataFrames. One subtlety was that tabula-py is just a wrapper around tabula-java to do the extraction, and needs a Java environment installed. The free tier of App Engine uses their standard environment where all I have is my code and “requirements.txt” to install my python dependencies, so it’s obvious how would I get Java into the deployment correctly.

Enter the scene install-jdk which can install the Java environment at runtime. That was sufficiently crazy hack to actually work, and it did work. Or so it seemed, since the data was processed and showing up in BigQuery, when I’ve sent test emails into the system.

Upon closer inspection, though, there were loads of duplicate lines. Between signing off in the evening, and checking it in the morning, I had bunches of them, and were still coming in…

I should have checked the logs earlier, because once dig in, there were bunches of “server errors” listed that didn’t connect to any programming errors that I might have made, rather than (here comes the epiphany) instances being killed for being out of memory / blowing their memory budget (of 256MB for the free tier). Thus what happened is:

• the Java run of tabula was just using too much memory while processing the PDFs
• it finished processing and like loaded the data but it takes a bit of time
• GCP catches up and kills the instance while that is still going on, and reports to the Mail API that the email hasn’t been properly handled (server error during that process)
• Whatever is handling the incoming email queue in GCP will just just keep the data and retries later
• The cycle repeats…

This didn’t seem very helpful and the repeat emails were piling up in whatever (opaque, to me) system GCP has, so needed a quick replace of tabulate with something lighter…

Worse is better and actually good

Going down the list of recommended libraries, next I looked at camelot-py which looks great, but needs OpenCV on the machine to do its work, so back to the “how to install OS packages on Standard AppEngine?” question. For some extra inspiration I was looking at camelot’s comparison with other similar tools page and it was a bit disappointing (though not surprising) that pretty much every other library is “worse” on various PDFs compared to camelot. Just for kicks I did try some out, and pdfplumber actually delivered:

• it does actually work on the example PDFs I had from previous bank emails
• nothing else beside pip install
• it can actually handle decrypting the PDF as well, so helper libraries can be dropped
• the extracted data is in Python tables, but it’s just an extra line to get DataFrames, so no sweat
• The extracted data was actually better quality than tabula’s, so had to do fewer cleanup steps!

This was a pure win, and indeed it’s worth looking stuff that works with the data at hand, not ignoring the edge cases, but also not overly emphasizing being able to do “everything” when there’s a clear target of what “thing” needs to work. (Potential technical debt considered too).

Data transformations and visibility

Now the data sits in BigQuery properly:

The raw transaction data loaded into BigQuery was the first step, but still need to answer the question: in this billing period, how much have I spent?

Not being a data analyst (or not yet?:), this took a bit of figuring out. As other novices share their bit of “clever code” when it’s actually trivial to the experts, I’m sharing here the bit of SQL queries in a similar “that was fun to figure out, wasn’t it?” way. I’m sure it can be much improved, but it’s a good reminder for myself as well.

Given that my billing period starts on the 23rd of the month, get the aggregated value of transactions for each billing period:

WITH
  Aggregated AS (
  SELECT
    DATE(TransactionDate, 'Asia/Taipei') AS day,
    TransactionAmountNTD
  FROM
    `personal-data-warehouse.finance.huanan` ),
  calendar AS (
  SELECT
    day,
    -- Find the last day before the new interval
    DATE_SUB(
      DATE_ADD(
        day,
        INTERVAL 1 Month),
      INTERVAL 1 DAY
    ) AS endday
  FROM
    UNNEST (
      GENERATE_DATE_ARRAY(
        -- Start date in the past before any data,
        -- on the right day of the month for
        -- the billing cycle.
        '2022-05-23', 
        CURRENT_DATE('Asia/Taipei'),
        INTERVAL 1 Month
      )
    ) AS day
) SELECT
  SUM(TransactionAmountNTD) AS `MonthlyTransactions`,
  COUNT(*) AS `TransactionCount`,
  EXTRACT(Year FROM c.day) AS `Year`,
  EXTRACT(Month FROM c.day) AS `Month`,
  FORMAT('%d-%02d', EXTRACT(Year
    FROM
      c.day), EXTRACT(Month
    FROM
      c.day)
  ) AS `Interval`
FROM
  calendar AS c
JOIN
  Aggregated AS a
ON
  a.day BETWEEN c.day AND c.endday
GROUP BY
  c.day

Good stuff on the date array and joining with a “between” statement, those are the main TIL. They also already came up at #dayjob, which was very satisfying.

From here the data I surface in a connected Google Sheet which is pretty practical, though leaves the “being notified when I approach/reach X” out, but that’s fine for now.

Testing and getting to “production”

One good thing about personal projects is that I can make them as “good” as I want to (or as “bad”, of course), which usually results in an unhealthy amount of tweaking, trying out various best practices to see if they work, and so on. Here I really wanted to get the system well tested, for example, which turned out to take loads more time than actually writing the original service. Actually, there’s nothing surprising about that for software engineering professionals, but still can catch people off-guard.

Here the tricky parts came from two areas: FastAPI settings and cloud service integrations.

The former is always a bit of an issue, depending on how the code uses the settings (whether things can be patched well at testing time), but here I also used a trick for the server to pull the PDF decryption key from Secret Manager, so I don’t have to deploy environment files, nor keep settings like that in version control, etc… But this meant a trickier flow of getting the FastAPI testing client up in a way that it worked without it talking to the cloud backends (and stalling, and failing…). Nothing that some good mocking cannot solve (says the person with hindsight).

For the cloud services part it meant mocking BigQuery connections, so that the test can actually pretend to “receive” an email all the way looking at the “database” and see the right information being there. Under the hood I’m using pandas-gbq, and thus it was interesting to look under the hood for their tests, borrowing some of them. Took a bit more time, but that’s working pretty well now. Still need to do some extra bits and pieces to do cover more of the workflow, but I’m already more confident about things working. Also, all this will be very useful on other projects that are interacting with BigQuery in any way (not just through Pandas).

Big evergreen lesson on testing: you have to write your code to be testable. Lots of code out there is not even not tested, but it’s even extremely difficult to actually test. This needs remembering in every development. Also, test writing never really stops, there’s always more thing to test for. And finally, can always try more advanced testing, such as using automated test case generation (e.g with hypothesis), and fuzz testing (e.g. with pythonfuzz). The next frontier, right after I’ve implemented the currently skipped tests. And finally, remember that code coverage is not case coverage, so the goals should be maximizing the latter, while the former is just a potential proxy for it.

Future outlook

It would be nice to take this idea of financial data analysis further and add some actual dashboard (say deploying Superset somewhere which is excellent for this). It would help to get more information into the system as well, though, currently it’s very sparse. That would mean adding other financial sources, maybe if finding an API in the end, or doing a bit of “pragmatic execution” and do a CAPTCHA bypass (since I quickly checked that my credit card provider’s CAPTCHA is completely readable by Tesseract, for example, so I could likely scrape things there if I really wanted.

I’m not holding my breath for having something like UK’s Open Banking here which enables apps like Emma so all this is accessible for people who don’t want to code. But where’s the fun in that (for me)? :) (In fact there’s a lot of fun in open access APIs, so this would be the real way of doing it…)

Finally, it’s good to remember how easy it is to corrupt “production” data sets, but also with the right tools (like snapshots), some of that pressure can be less. There are always bugs, the question is how to mitigate their effect.

The post A personal finance data pipeline project appeared first on ClickedyClick.

In this rather graphic post I try to summarize the process of getting a MOICA card as a foreigner in Taiwan, setting it up, and some (opinionated) lessons to learn from it.

The Process

The process of getting and setting up the card is outlined in this leaflet that I’ve picked up.

Though – not surprisingly – in practice everything is a lot more complicated than these 8 bulletpoints.

There’s also a Chinese version of this leaflet, just for the record.

Registration

As it is included in the National Immigration Agency’s (NIA) own announcement, the process is kicked off by visiting one of the service centers (here’s a list, but people generally know which center they belong to, they have to), and requesting the card. Even though I was applying (in the Taipei center) on one of the first days when it was available, there was no surprise or difficulty. Picked up a number, the at the counter forms were filled out for me all digitally, and then got a receipt like this (sensitive info blacked out here):

It included my application number, name, email, phone number, and a registration code (the last blacked out line) which was needed for the online payment. This is because, for some reason, the bill cannot be settled at the time of the application, but needed to be done online.

Payment

The card is only sent out to me after the online payment, and if no cash is received in 14 days, the application is cancelled. Thus I wanted to settle things very quickly. Originally I thought, based on the info I’ve received at the NIA that I will receive an email or other electronic communication with further info on how to pay. Instead, for the next 2-3 days I got a daily nudge message that I need to pay, but no information on how.

Visiting the MOICA website for foreigners (“aliens”) does not make me smarter, it looks like this below, with a couple of options on the side, but no “PAY HERE” or similar.

After one and a half weeks the time was running out for the application, so gave in and called the 0800-080-117 number from the message for help. They did help: see the funny looking Chinese-only banner with two cartoon figures? Have to click that, and then go onto this Chinese-only interface to continue.

Using the code from the receipt and logging in, there was an online payment interface. I really should have taken a screenshot, but so taken aback by it, that I forgot. It’s a HTTPS page (for a change), with big (Chinese-only) announcement that “because the site uses SSL your data is completely safe and nothing to worry about”, plus Chrome giving me a big security warning that the website’s SSL settings are vulnerable. Who do you trust in this situation? Rhetorical question: of course Chrome but still pay the government…

At least their messaging service works a lot better than their website, within seconds I got dual language notification that the credit card payment went through fine.

There’s not much to do after this, but wait till the postman arrives with the card.

Card Activation

The card arrived with two page of instructions, sticked in the middle, highlighting the security features of the card.

On a side note, not sure how all these visual security features (which likely serve paper bills well) help with computer security, since the only important part of that card is the chip inside, but I digress.

The minimum system requirements are Windows XP SP3 and IE 6.0. I’m using Linux, but fortunately have an XP in VirtualBox, and Internet Explorer on it, so might just work. The instructions said to go to the MOICA website again, and activate the card there. Except, there was no activation-related menu there at all, and the instructions remain tight-lipped. I was hoping that the back-side of the instructions, that are in Chinese, will contain some clues, though it turns out that they are even less useful, if that’s possible.

It’s time for another phone call to the 0800 number, and turns out that the “helper kids” are the ones we need again. Clicking that Chinese-only banner on the otherwise dual-language website brings up the page that I used earlier, but now instead of logging in, have to click the last menu point on the left, saying 憑證IC卡開卡. That takes me back to a page that looks very similar to the original MOICA page – except the menu is full. Indeed, the first page only contains “Commonly Used Functions” in the menu and have to do this extra trip to get a complete list of available “Certificate Procedure“[sic].

Here the activation can begin in earnest. First steps are:

• the site prompting for the install of half-a-dozen plugins,
• there’s an external program for the card reader smart card functions called HiCOS (get it from the download page)
• besides the card reader’s own drivers (I have an EasyATM Pocket Reader 111, seem to work well both in Linux and Windows)
• and a “gcms-plugin” linked from the activation page, what this does is unclear to me.

Most of these are also outlined (in Chinese) on one of the helper pages.

I did all this, loaded the card data (取得資料), filled out the page’s form, and submitted (the 開卡 button, “enable card”), but the process just hung there for about 5 minutes without any result. Tried more times that I probably should have…

One difficulty is that the page asks to “enter” subscriber code, and “set” PIN code: I didn’t know any subscriber code (probably meaning “username”), and “enter” is not “set” but who knows how the translations work on these sites?

On the page there was a link to help if things don’t work, which takes us to the MOICA blog, lo and behold:

This doesn’t strike me with a lot of confidence. Not just that it’s all in Chinese, the post is more than a year old, the color scheme is going crazy, and basically doesn’t contain any new info. Mainly it’s about using an external site (together with advertisements) for an important information like this.

Further down the page there was another link to even more help on this blog to how to set things up for the MOICA site to work with the smart card reader.

The info is in Chinese, but more or less says this:

• Enable compatibility mode in IE for moica.nat.gov.tw and  61.60.9.42,
• In Internet Options / Security / Trusted Sites / Sites disable “require server verification (https:) for all sites in this zone”, and add moica.nat.gov.tw and  61.60.9.42 to the trusted sites
• Internet Options / Security / Trusted Sites / Custom Level enable everything releated to ActiveX (in my opinion, set to “Prompt” instead of “Enable” whenever possible)
• Clear cookies and browser data and try afresh

This is pretty horrifying again, but what to do, have to make that card work somehow. The IP in that list is actually the IP where the form is served, within an iframe on the MOICA site… No security, open everything if you want to use the card.

The final comment on the page says, that if the activation form still hangs, then something is still not set up correctly. Of course my form was still hanging, but I was quite confident, that I haven’t missed any part of the setup. It’s time change strategy: to put myself into the shoes of the developer of a system like this. Imagining it as someone is making a site to technical specs only but not for usability, I was questioning what functions I might need. Looking through the menu, there is a “How to change subscriber code” page – and it does not ask for existing subscriber code, only other information! So it is more like resetting a forgotten code/password, this might be what I need!

On that page I’ve added my Alien Residency Certificate number, date of birth, and my MOICA card’s number as requested, and got a brand new subscriber code.

Putting this new subscriber code into the activation page, setting a PIN and finished in a second like a charm: Success! The activation was also verified by going on the Certificate Query page (using the 64-digit certificate serial number from the Activation page that can load the card data):

The lesson seem to be, unless I’ve missed something, that a subscriber code needs to be created first (not mentioned in the help), and the activation page hangs on all kinds of issues such as missing/wrong subscriber number as well, not just when the technical setup is incorrect.

Now what can I use this card for? I’ve seen some articles saying that there are 3000 different services to be used with, but I guess that’s only sort of correct. The FAQ says:

[…] electronic highway drivers services, tax filing, internet application system of the Labor Insurance Bureau, electronic gateway of the Land Administration, household registration internet applications, ChungHwa Telecom mobile phone (citywide) query, company online application system of the Ministry of Economic Affairs, public safety inspection system for buildings, internet customs clearance system for air cargoes, 1988 tax information instant messaging, and so on.

Foreigners cannot do tax filing for the 2015 tax year (just being filed now till the end of this month), but should be good from next year (I’m using my stock trading file-based certificate so I’m good to file online anyways). Other uses I guess need some exploration.

Endnotes

Tech

I haven’t really used smart cards for anything so far, the only other one is the National Health Insurance card earlier this month, that was a tiny bit more straightforward, and sort of immediately useful.

Under Linux I can use pcsc-tools and the CCID hardware driver for my reader to get some info about the card’s hardware, for what it’s worth:

Thu May 26 15:23:24 2016
Reader 0: Generic USB2.0-CRW [Smart Card Reader Interface] (20070818000000000) 00 00
  Card state: Card inserted, 
  ATR: 3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5

ATR: 3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5
+ TS = 3B --> Direct Convention
+ T0 = B8, Y(1): 1011, K: 8 (historical bytes)
  TA(1) = 13 --> Fi=372, Di=4, 93 cycles/ETU
    43010 bits/s at 4 MHz, fMax for Fi = 5 MHz => 53763 bits/s
  TB(1) = 00 --> VPP is not electrically connected
  TD(1) = 81 --> Y(i+1) = 1000, Protocol T = 1 
-----
  TD(2) = 31 --> Y(i+1) = 0011, Protocol T = 1 
-----
  TA(3) = FA --> IFSC: 250
  TB(3) = 52 --> Block Waiting Integer: 5 - Character Waiting Integer: 2
+ Historical bytes: 43 48 54 4D 4F 49 43 41
  Category indicator byte: 43 (proprietary format)
+ TCK = A5 (correct checksum)

Possibly identified card (using /home/user/.cache/smartcard_list.txt):
3B B8 13 00 81 31 FA 52 43 48 54 4D 4F 49 43 41 A5
	citizen digital certificate (PKI)
	http://moica.nat.gov.tw/

Regarding the websites and web services in general I see more issues than good sides:

Chinese and English are used in a very strange way. English translation is often uninformative (at best, misleading at worst), or missing. Sometimes resetting a form changes all the labels into Chinese, the relevant FAQs are often in Chinese…. All this wouldn’t be a problem, if it wasn’t advertised to be usable in English too. So far it’s about a 20% solution. Getting people to translate, doing consistent writing, not having text in images (so copy-paste translation can work as a last resort), and just thinking about the users in general would go a long way.

Paper documentation and blog that contains help and advice is out of date and incomplete. The website has changed since the screenshots, IE has changed since the screenshots, not all steps are described, some technical info is not presented in enough detail to make it work easily. This is again I think an issue of low manpower for maintenance and people not putting themselves into the user’s shoes when writing documentation (that’s a common issue in all tech).

The site’s front page showing Commonly Used Functions only, and no apparent way to see all functions. That’s just a big usability issue (people needing to call to use the website), that should come up if they check what people call them about. So I guess there’s no monitoring and review of support calls or no feedback to the site design.

A big no-no is not having SSL enabled everywhere, it would be so easy to do. Chunghwa Telecom that they used for some of the smart card software development is the local Certificate Authority and the government has its own too, can’t imagine why there isn’t security set up. There’s even Let’s Encrypt for last resort now, so no excuses. Those pages that have SSL setup seems to be done by people who are not familiar with it, or not maintaining it properly. It’s a big misconception that “if HTTPS is in the URL, then the communication is secure”. In the public sector good sysadmins don’t fall for this, so the government and public sector need to step up their efforts. Staying with this immature security thinking will just lead to more issues (such as the just hacked Chungwa Post).

Using IP address-only designation and iframes for the smart card functions is also baffles my mind. I cannot imagine legitimate technical reasons for that, would be happy to hear if anyone knows one. Right now it looks like an alpha-quality software.

Relying on ActiveX for all these functions obviously is the result of the age of the project but it is definitely not future proof. Win 10 and the latest IE drops ActiveX as well, so there are no up-to-date systems that could use this smart card functionality. How many people will keep an XP around just to use the government websites? (The answer is probably not many and still more than should be).

Alternatives

With all this technical background, I really feel that this is a dead-end solution, which might work for a while still but has no future. To have a future, it needs to break from ActiveX, and go cross-platform too. Probably would need to break from the smart card format as well (at least as it is now). I’m in no way an expert in this area, these are some ideas based on my experience so far.

If trying to keep such certificate-based services around, then there has to be some serious research put into either how to use smart cards better (maybe based on OpenSC?), or change hardware. I’m thinking something like Yubikey could be an interesting model, especially with it’s Personal Identity Verification PIV support. Probably PIV and the related standards (that I have very tiny idea about so far) would be a viable path to look at.

Another alternative comes to my mind thinking about how bank logins work in some countries, using a Two-Factor Authentication (2FA) device that generates a one-time password for logging in. I see the model of this good in the sense of not having any hardware attached to the computer so it’s by default cross-platform and much less setup. Of course 2FA alone is surely not enough (eg. if electronic signature functions are used with MOICA), but the current model also just seem to be a 2FA (the factors are the card and the pin) login for the services. Maybe a mixed model, with simplicity in mind would be a good path.

Future

The idea of MOICA is fundamentally good, and pretty much inevitable in one form or another. The main thing would be thinking over the use cases and user experience a lot better. It is an interesting project, to a level that I almost wish I could work on this: technology development, QA, user & system testing, project development consulting, security enhancement, evaluation… It is unlikely to happen, so I really-really hope the Taiwan government (ie. the relevant ministries) have a decent sized and quality team working on this. Otherwise…. disaster and/or becoming irrelevant is also pretty much inevitable, only question is time.

Would love to hear if anyone else goes through this process and what do they use your cards for?

The post Taiwan Citizen Digital Certificate appeared first on ClickedyClick.

There are 26 maps in the collection, made by the US Army by flying over different parts of the island, and mostly I guess stitching together aerial photographs. The maps themselves were not that easy check in an image viewer, since there’s no context, zoom is clumsy, and have no idea where about half the places should be located. Instead, I thought it would be great to have them as an overlay on top of current maps and satellite imagery on Google Maps.

The result is Taiwan City Maps overlays, which does exactly that. Feel free to click the link and explore right now! In the rest of this post, I try to first show how that page was made, and also some history lessons I gained by making it.

How it was made

The Google Maps Ground Overlay API does exactly what I needed, so can’t claim much (almost any) innovation in the page, though I stumbled on it after a few other “Overlays”, as Google Maps has a couple, all working slightly differently.

Ground Overlay basically takes the following parameters:

• the map’s center,
• zoom level,
• an image,
• the image’s N/E/W/S bounds in geographical coordinates.
• whether to show satellite or maps view underneath

Even cooler, the opacity of the image can be adjusted, so the overlay and the base layer can be viewed together much easier!

In that list above, the trickiest part is of course figuring out the boundaries of the image – basically the way to overlay them – for multiple reasons. First, all the maps have borders with information, titles, legend, and so on. Here below is the Taipei city plan from 1944. I can’t really tell where the corner of the image should be geographically, if there’s nothing to compare to.

Then there’s also the problem of things have changed in the last ~70 years, so a lot of things are different on the maps, if I want to match them up. Lastly, the maps are not perfectly accurate either, they are stitched from multiple photographs of unknown quality.

Been trying to match the maps by hand for a while, and codings some tools to make that process easier – but it didn’t really get easier. I needed to get to get around all these issues, and have a reasonable, objectively good fit. Objectivity is a key, and math helps with that. What I ended up with doing a linear fit of geographical and image coordinates.

• look for landmarks that are reasonably the same back then and now
• note their geographical coordinates, and their pixel coordinates on the image
• note the image size
• finally: run a linear fit that gives back the geographical coordinates of the image edges that would best match the landmark data in both sources

In really ugly Python it would be something like this:

import numpy as np

# # For example the contents of "taoyuanfit.csv":
# # Landmark data: image vertical/horizontal (px), latitude/longitude
# 1677,2861,24.991962,121.322634
# 1597,524,24.992282,121.304856
# 2385,1348,24.986873,121.311122
# 764,1342,24.998116,121.311411
datafile = "taoyuanfit.csv"
points = np.loadtxt(datafile, delimiter=",", comments="#")
imgwidth, imgheight = 3975, 3209

# Points on the image, scaled to range 0 to 1
px, py = points[:, 1]/imgwidth, (imgheight-points[:, 0])/imgheight
# Points on the map
gx, gy = points[:, 3], points[:, 2]
# Do the fitting
lx, ly = np.polyfit(px, gx, 1), np.polyfit(py, gy, 1)

# Fitting parameters give map boundaries, results in this case
west = lx[1]          # 121.300983
east = lx[1] + lx[0]  # 121.331141
north = ly[1]+ly[0]   #  25.003429
south = ly[1]         #  24.981204

This is now pretty objective, and then quality of the overlay depends on the quality of the landmarks I find, and the maps that they are on.

The best landmarks I could turn up were notable roundabouts like below…

… and recognizable streets and intersections, like below…

… but not too small and not too large streets. Large streets have fewer recognizable locations (unless some notable mid-sized street is intersecting with it), and small streets are usually not placed too carefully on the map, so can’t really trust their positions.

What I couldn’t trust, though: rivers (they changed a lot), bridges (usually can’t tell if they are the same bridge in both era, or a new one built close to the no longer there old one, as it is a common practice), coastlines, lakes, large structures.

The minimal number of landmarks needed for a fit (ideally) is just two, and surprisingly, there were a couple maps where I could get along with about three. In most cases, though, I needed at least 4-6 landmarks, and there were some I needed to discard for overall better quality of fit. Some maps show pretty good accuracy, while others (such as e.g. Kaohsiung) are quite poor – compared to today’s satellite pictures, of course. It’s all enjoyable, though.

Then added a few more functions to the page, such as changing the opacity, being able to link to the exact location and zoom level I’m looking at, and some page-load animation (the maps are in the range of 4-10MB, not that quick to load).

The source code for the site is on Github, and pull requests are accepted, naturally.

History lessons learned

I was very interested to note the things that have changed, and things that haven’t in the decades passed.

Things that mostly stayed the same

Schools are usually at the same location, even stayed the same type (e.g. girls’ high school staying that in Changhua).

Airports are usually at the same location (duh!), though usually greatly expanded, while the old runways can be matched to the new ones (like in Hsinchu).

Train stations, of course, although tracks might change, or entire stations disappear (like in Beigang or in Hualien).

Military installations, which surprised me, can stay the same, looks like the KMT troops just took over the Japanese posts and continued to use them. Some turned into schools, or could disappear too.

Many roundabouts, outlines of city blocks (like the CKS Memorial Hall now, or Yilan’s city outline), mountains, also stayed mostly the same.

Things that changed a lot

City areas changed enormously (duh!), just look at the capital, Taipei, the west side stayed mostly the same but built in about twice/three times as big area on the east. Most other cities grown just like that, as it’s very clear for example for Taitung. It’s no wonder, 5.8 million people on the island in 1940, while above 23 million by now (according to Wikipedia). This also brings the changes of a lot of streets and roads.

Water is definitely very changeable. Rivers got diverted a lot, most notably on the east side of Taipei (near Neihu and Songshan districts) the Keelung river was straightened up, but the roads can still mirror the old path. Other rivers (or channels?) got covered up, and now only live on as roads – though I guess they might still flow under our feet (just like Fleet Street)? Some more dramatic change is an entire missing island in Danshui, I wonder what happened to it? Lakes change too, mostly I guess due to development around them (like in Zuoying). The sea shore is different for the many cities, as probably they were developed and also the shallows were naturally filled by the rivers (like in Hsinchu and in Tainan). Harbors don’t stay the same either (like in Penghu).

Some landmarks such as shrines have changed too. The most notable I could find so far is in Taipei / Yuanshan, where the current Grand Hotel / 圓山大飯店 replaced a shrine in 1952 (yellow rooftop on the left below), while the Taipei Martyrs’ Shrine seems to be built on the site of another previous shrine in 1969 (yellow rooftop and cross-shaped grounds on the right below). On the map there’s another temple-complex looking structure in between them, that’s now the Taipei American Club and the International Community Radio Taipei (ICRT).

Race courses disappeared too – together with the cavalry, I guess (like in Tainan).

I’m sure there are a bunch of other things too, it will take some time to discover and understand things…

Other notes

The legends and information on the maps was pretty interesting. For example here’s the coverage map for Taipei / Taihoku, showing what aerial maps they stitched together, and when were the flights that took the photos:

The Internet also taught me a few things, such as this very insightful comment on Reddit that traced back the Chinese / Japanese city names of these maps.

I was also amazed by how detailed information the Americans had about the functions of the buildings – down to the type of factories, the type schools the buildings were. I would not be surprised if the maps were supplemented by some real intelligence from the ground. Especially likely as the big cities had so much more such details than the smaller locations (where there might not be more than a few likely military installations noted). I wonder if there’s any good source to learn about this – about American intelligence behind enemy lines in the Pacific theatre of WWII.

To the Future

Now that everything kinda works, I think I’ll spend more time looking at the maps instead of working with them, try to learn more.

Hosting this project on Github is definitely pretty easy and convenient, though I start to see that their speed is not necessarily very good for big assets like the map images. Could add Cloudfront very easily to speed things up, just I don’t necessarily want to pay for that extra speed just yet. :P

To be able to show the loading animation I currently need to use jQuery to preload the image, then rely on the browser caching to not to download the image for the second time when the image is attached to the map object to create the overlay. It works pretty well… except in the Facebook Android app’s built in browser – that happily ignores caching and downloads the image twice (ouch). Would be awesome to figure out how can I get directly from the Google Map API when it finished downloading the image.

Might revisit a couple of the maps to see if I can improve the fit, though really just a couple of them, most are totally fine for their quality. Should also add the Japanese writing of the city names for more authenticity and discoverability.

And in the meantime, keep checking the map, and any comment or feedback is appreciated! A big thanks to the University of Texas Libraries and the Perry-Castañeda Library Map Collection for the public domain maps!

The post Taiwan WWII Map Overlays appeared first on ClickedyClick.

Many of the organizations improved their setup since then, and it became quite troublesome to manually check each bank and each change, update the table and so on. It’s also good to have not just a snapshot in time, but a continuous record of how they were doing.

Thus I’ve hacked together some monitoring scripts, put the results online, and here’s the Taiwan Financial Institute SSL Status page.

Page features include:

• Automatically run once a day
• Highlighting issues, showing grade evolution
• RSS feed of grade changes
• Automatic tweeting of daily status and changes as @twbankssl

This is quite a bit more than “minimal features”, but wanted to make something that is actually useful.

Notes on the tech

In a nutshell:

• to run the SSL Test queries, I’m using ssllabs-scan, a official program written in Go to do just that.
• scrapting is parallelized and managed by a Python script.
• the results are parsed and output JSON, RSS, tweets are generated by another Python script.
• the site displays the results as a single-page app, pulling in the results through the generated JSON file.
• the grade sparklines are using an external library.

All the code is open source on Github. Tried to make it reusable for people wanting to monitor any other sets of sites (eg. other countries’ banks, government institutions, own sites), though I did not completely succeed. There are hard-wired parts that could use a rewrite, but can be a starting point for any other project for sure. By the way, patches / pull requests are welcome if you see something could be improved!

Future

I hope to leave the scripts and page running for a while to see whether it actually works autonomously, whether it’s useful for anyone, and as a tool to push for change for these financial institutions. Will also try to connect to people here in Taiwan who can make that change happen faster.

Some code rewrite and expanding the documentation is also inevitable – once my current coffee high has been metabolized. :)

What do you think? How would you use this data, or how you’d improve on the service?

The post Taiwan Bank SSL Continuous Monitoring appeared first on ClickedyClick.

SSL Test Overview

I had a list of 43 banks, and for a quick overview I took into account a few key features only. The first is whether there are any active vulnerabilities against the site according to the test (these were mostly CRIME, FREAK, and POODLE attacks). The second is whether RC4 encryption was enabled, as it is now prohibited, and having it on is an automatic Payment Card Industry Data Security (PCI) compliance failure, according to one of the commenters. Other various warnings are mentioned when something really stands out, they are not crucial but more nice to have (though I’d contend that Forward Secrecy and HTTP Strict Transport Security is more than “nice” for anything financial).

Edit: Since publishing this post, there’s a brand new password recovery attack against RC4, so it’s even more urgent to switch it off.

Test Outcome

The only A rating was for Deutsche Bank, which uses its global site for all regional sections too, so not that surprising that it was better than average. The rest of it in a nutshell:

• 22 out 43 banks (51%) got an F,
• 2 of them did not have SSL (though no online banking either, fortunately),
• 1 SSL enabled site couldn’t be tested somehow, possibly because of redirections
• Only 4 of them (9%) disabled RC4, the rest of it (91%) is vulnerable to password recovery attacks,
• 24 of them (56%) are vulnerable to common attacks, 5 (11%) to multiple different attacks (besides RC4), POODLE being the most common
• There was one site where the general corporate website’s HTTPS was better configured (B) than the e-banking (F)…
• Handling redirects and subdomains is generally very confusing for banks
• Most banks have multiple subdomains for different services (“bank site”, “e-banking”, and “web ATM” are the three most common)
• Of the 4 banks I have account at, there’s one B and three F-rated (not naming names)

The raw overall data and test links follow (with an outlook afterwards). Usually I’ve linked to the e-bank pages of banks, unless I couldn’t see it or was the same domain as their corporate site.

Update (2015-03-19): Some banks started to fix their systems, noticed changes will be added to the bottom of this post!

Update (2015-03-25): Instead of keeping up with the changes manually, made a page for automatic status tests. It’s still in development but you can check it here: Taiwan Financial Institutes SSL Tests.

Taiwanese Bank SSL Test Results

What’s next?

Financial fraud in Taiwan is pretty prevalent, though it’s usually the “old fashioned” phone scam type. On the other hand, people here seem to be very lucrative target of groups based in China and Philippines. While I don’t know about the latter, the former likely has black hat teams with a lot more computer savvy than most users here (or anywhere else). It would be very-very good to fix up these systems.

Since these banks don’t seem to be on Twitter in general (except maybe the international ones), would be good to look up the local tech contacts, and bring the problem to their attention. One thing that makes me optimistic about it on the long term, is that Taiwan has a lot of computer savvy and outspoken activists. Will try to reach out to them, and use the local talent (and local ways) to approach this. Also, local administration (ie. city government) seems to be more powerful here, and we have a quite techy and driven new mayor in Taipei city. Putting pressure on the banks through them is not inconceivable.

I like it a lot where technology is going over here (many days I feel like being in a science fiction), and hope to make it work out well by keeping an eye on the practical implementations such as this.

If you find any problems in the dataset above (or any updates as sites are fixed:), please let me know!

Updates

• 2015-03-19: Chinatrust fixed their POODLE vulnerability, improving their score from F to B! Hope they’ll fix RC4 next.
• 2015-03-19: Bank of Taiwan went from F to A-, the first Taiwan-based bank to achieve an A!
• 2015-03-23: The ANZ Credit Card site went from F to B as well, which is pretty good, even if their corporate site remains F.
• 2015-03-24: Mega Bank went from B to A- by disabling RC4.

The post SSL status of Taiwanese banks: a sad affair appeared first on ClickedyClick.

It turned out that I needed both Kickstarter projects to make one good device.

The Foldable USB Solar Cell looks awesome and not bad at 7W and 5W (the two pieces I have). The voltage output doesn’t seem to be very stable, or has strange behaviour as it doesn’t charge my attached phone when there’s too much direct sunshine. Thus I cannot really use this directly with devices.

The tiny solar panels in the Solar Pocket Factory are very fragile and I couldn’t really make them into an actual working cell yet. On the other hand, it came with a 2000mAh battery and a circuit called Li-Po Rider Pro, by Seeed Studio. This circuit us solar cell or USB input to charge a battery , plus have a good output circuit to charge a USB device either from the battery when there’s no sunshine or from the input when there is.

Putting together the Foldable USB Solar Cell and the Li-Po Rider Pro, we have something that kicks arse indeed! Since I only have one circuit and Seeed doesn’t seem to sell it separately, I thought I could improve on things if I use both solar

USB adapter for parallel panels

The easiest idea I could come up with is creating a USB adapter to connect the panels in parallel (thus practically summing them up as current sources).

This plan needs two male USB A connectors (the “cable” type on the picture) to plug into the panels, and one female USB A (the “device” type) to provide the single output. Then female VCC pin is wired to both male VCC pins, and the GND pin similarly to the GND pins. The D-/D+ pins are not in use in this case.

The connectors came in as $0.40 each, the wire I don’t remember but probably a few cents. I fortunately had both black and red to use with the GND and VCC. Things are better colour coded.

First the middle two pins of a male and a female connector is soldered together, just to provide mechanical support. The end of the cable is stripped and hooked around the touching pins (the GND pins on this picture). The two pins + wire is then soldered together, with enough solder to stay, but not that much that the other pins could touch.

Did the same thing for the VCC pins + red wire. Finally added some more solder to the central pins. It was pretty stable like this, though I guess it would be better to have some sort of external housing for it or another way to increase the mechanical rigidity of the connection.

Finally the other connector is soldered, making sure that the right pins are connected (the same as the other male USB). The connectors are then wrapped in a bit of duck tape for some rough insulation. These parts will be outside, both in sun and rain, nothing much can break, but better not test whether I’m right about this particular point…

Plugged into the secondary (5W) solar panel, and into the primary (7W) one as well. Duck taped everything down onto the roof well enough that they are not blown away too easily (though I’d better check on them in a bit, the wind is howling just now outside).

As an aside, the duck tape doesn’t seem to like the sunshine. It sticks its sticky parts on everything, that part doesn’t seem to spell fun for the future.

Ready to charge

When everything is connected, the panels will get pretty good sunshine for the bigger part of the day. I don’t think they reach max capacity, because the angle is never ideal, but from the practical point of view, they get enough sunshine to max out the attached 2000mAh battery between device charges.

The Li-Po Rider Pro circuit is under the roof, so it doesn’t get any rain (at least I hope!), and has a little ledge to charge a phone safely as well.

Since all the devices were pretty much full when I checked in the Hackerspace after I finished this setup, the test whether combination of the two solar panels does indeed increase the power is not foolproof. Otherwise I would have charged a device from a low battery level to a bit higher with one panel, then later switch to two panels and see the different slope of the batter charge versus time. Normally at high charge levels (90%+) the slope is varied by the charging circuit (to preserve the lithium battery’s life), thus that level is not really good to test the difference between the single / dual panel situations.

One thing does suggest, though that the plan worked. Before this modification, charging a phone discharged the Li-Po Rider Pro’s own battery (there’s a touch switch on the circuit to get an approximate charge level, indicated by the lighting up of 0 to 4 LEDs on an LED bar), even in full sunshine. This time it seems that both the phone and the storage battery is charged up, indicating larger incoming power than before. I will check it again next time, though (the Nexus 7 tablet we have will be great for that, it has much larger internal battery than my HTC Butterfly phone)

Possible improvements

There are a few things I could improve on the setup, focusing on usability.

I hope to get a longer USB cable to reach from the roof to inside the Hackerspace, and set up a charging station within the room. This way people don’t have to keep their devices out of reach while charging, and there’s no chance of being ruined by a sudden rain.

I hope to get a larger capacity Li-Po battery. 5-10.000mAh could be good, then it would likely have enough power all the time to keep charging people’s devices, and not standing idle being full when not charging something, while easily emptied by a single device.

The roof also has plenty of more space, so getting a bunch of other panels, combined with the first two upgrades, would make it really beneficial. These foldable panels would be better portable, and keeping them in one place feels a sort of waste.

Ultimately it would be awesome if I could have a purely solar-powered phone. Because it would mean some money saved, but also, and mostly, because I can.

The post Make the most of solar power appeared first on ClickedyClick.

Just a while ago, it had its first sale (hopefully the first of many:). *champagne!*

I guess it is a very humbling experience that it took some 5 days of promotion effort to make that sale. That is even before I count how much time I have spent converting the Word doc into Markdown, and tweak the looks. The technical details to that I have written up in a guest blog post on the books website.

The marketing side is still under development, and the numbers are too low to draw a lot of conclusions from. The most surprising thing to me is how many people actually click the “buy this book” link and then they don’t follow through. Then also the half a dozen people who I followed up on earlier discussion to tell them about the ebook version, they say “ah, it’s awesome, I’ll go get it, I’ve been waiting for it” – then nothing happens.

Of course, there can be a lot of reasons for this, here are some of my guesses:

• The website UX does not work well for them (ie. the are put off by the process)
• People don’t know/like Leanpub
• People are lazy
• The price is too high
• The sales copy is bad, so people don’t think a business book can worth that much for them (ie. price is perceived to be too high)
• Spreading the word at the wrong places, thus wrong audience

… and probably a hundred other reason that I wouldn’t think of – or 171, since the book’s page had that many unique visitors in the last few days who left empty handed. I wonder what would be a natural next step to improve on this conversion rate.

It was a very fun thing to do, though, and got me psyched up to get my two idea-stage books going. And about ebook publishing in general, will definitely try to get more people onboard who have writing tendencies.

Also, this invaluable learning, as I’m setting up my startup now, a good reminder that people will not easily/often buy into my “awesome” idea, will have to work cleverer on that. Rejection therapy in its earnest.

(Disclaimer: I have no financial interest in the book,  ie. I won’t get any of the sales, which makes things even nicer. I received one original paper copy last spring to review and edit.)

The post First sale of an ebook I helped to publish appeared first on ClickedyClick.

There were 8 speakers scheduled from different companies and background: RS Components, Via, Broadcom, Motoduino, TMI Holding…. and me from Taipei Hackerspace. First I was wondering how do I fit in there, and maybe my talk *was* out of place a bit. Most talks were in Chinese so I could grasp only basic stuff from them, although the slides helped – most people made slides following the “slides are my notes” style, which is not my style, but was welcome this time. It was also great to see seasoned speakers like Richard from Via, and Lucas from TMI giving fun (and informative!) talks.

I did feel I’m in the right company, though. Open Source Hardware is becoming more and more of my focus – or maybe I’m just realizing it now that what I do is called such. Part of the audience were industry professionals and part enthusiastic hobbyists & students, and I had some great chat in the breaks with both kind of people.

I heard about the National Science Fair here (what? I got to check that out!), and how much interesting work people do with interactive hardware and elementary/high school kids. There is waaaay more going on that I can imagine, and there’s a lot more potential to tap into.

Had a chance to gather some industry experience too. Naren from Broadcom, who was responsible for getting Raspberry Pi into production was telling the story how they were expecting only 10,000 orders altogether and got 350,000 on the first day (sold 1.8 million to date), so they had to scramble an entire new supply chain. Thought that came to me was that maybe that’s one of the biggest value of  Kickstarter/Indiegogo is to be able to get an order of magnitude estimate of the demand.

Also heard about Via’s experience in pursuing new design and materials with their APC platform (such as using paper for housing), and interacting with their community. Heard a lot from RS Components how they are building tools for their community, and building the community itself. The demo was fought with technical difficulties (first rule of presenting is not to assume a working/fast internet connection), but it was inspiring nonetheless, and gave me some big (and difficult) ideas for my project: have to see if there’s any good community for scientific/laboratory electronics and hardware building, and if there isn’t then build one.

One of my favorite part in these events though to meet friends’ friends. This time was no exception, even if there was too little time to talk, I left with quite a few people in mind who I will need to contact soon, because they are doing something awesome and connected to my area of interest, either in hardware or in startups. This is one reason I’m trying to be really generous with my time and making introductions between people who I recon would hit it off well: I was the recepient of that so many times that I got to give back too.

My talk

I was scheduled in the middle of the afternoon just before tea break, and I was one of two people who used English for their talk (out of the 8 speakers). I was more nervous

I’ve uploaded the slides used for the talk in advance as well, so now it’s all out in the open.

My favorite part writing this talk was probably the attempt to summarize the philosophy and values of the maker movement, in a way that would inspire others. Some bits:

• Don’t accept crappy – everything can be changed and improved upon.
• Aim for collaborative creation. Celebrate the weird. Don’t mock.
• Do and then share the results for everyone to learn from it.
• For things to happen, you have to show up. Don’t wait for someone else to start, build up and inner motivation
• Everyone’s values are different, a ‘space is often a different canvas for everyone.

So far the feedback I got about the talk is that I should have mentioned the projects people were working on in the ‘space, and upcoming events. That would have made a better ending for sure, and I had an extra 5 minutes or so to do that. Definitely going to emphasize the practical aspect next time.

Another thing I noticed listening to (a bit of) the video is that I need to use much less “ehm” and “ahm”… I certainly don’t remember using any, and consciously trying to avoid it in my talks in the recent years, I guess I need to listen more carefully (and prepare better).

Any more suggestions? What else wasn’t good? Bring it on, I want to become better at this.

More about the Taipei Hackerspace is on the mailing list, which is open for everyone to sign up, ask questions, show their projects, and hear more about what’s up.

The post Taiwan’s Open Hardware movement gets a boost appeared first on ClickedyClick.

Originally all I wanted is just some laid back pressure recording, so maybe I can use that to predict the weather a bit. “Pressure falls: bad weather comes, pressure rises: things will clear up”. I was recording for about a week, and nothing really noteworthy came out of that.

Then it was the news, that the year’s first typhoon is on the way to Taiwan, and it was supposed to be a big one. Obvious that I will try to record the barometric pressure pattern of its passing, but wanted to make it more interesting and informative. More visual than just the timeseries plot of pressures.

The Japanese Meteorological Agency (JMA) is a good place to watch for information about typhoons. They list path prediction, typhoon properties like strength, wind speeds, and central pressure, have satellite imagery. Putting these together, two days before the typhoon arrived, I set up a script to download the satellite imagery as it became available.

The JMA publishes usually 2 satellite images in an hour for our North Western Quadrant (at :00 and :30), one of them covers the whole area, the other covers just the top 80% or so, leaving a dark band on the bottom. Nevertheless, matching up the pressure reading with the satellite pictures would be a good little project for this time.

Friday came, the government gave the afternoon off, though it turned out no landfall happened till everyone supposed to be off anyways, just a bit of on-and-off rain. People stocked up on convenience store food (I now have a good supply of instant noodles:) and water, taped over their glass windows, take in their plants and BBQ equipment from outside – well, those who have planned.

Around 10pm the big rain has arrived, here’s a video of how it looked from my window. Went to sleep later, and got woken up around 3:30am by the rain having changed into pretty darn big wind. Here’s another video of the violent part of the typhoon that time in the morning, that doesn’t even really do it justice. The houses around here are pretty tall, and I wonder if they have protected from the wind, or been artificial canyons channeling it. Some things got broken, though not as much as I expected – which is a very good thing.

In the meantime by the power of the Internet I have checked out the pressure reading, how is it going a few miles away in the Taipei Hackerspace, where I have left the barometric pressure sensor (the geolocation is 25.052993,121.516981)

Here’s the entire recording of the approximately 2 days of typhoon. It was pretty okay weather in the start and end of the plot.

The readings have been corrected to sea level (from about 20m height, where the Taipei Hackerspace is), should be good within 1hPa or less.

The the pressure was indeed dropping like a rock, and the dip on the graph coincided with the most violent wind that woke me up. According the JMA, the central area of the typhoon had pressures down to 950hPa, which means that core must have passed pretty close to here, having readings below 958hPa, though probably not directly, as it didn’t stay down there for long.

I made a video syncing up the pressure reading and the satellite picture. The red dot on the video marks the recording location. (Watching it in full screen and HD makes it clearer.)

I would wonder what was the flat part in the readings while the typhoon was leaving. Maybe sign of changing direction, by the look of it.

Either way, this was fun to do, and I am glad that only a few people got hurt here, much fewer then even during the less powerful typhoons. Maybe getting people scared a little (like with this “super typhoon” stuff that went on) helps them keep safe? Just don’t use it too often.

Extra material

I put almost all material used here into a gist: the satellite imagery download script, the plotting, the movie frame generation, the movie generation script, and the complete barometric recording. Because this last part is pretty big (5Mb), Github truncated the rest of the scripts. I guess it’s okay to check check it out. Will add the Arduino sketch to read the sensor and the logging script later.

The satellite imagery weighs about 60Mb, so don’t put it online, but if anyone wants them, let me know.

Keep safe!

The post Barometric recording of Typhoon Soulik appeared first on ClickedyClick.

Of the two events, this one was the trickier to do. For Create@Public, all I had to do is pack a lot of stationery and start making stuff myself. For a hackathon, there’s much more preparation to do. Fortunately, I had some great mentors to get things going, James of Startup Digest Taipei and Volker from Yushan Ventures. They have a lot of experience pulling off great events, and I was glad to hear their advice. I was surprised that in less than 2 weeks something like this could be put together, even under not totally favorable circumstances.

First had to find a place to host it, and preferably with zero budget. Looking at who do I know, ended up at appWorks, a startup accelerator, whose founder, Jamie is indeed a hacker at heart, so wasn’t actually that bad to convince him to give us some space on a weekend day. Even got one of their teams working there, Fandora, to help us.

That’s a good first step, now have to get some people to participate. I set up the Hack+Taiwan blog (on Octopress, just trying something new, that was an “interesting” experience as well), the event, a sign up sheet and started to spread the word. It all went well, until one week before the event it turned out that the Open Source Developers’ Conference has a hackathon pretty much the same day & time as I planned. That freaked me out a little, and after asking around, I got many different advice: cancel so not to compete with them, cancel to take a rest, and give it my best try nonetheless. Of course the craziest option won, so started to spread the word even more, trying to invite mentors, getting catering, figuring out how the day should work and so on.

Let’s get to it

Since people don’t like to fill out sign up sheets, and Facebook event “join” is so easy, I had absolutely no idea how many people will come. It was just winging it, I knew I’ll be there and a couple of people who were helping me, but that’s all so far.

I ordered breakfast for 30 (from Magic Bagels), and let’s see what happens. Around 9am, the advertised start time, people started to come, first a couple, then some more, and around 10 o’clock there were almost a dozen people. That’s not bad at all, even considering that a third of that was mentors and other organizers, but they took part just the same. Since most people didn’t bring any ideas (some even didn’t bring a laptop, now that I don’t understand), had to get them to come up with some stuff on the spot. It took a little pushing, some ideas were so specific and pretty much the same they are working on their normal time, but fortunately there were some interesting ones. In about half an hour we had 3 teams working on three completely different. In the afternoon there were even more people who dropped by for some time, and got another project working.

The projects we had:

• Mobile reminder app + website
• Android programming (the team had to leave before demo time)
• Visualization of the Taiwanese power grid for monitoring and supervision
• Arduino piano

At 5:30pm we had a demo time, everyone showing off what they made, people switched to speaking Chinese since I was the only foreigner around, so I got to train a little better next time to be able to follow tings. I could see their demos, though, and it’s so impressive how far people got in pretty much 6 and half hours….

Here’s the photo album of what went down. I’ve also taken a timelapse image series of the event, been planning to make a video of it….and then successfully overwritten all the photos with a bad command of ffmpeg. That’s my way.

Lessons learned

• Coming up with ideas is hard
• It is still possible to come up with ideas. Look for issues that kept bugging you for a while, and do something about those
• Everyone gets out different thing from these events
• Don’t force things, everything will work out anyway
• Don’t have to overplan, before one establishes their name in the community as good organizer, it is usually more likely to overestimate how many people will come to an event
• Everyone loves bagels and cake (okay, that’s not new lesson)
• Unless there are enough people, don’t have to focus the topic of a hackathon, see whatever people come up with. More difficult brainstorming, but better chances of success
• Don’t drink too much coffee – I couldn’t get much done from the shakes

I really liked the feedback I had from the participants, and also from other people: there were some who were just working in the background on their own stuff, and when we finished they came and asked what was it, because they’d like to take part next time. That’s the spirit! And we might just do that, for example at the end of the summer. In the meantime, I will try to focus on earning back my own hacker badge.

The post Hack+Taiwan appeared first on ClickedyClick.