Author: Gergely Imreh

Physicist, hacker.
Enjoys avant-guarde literature probably a bit too much.
Open source advocate and contributor, both for software and hardware.
Follow these posts on the Fediverse by @gergely@gergely.imreh.net

Categories
Programming

Automating the hell out of it

Even before the 4-Hour Work Week made me more serious about this, I really enjoyed automating tasks, that benefit from not needing to remember to do, or would be troublesome to do otherwise. This frees up a lot of time, keeps a bunch of problems away, and it is actually quite fun when the information comes to me instead me going to it.

Now I have automated checking my bank account and credit card balance, updating dynamic IP of server, ebook sales numbers, and network clock synchronizing. There are some general ideas that I summarize, then give an intro to all of those scripts.

Banking script

Tools

Most of my scripts are written in bash, because it’s relatively straightforward to hammer out simple stuff, and it is surprisingly simple to do a lot of things once I have thought enough about a problem. The Advanced Bash-Scripting Guide is always on my reading list, but I usually get to check only the parts that are relevant to the given problem. You can get quite far with a few simple constructs.

The most common parts I seem to come across:

  • if-then-else constructs: if [ -f ‘directory ‘]; then echo “Found!”; fi
  • for loops: for f in *.png; do optipng $f; done
  • loading the results of a command into a variable: VAR=$(command)

For most other problems with a little keyword-fu there’s always an answer on StackOverflow or on the web.

Another group of scripts uses Python, when a bit more data-manipulation is needed, like web scraping or JSON parsing. Actually, all of the scripts could be rewritten in Python for consistency, and it would probably be be simpler too, which is something for the future.

As a general tip, most of these scripts need tweaking, and all of them are sort of alpha-beta quality code. To facilitate hacking and reduce heartache of mangled clever code, I keep everything in git repos. I share those repos online, so have to make sure there are no secrets checked in, ever. It helps to strategically use .gitignore, separate files for the secrets, and having an example how that secrets file should look in the inside.

Most of these scripts are run periodically by cron, so it is worth having some basic knowledge about how to schedule it.

Some scripts send me emails under specific circumstances (some after every run, some when new information appears), and for good delivery I have set up postfix to use Gmail as an SMTP relay. This way I’m sure to receive the emails and receive them quickly.

Scripts

These are the scripts I use most often and the longest. Still, many of them are under development and adjust them whenever I learn how to do things better. I list the links to all their repos, where it can be improved.

Banking account balances

My two main bank accounts are queried once a day for available balance and I’m notified by email. Both accounts needed quite a bit of web scraping (and got them done at two different OpenHack Taipei events). The banks’ websites are pretty awfully organized (iframes within iframes within iframes; not using CSS classes and id), though it doesn’t have to be good for me, it has to be good for the bank.

Cathay United Bank

The cathaycheck (click for repo) script queries the available balance at Cathay United Bank by logging in with curl, and parsing the final page with Beautiful Soup. The script can be a skeleton for any other website where on has to log in and then navigate over a series of pages to get the information. The required HTML variable names can be extracted with the help of the Inspect Element tools in Chrome.

At the moment the credentials is stored in the crontab command, which is not really ideal, should rewrite to use a secrets file, though given that it runs on a server where I’m the only user (and root), for me there’s no practical difference at the moment. I have set it up to receive an email at the end of the day with the current balance.

ANZ Taiwan credit card

The anzcheck (click for repo) script queries my spending with the ANZ Taiwan credit card. Again bash for logging in and Beautiful Soup for parsing the final page. It needs a bit more logic extracting information from a table, because the websites developers added no classes or ids to the items to make it easier to understand – or for them to style, but that’s not my problem.

Just recently updated that it extracts the spending items added to my balance on a given day, so I can will never be caught by surprise again (hopefully). Since many of my charges go to companies that have Chinese names, I quickly run into the problem of having to tell my Heirloom Mailx (that I use to send emails on my ArchLinux box)  that the text I want to mail is plain text, not an attachment. With some hacking the solution was to add a few more commands to “mail” so it knows that the text is UTF-8. From “sendthatmail.sh” in the repo, the parameters needed are:

-S sendcharsets=utf-8 -S ttycharset=utf-8 -S encoding=8bit

I could still extract some more information from the bank’s website, though nothing really urgent.

No-IP address updater

At the Taipei Hackerspace we have a handful of servers running, but the residential internet connection is provided by Chunghwa Telecom only gives us a dynamic IP address. Applying for a static IP seems to be pretty troublesome, so in the meantime I’m using a script on one of the servers to update the IP address associated with our dynamic tpehack.no-ip.biz address.

The no-ip-bash-updater (click for repo) script is forked originally from elsewhere, but I have rewritten it quite a bit so that it

  • needs no extra file to store the current IP address, but compares external IP with a DNS query
  • stores no secrets in the file

It uses a pretty straightforward API call with HTTP authentication, the only real logic in there is to check when that call actually needs to be made.

E-book sales

Recently I have helped a friend to publish an ebook version of How to Start a Business in Taiwan on Leanpub, and of course I want to know when there are any sales are made (disclaimer: I don’t get a cut of the sales, all goes to the author). The leanpubsales (click for repo) script is written in Python, because using JSON there is easier than it would be with bash. The call otherwise is quite simple, just keep an external file around to check if the sales number have increased or not, if yes then send an email. To send an email conditional on the output the the script the “ifne” command from moreutils is very useful (meaning: “if input is not empty”).

The query is run periodically, and lovely to receive the results. I will surely set up a script when I get my own book ideas published on Leanpub.

RTC correction

As a physicist in atomic physics, which is the area of science very much concerned about keeping precise time, keep all my servers’ times synchronized with network time protocol (NTP) using chrony. One difficulty is that the real-time clock (RTC) of those computers is pretty crappy and drifts away. Wouldn’t be a problem if I never restart them, but a pain if I do: after restart it can be tens of seconds away until the time is synchronized again.

Chrony can sync NTP and the RTC, but it doesn’t do that automatically, I have to trigger it manually. Instead I have written up an rtccorrect (click for repo) script that is run every 2 hours or so (could be done just once a day, actually), and eliminates the drift of the RTC.

Server backup

For backing up data between servers rsync has proven invaluable. I have a couple of scripts that do just that, though those are among my oldest ones and at that time I haven’t separated out personal information (way too easy to inline every credential, email, login, and all that), so I need to sanitize that. A couple of  ideas about these backup scripts:

  • sometimes higher transfer speed can be achieved by messing with the ssh algorithms, eg. passing “-e ‘ssh -c arcfour'” to rsync
  • more often there’s even better performance when there’s an rsync daemon running on the remote computer (though with Raspberry Pi, both cases are still frustratingly slow)
  • can exclude some files if no need to transfer them, eg: “–filter=’- *.part'”
  • using rsync not just to transfer but to mirror, the “–delete” (delete at target if doesn’t exist at origin) and “–archive” are pretty useful

For these backups I also use the Dead Man’s Snitch to know when things didn’t work out, e.g having a similar command in the cron list, where backup.sh is my script’s name, xxxxxxxx is the snitch ID from my account:

backup.sh && curl -s https://nosnch.in/xxxxxxx > /dev/null

This way I got to know when my backup server was dying all the time because of bad heatsink, or my host server by flaky hosting company….

Afterword

I guess there will be just more automation in the future, and maybe many of these scripts can be ported onto a common base so new ones are made much easier. What else do you guys automate?

Categories
Startups Taiwan

First sale of an ebook I helped to publish

Recently I was helping a friend to publish the ebook version of his paper book, called How to Start a Business in Taiwan. About 50% the reason was that I like the book and hoped to get into more people’s hands, and 50% I wanted to see what is it like getting an ebook out the door and bought by people.

Just a while ago, it had its first sale (hopefully the first of many:). *champagne!*

How to Start a Business in Taiwan on Kindle
How to Start a Business in Taiwan on Kindle

I guess it is a very humbling experience that it took some 5 days of promotion effort to make that sale. That is even before I count how much time I have spent converting the Word doc into Markdown, and tweak the looks. The technical details to that I have written up in a guest blog post on the books website.

The marketing side is still under development, and the numbers are too low to draw a lot of conclusions from. The most surprising thing to me is how many people actually click the “buy this book” link and then they don’t follow through. Then also the half a dozen people who I followed up on earlier discussion to tell them about the ebook version, they say “ah, it’s awesome, I’ll go get it, I’ve been waiting for it” – then nothing happens.

Of course, there can be a lot of reasons for this, here are some of my guesses:

  • The website UX does not work well for them (ie. the are put off by the process)
  • People don’t know/like Leanpub
  • People are lazy
  • The price is too high
  • The sales copy is bad, so people don’t think a business book can worth that much for them (ie. price is perceived to be too high)
  • Spreading the word at the wrong places, thus wrong audience

… and probably a hundred other reason that I wouldn’t think of – or 171, since the book’s page had that many unique visitors in the last few days who left empty handed. I wonder what would be a natural next step to improve on this conversion rate.

It was a very fun thing to do, though, and got me psyched up to get my two idea-stage books going. And about ebook publishing in general, will definitely try to get more people onboard who have writing tendencies.

Also, this invaluable learning, as I’m setting up my startup now, a good reminder that people will not easily/often buy into my “awesome” idea, will have to work cleverer on that. Rejection therapy in its earnest.

(Disclaimer: I have no financial interest in the book,  ie. I won’t get any of the sales, which makes things even nicer. I received one original paper copy last spring to review and edit.)

Categories
Life

On Going Fast Nowhere

I do feel like I’m deceiving myself. It feels such that the problems came and pulled me under even before they existed. Like time-traveller problems: come back from the future and they prevent their own birth, by making me put off entrepreneurship, and look for something else. Like open a coffee shop. Or go back being a post-doc…

It’s of course very harsh on those problems, since it’s me who created them, by becoming the not-at-all mythical “wantrepreneur”. I keep talking that much that I convinced a lot of people that I am up to more than I am actually accomplishing. It’s not for the lack of trying, mind you…. depending on the definition of “trying”.

If reading stacks of books about the subject of entrepreneurship counts as trying, then I’m pretty darn hard at work. If networking, or going to events, hearing other people’s experience, asking them for advice counts as trying, I’m still right there in the middle of the pack. If the only thing that counts is whether I made a sale of any kind – well, I got nothing. Let’s even be more generous, count learning as trying in the Lean Startup way – still got nothing. Have plans to have more than nothing, but those “urgent” plans are at the same stage as they were a month (maybe even two months?) ago.

Main conference room on the second day with speakers
APEC Startup Accelerator Leadership Summit 2013

This is all not because I’m stupid, I’d like to believe: my shoelaces are correctly tied  (just checked now), I regularly converse in languages other than my native tongue, and I can keep up a conversation pretty well, according to others. But still, sitting at Starbucks, just having put down the most recent book I’m reading of “this is how you do your business”, I have to ask myself – why am I doing this, and what’s “this”?

Maybe it’s all just fear, fear of the unknown future, where those time-traveller problems come from. Or maybe I don’t care enough. Or maybe I’m holding it wrong. Or maybe I’m just taking this all too serious? Premature optimization of things, before I haven’t tried anything? Waiting for others to show my path, and in the same time rejecting pretty much everyone else’s opinion as they cannot possibly understand and have the same amount of passion to my idea(s) as I have? That even if I yearn for the opposite, I’d rather spend an evening in with the sci-fi novel I’m reading (and which is brilliant so far), than push the limits by testing out the great ideas I convinced myself to have come up with?

My guess: all of the above, and then some. So many rhetorical questions, good grief…

That’s the current status, being full of aspiration and total helplessness. Seeing a lot of great examples, and thinking “I could do that!” and “I could never do that!” in the same time. What causes this? Can all this reading and preparation instead of helping, just piled up inside of me as intellectual pus? That might fit – something that tries to help me, and I have to get rid of to be okay.

Okay, boy. Let’s try something different: I’m calling in a learning ban, and get out to where the real action is, where the real fighting goes on with real bullets. What good is your runway, if you never spin up the propellers to take off?

(May I still finish the one I’m reading now? ….  Fine, but that’s the last one.)
(What if I could avoid some common mistake if I just have read X by Y? ….  Fuck it, use use your common sense instead)
(But, but ….  No but, can go back to your books when you have something exact to learn. You know enough. )
(H…..  Shut up.)

 

Categories
Maker Taiwan

Taiwan’s Open Hardware movement gets a boost

I sure felt different yesterday morning waiting for my turn to give my 30 minutes talk, but in retrospect I’m really glad to have been invited to the Boost Open Source Hardware Movement event, organized by the CTIMES magazine over here in Taiwan. It was the second time try, after Typhoon Soulik cancelled the original event.

CTIMES personnel introducing the Boost Open Source Hardware Movement event
Kicking off the event early Saturday morning

There were 8 speakers scheduled from different companies and background: RS Components, Via, Broadcom, Motoduino, TMI Holding…. and me from Taipei Hackerspace. First I was wondering how do I fit in there, and maybe my talk *was* out of place a bit. Most talks were in Chinese so I could grasp only basic stuff from them, although the slides helped – most people made slides following the “slides are my notes” style, which is not my style, but was welcome this time. It was also great to see seasoned speakers like Richard from Via, and Lucas from TMI giving fun (and informative!) talks.

I did feel I’m in the right company, though. Open Source Hardware is becoming more and more of my focus – or maybe I’m just realizing it now that what I do is called such. Part of the audience were industry professionals and part enthusiastic hobbyists & students, and I had some great chat in the breaks with both kind of people.

I heard about the National Science Fair here (what? I got to check that out!), and how much interesting work people do with interactive hardware and elementary/high school kids. There is waaaay more going on that I can imagine, and there’s a lot more potential to tap into.

Had a chance to gather some industry experience too. Naren from Broadcom, who was responsible for getting Raspberry Pi into production was telling the story how they were expecting only 10,000 orders altogether and got 350,000 on the first day (sold 1.8 million to date), so they had to scramble an entire new supply chain. Thought that came to me was that maybe that’s one of the biggest value of  Kickstarter/Indiegogo is to be able to get an order of magnitude estimate of the demand.

Also heard about Via’s experience in pursuing new design and materials with their APC platform (such as using paper for housing), and interacting with their community. Heard a lot from RS Components how they are building tools for their community, and building the community itself. The demo was fought with technical difficulties (first rule of presenting is not to assume a working/fast internet connection), but it was inspiring nonetheless, and gave me some big (and difficult) ideas for my project: have to see if there’s any good community for scientific/laboratory electronics and hardware building, and if there isn’t then build one.

View of the room during the break time
Break time between the talks, everyone’s scrambling for chat and cakes

One of my favorite part in these events though to meet friends’ friends. This time was no exception, even if there was too little time to talk, I left with quite a few people in mind who I will need to contact soon, because they are doing something awesome and connected to my area of interest, either in hardware or in startups. This is one reason I’m trying to be really generous with my time and making introductions between people who I recon would hit it off well: I was the recepient of that so many times that I got to give back too.

My talk

I was scheduled in the middle of the afternoon just before tea break, and I was one of two people who used English for their talk (out of the 8 speakers). I was more nervous

I’ve uploaded the slides used for the talk in advance as well, so now it’s all out in the open.

My favorite part writing this talk was probably the attempt to summarize the philosophy and values of the maker movement, in a way that would inspire others. Some bits:

  • Don’t accept crappy – everything can be changed and improved upon.
  • Aim for collaborative creation. Celebrate the weird. Don’t mock.
  • Do and then share the results for everyone to learn from it.
  • For things to happen, you have to show up. Don’t wait for someone else to start, build up and inner motivation
  • Everyone’s values are different, a ‘space is often a different canvas for everyone.

So far the feedback I got about the talk is that I should have mentioned the projects people were working on in the ‘space, and upcoming events. That would have made a better ending for sure, and I had an extra 5 minutes or so to do that. Definitely going to emphasize the practical aspect next time.

Another thing I noticed listening to (a bit of) the video is that I need to use much less “ehm” and “ahm”… I certainly don’t remember using any, and consciously trying to avoid it in my talks in the recent years, I guess I need to listen more carefully (and prepare better).

Any more suggestions? What else wasn’t good? Bring it on, I want to become better at this.

More about the Taipei Hackerspace is on the mailing list, which is open for everyone to sign up, ask questions, show their projects, and hear more about what’s up.

Categories
Thinking

Beyond the Kickstarter model for science crowdfunding

In the startup world people pitch ideas quite often in the shorthand of “we are X for Y”, drawing on an existing (and successful) “X” and a new market or audience “Y”. It’s very convenient, while have to be extremely careful not to be sucked in by its simplicity and really equate it with “we are doing everything completely the same as X, just for Y”. One such combination where I consider that very harmful is the topic of science crowdfunding, and the multitude of sites being “Kickstarter for Science”.

The Kickstarter model works very well, when there are things to give away, when the end results are physical products, or can produce something physical related to the project (for example postcards and DVDs for performance and art).  For most of the science projects, that just doesn’t work, quite often there’s nothing really to give away for a wide audience, that would attract enough funding (though will come back to this later). I think the whole model of what do supporters get needs to modified, and on a higher level, have to answer the question of “What is the purpose of all this?”

Laser sign built for the lab
Laser sign built for the lab

To get to some possible answer, first a bit of exploration. One of the first things to see is who could be the beneficiary of the project? The star professors working at big universities, they often have more money from grants than they can spend, probably they are not the first ones to think of. On the other hand, there are smaller colleges with ambitious researchers and students. There science fair students. There are independent scientists, hackerspaces, non-profits. The funds for many of these are a problem, but there are even more common problems I heard so far: lack of knowledge and lack of community/connections. The latter can affect very successful professors as well, in less well connected places (like here in Taiwan).

Another insight I got from watching Zack Braff’s reply to his critics, over his Kickstarter campaign for a new movie project. In that video he mentions what got him onto Kickstarter in the beginning: being able to have a backstage view to awesome projects, by receiving first-hand experience of the makers. That’s what he put at the centre of his campaign too: (paraphrasing) “If you fund me, I’ll be able to show you how a movie is made from beginning to the end, like you’ve never seen it before” (and apparently 46,520 people thinks it’s an awesome idea, myself included). Maybe that backstage view would be motivating to just enough people to support a particular science project that they are interested in?

So to come back to the question of purpose: what if the driving principle was “Promoting and enhancing Open Science“? Would this be a focused enough mission to create a sustainable model?

Fitting this into the previous examples: what if science fair students can get support for their projects: monetary from family and community who has such vested interest in them, but beyond that, the project would also try to connect those students with professors in the field their project is, while providing feedback and education to do their projects in a transparent, responsible, and interesting way. Besides the benefit of getting their projects done, could this create better scientist and more interested public in the long run?

Similarly for independent scientists and professors alike, openness, collaboration, and more transparency could enable them to do better and more science. I feel, as Tim O’Reilly said about writers, for scientists as well the problem is not that someone will steal your idea, but that nobody have heard about what you are doing.

That’s not to say, that scientists should spend all their time promoting their research, but to make it accessible. People should be able to see how a great lab-book looks like, what raw data is, what is a failed experiment and why did it fail, see how long certain things take in the lab, what is the thought process of a discovery. This is of course some overhead – when people are writing things up, they are not doing the experiment. On the other hand, I am a firm believer, that taking some time to think and being “forced” into putting thoughts down into coherent form has an amazing positive benefit on the long run. That’s how real understanding can come about too, and it will make a better scientist, compared to putting short-term efficiency as top priority.

All in all, I think a platform that have a priority list of first Openness, then Collaboration, and finally Funding would work the best for the mission of enabling more people doing more and better science. Some sign of this I can already see from existing places, like the experience of the Serengeti Live project which found collaboration with Zooniverse.

There are a LOT of science crowdfunding websites (as I’ve seen many on a list on Reddit), though many of them are closing down or inactive as founders moved on, like IAmScientist, Petridish. There are others still going on, for example Microryza, which I admire as founded by scientists since they wanted to get funded as well, though I think the reward of nicely formatted report at the end of each funded project falls short its potential.

Who will build something based on this and will it work? I don’t know yet, but I’ll sure do everything I can for Open Science and discovery in the meantime, and work to become a better scientist myself.