Category: Computers

Categories
Programming Taiwan

Barometric recording of Typhoon Soulik

It all started a few weeks ago with Sparkfun having “20%-off” day, when I got myself (among other things) a BMP085 barometric pressure sensor. When it arrived, I have soldered some pins on it, and set it up with an Arduino Nano, to have the readings off it easily.

View of the circuit
BMP085 barometric pressure sensor breakout board from Sparkfun

Originally all I wanted is just some laid back pressure recording, so maybe I can use that to predict the weather a bit. “Pressure falls: bad weather comes, pressure rises: things will clear up”. I was recording for about a week, and nothing really noteworthy came out of that.

Then it was the news, that the year’s first typhoon is on the way to Taiwan, and it was supposed to be a big one. Obvious that I will try to record the barometric pressure pattern of its passing, but wanted to make it more interesting and informative. More visual than just the timeseries plot of pressures.

The Japanese Meteorological Agency (JMA) is a good place to watch for information about typhoons. They list path prediction, typhoon properties like strength, wind speeds, and central pressure, have satellite imagery. Putting these together, two days before the typhoon arrived, I set up a script to download the satellite imagery as it became available.

Satellite picture of Typhoon Soulik and location of Taiwan on 2013-07-12 morning
The morning before the typhoon arrived

The JMA publishes usually 2 satellite images in an hour for our North Western Quadrant (at :00 and :30), one of them covers the whole area, the other covers just the top 80% or so, leaving a dark band on the bottom. Nevertheless, matching up the pressure reading with the satellite pictures would be a good little project for this time.

Friday came, the government gave the afternoon off, though it turned out no landfall happened till everyone supposed to be off anyways, just a bit of on-and-off rain. People stocked up on convenience store food (I now have a good supply of instant noodles:) and water, taped over their glass windows, take in their plants and BBQ equipment from outside – well, those who have planned.

Around 10pm the big rain has arrived, here’s a video of how it looked from my window. Went to sleep later, and got woken up around 3:30am by the rain having changed into pretty darn big wind. Here’s another video of the violent part of the typhoon that time in the morning, that doesn’t even really do it justice. The houses around here are pretty tall, and I wonder if they have protected from the wind, or been artificial canyons channeling it. Some things got broken, though not as much as I expected – which is a very good thing.

In the meantime by the power of the Internet I have checked out the pressure reading, how is it going a few miles away in the Taipei Hackerspace, where I have left the barometric pressure sensor (the geolocation is 25.052993,121.516981)

Here’s the entire recording of the approximately 2 days of typhoon. It was pretty okay weather in the start and end of the plot.

Plot of pressure readings
Pressure reading during the passing of Typhoon Soulik, recorded at the Taipei Hackerspace

The readings have been corrected to sea level (from about 20m height, where the Taipei Hackerspace is), should be good within 1hPa or less.

The the pressure was indeed dropping like a rock, and the dip on the graph coincided with the most violent wind that woke me up. According the JMA, the central area of the typhoon had pressures down to 950hPa, which means that core must have passed pretty close to here, having readings below 958hPa, though probably not directly, as it didn’t stay down there for long.

I made a video syncing up the pressure reading and the satellite picture. The red dot on the video marks the recording location. (Watching it in full screen and HD makes it clearer.)

I would wonder what was the flat part in the readings while the typhoon was leaving. Maybe sign of changing direction, by the look of it.

Either way, this was fun to do, and I am glad that only a few people got hurt here, much fewer then even during the less powerful typhoons. Maybe getting people scared a little (like with this “super typhoon” stuff that went on) helps them keep safe? Just don’t use it too often.

Extra material

I put almost all material used here into a gist: the satellite imagery download script, the plotting, the movie frame generation, the movie generation script, and the complete barometric recording. Because this last part is pretty big (5Mb), Github truncated the rest of the scripts. I guess it’s okay to check check it out. Will add the Arduino sketch to read the sensor and the logging script later.

The satellite imagery weighs about 60Mb, so don’t put it online, but if anyone wants them, let me know.

Keep safe!

Categories
Programming

I got my Pebble, now let’s play

Just received my Pebble smartwatch yesterday, a bit more than a year after the Kickstarter campaign has ended. Its pitch is being a watch with e-paper display, Bluetooth communication with Android and iOS, a Cortex-M3 ARM processor, with the ability to run programs that change the watch look (“watchfaces”) or provide extra functionality (“watchapps” ).

I almost forgot about that I have supported it, until my friends on Facebook started to receive theirs, and I run into a person recently wearing one (the red coloured one, mine is gray and that was manufactured later). Even if I’m not really a wristwatch person – haven’t had one for years -, I like watches in general and clever new tech all is always welcome with me. (I wonder if any of these will be  in pocket watch form some day).

Pebble watch showing the time, 2:45
Pebble watch in action. Or just chilling and showing the time anyways.

Setting it up was pretty easy, the Pebble app on Android does pretty much everything. Setting up notifications for emails, Facebook messages, calls and SMS is one of the first thing. Until I get overwhelmed them and will turn them off, probably. The app also sets up such that when my phone downloads a file with the right extension (.pbw), it will automatically sent to the connected Pebble. The updates are pretty easy and quick too. I got a bunch of different watchfaces from My Pebble Faces, an unofficial repository that has much more than the official one. Maxed out around 10 or so, then had to use the app on my phone to remove some of them. Easy, though now really feel it all depends on the phone I use as well, the watch is mostly as clever as the accompanying phone app.

Fooled around with it a bit more, checking the alarms, the automatic Runkeeper integration (pretty neat, if only I could switch from “pace” display to “speed” for the watch), some small watchapps. In the case of watchfaces, I switched back to the original “Text Watch” face, which is pretty neat and unique for non-smart-watches. If only it could be modified to write things like “ten oh seven” for “10:07” instead of “ten seven”, it would feel more natural. There’s a watchface called Words o’Date that does that, except that it also adds the date, making the front look a bit too crowded for me. So keeping the original at the moment.

With this playing around I noticed one possible strength of the display, that almost everything can be animated, and that animation is pretty smooth. [Update: it’s actually e-paper, not e-ink, so my bad, the following comparison is not really valid Thanks for the comment.] From my limited experience with e-ink in the form of reading on my Kindle Touch, Pebble is really agile. Looking in the the Software Development Kit (SDK) documentation, there is a lot of functionality dedicated to animation.

After the whole day of usage I was thinking that when I was young(er), I would start hacking on interesting things right away, would have a lot of ideas, put in loads of effort, oftentimes stay up late to make something new to work. These days it’s less like that, even if I have much more toys lying around waiting to be hacked on. Around dinner time I decided that it cannot continue like that, so before going to sleep, will have my very on watchapp done.

Magic 8-Ball

Didn’t have many ideas, though, and haven’t looked yet too much, just wanted something simple. A Magic 8-Ball came up as a possibility: an app that gives you answers to your yes/no questions. It’s pretty easy in the core: a bunch of stock answers (20 in the original case), a random number generator to choose from them, and display it. Can do just bare text for the first time.

To set up my Linux box for the development, got the latest Pebblekit from Github, which is the SDK and examples and all the tools together. Unfortunately everything inside relies on “python” being Python 2.x, and on my ArchLinux this is not the case. The quickest workaround I found was to use Python’s virtualenv. I have also needed a different version of GCC that compiles for the Pebble’s ARM processor, the arm-none-eabi-gcc, fortunately in the ArchLinux User Repository (AUR). It just took a looooong time to compile. These two steps put me right into business and been able to compile the example watchapps and watchfaces.

I don’t remember much of my C skills (never had that much), that are mostly kept alive just by Arduino programming. Fortunately the Magic 8-Ball is simple enough program that looking at the examples, Stack Overflow, and Github, I could find all the pieces I needed (array of strings, random number generator in C, text display).

The Magic 8-Ball app is showing its result to my question: ask again later.
My first watchapp, the Magic 8-Ball

By around 1am I had a working version, and it’s totally fine. It’s even fun, even if very simple. People can get it directly from its page on My Pebble Faces, and apparently while I was writing this, 3 people already did, sweet!

In the future, I would like to keep up this creative hacking. For the Magic 8-Ball I could include some graphics, maybe neater transitions, make it less an “example” app and more an app. It’s really cool, that so many of the uploaded watchapps and watchfaces on My Pebble Faces also share their source code (50% of the 500 items, can filter for it in the search form), so I can learn from that. How much better Android apps would be if they’d do the same? My code is also on Github, naturally, in the magic8 repo.

For Pebble in general, communicating with the phone over Bluetooth can be very useful, if a useful Android app complements it. Smart clothing and accessories are just going to be more prevalent. There are already “next generation” smartwatches out there, like the Agent, that will be cool in a different way (I haven’t supported yet, they got 10x funding anyways).

Also, I think I should put some effort into compiling Libpebble for my laptop, that would make it possible to cut out the phone as a middleman and make things more future proof.

And as a first priority, I should be getting used to wearing a watch again, it’s been a while.

Categories
Admin

Switched to SPDY and now Google’s confused

Out of interest, I recently switched this site to SPDY, party because I like to try out new things, and partly because I would want to make things be better and faster. So far it’s a mixed experience, with some puzzling changes, that I cannot make heads or tails of.

The first step for the switch was bringing everything onto HTTPS, which I have done with a free SSL certificate from StartSSL. Redirected everything from the HTTP to the secure connection, with the 301 http code so I thought Google will be able to follow it well and replace the addresses in their index. Then enabled the SPDY module in Nginx, and checking the result looked like I was in business.

Some time has passed, and a scary graph started to manifest itself in Google Analytics:

Google Analytics impression count, the site has changed around May 8.
Google Analytics impression count, the site has changed around May 8.

Right after I have made the changes, my impression count on Google dropped like a brick, now being exactly 0. That’s not really the change I wanted to see. Digging more into it, though, it looks like I still have a constant stream of visitors from Google Search:

Visitor numbers from Google Search, same time interval as the impression count.
Visitor numbers from Google Search, same time interval as the impression count.

How can I have zero impressions, but still a half a dozen visitors from Search? The results in the Webmaster Tools mirror things: dropping impression count, no crawl errors, same or even better indexed count, and relatively good stats:

Google Crawler stats, with a big spike when switched over HTTPS/SPDY when needed to reindex everything
Google Crawler stats, with a big spike when switched over HTTPS/SPDY when needed to reindex everything

The crawl seemed to have gotten a bit slower (the bottom plot of the three), but more consistent.

I wonder what could be the change, does the impression count depend on the method of access (http/https)? Or did I made some braking changes? If so, then why’s the conflicting information?

Being a scientist, my main concern is not actually the raw value of any visitor count, but understanding the reactions to my actions, and consistency of the “experimental results”.  I wonder what kind of technique I could use to debug all this?

Update 2013/May/28: 

Following some recommendations from the comments, it looks like that the https:// version of my URL has to added to the Webmaster Tools separately. Now there’s a http://gergely.imreh.net and a https://gergely.imreh.net section as well. In the latter section, I can see that there are some impressions reported. Some weird things still exist: the sum of impressions from both is less than how many visitors I reportedly get from Google Search; the crawl stats is shared between the two sections (ie. the https version reports a lot of crawl stats even from the time there wasn’t https enabled), while most other data is separate for the two sections (e.g. impression, search queries, sitemaps). Still probably this is on the right path.

The impression count after adding a https version of my site's records to the Webmaster Tools
The impression count after adding a https version of my site’s records to the Webmaster Tools

After the Webmaster Tools changes, I have just switched the Google Analytics association from one WMT property to the other. Hopefully this will freak me out less, though it will likely take some days to see the changes in the result.

Categories
Computers Lab Programming

OSLO is a strange one

Experimental physicists have to be the jack-of-all-trades, even on a good day, but building a new laboratory, on a budget (for any definition of “budget”) and with a limited team, is an exercise that would test anyone.

For my field, atomic physics, one needs a lot of different expertise: vacuum systems, computers, electronics, machining, optics are among the ones first to my mind. Today I’m looking at one of the tools that we are using for the last topics: optics.

In some ways, optics is pretty straightforward (both geometrical and diffraction optics), many of the calculations one could write in Python/Numpy just as well, as any software can do it. They are just pretty tedious, and there are a lot of them, optics is not a new profession. Some optics design software is very useful then, and there are not that many of them, lots of work goes into the ones that exist, thus they can be extremely expensive. One of the industry standard seem to be Zemax, and many lens manufacturers provide the specs of their lenses in a Zemax format – though it is fortunately just a relatively simple text file. On the other hand, its cheapest edition is $2500, and goes up to $9500…

Another serious competitor is OSLO (Optics Software for Layout and Optimization) which is still very well known, and fortunately for us, has a dumbed down (limited number of surfaces, not all the functions available), EDU version, for free. It even runs on Linux with Wine.

I instinctively resist any software in science & research that is not free & open source, because I feel that it closes some people out, though had to use something, and in the meantime I have grown to OSLO quite a bit.

OSLO ray graph of an imaging system
One of our imaging system design

Using OSLO, I have found that one big advantage of these software is their catalog: the glasses (oh-very-important for optics) and stock optics of different companies. This alone makes everything much easier.

What does the lens designer software do? You enter the parameters of some lenses by defining different surfaces and materials & distances between those surfaces, give it some lightbeams, and see what comes out of it. Most of this happens in this window:

The window to enter the system parameters
Lens spreadsheet

After these settings, one can run a number of different analytics code, checking the performance and behaviour of the system by calculating a bunch of physics parameters, or can modify the system to optimize that performance one way or another.

This is where the number 1 strangeness – and big bonus – comes in: OSLO can be scripted in a language called Compiled Command Language (CCL), which is a subset of C; and not just scripted, but the whole program is written on top of the CCL compiler (as I understand)! The result of this is that:

  • Your scripting can be just as powerful as the entire program, since both sides have the same functions accessible
  • There are a bunch of scripts included in the main version and even if CCL itself feels poorly documented, there are plenty of examples to learn from
  • Once your script is compiled for OSLO, it can be totally part of the system, your own menuitems, options, defaults, everything
  • Scripting is not just bolted on later, but the whole software feels like a result of “eating your own dogfood“, resulting in a much better experience.

Optics design is tedious, so once I figured out how to do the scripting, everything just become nicer. Haven’t had time to do a lot of things, but the script I have so far is open source, and hope to add other things later.

From the programming point of view, there’s one more strangeness. Starts with that the output of numerical calculations looks like this:

Window showing numerical results of a lens' calculation
Looking at some numerical results

Those are all results from different functions, calculating spot sizes and wavefront distortions and what not, and those can be run from your own script – but the return variables are just printed. And they are a table. Those are table values that have to be accessed, so for example if I want to have the Strehl Ratio there, I would have to assign the value of something like “c1” to my internal variable – c for 3rd column, 1 for first row. Though to be this simple my code would have to clear the existing table before running the wavefront function correctly. This results in a lot of counting on the screen to see what row/column output of a function I want to actually use.

Internally, I think most of the calculations are following a finite number of rays. Finite but very high (~1000). The big problem, that could get the unsuspecting people (like I was), that what looks like a continuous calculation, is actually digital. For example if I set an aperture within the system that cuts into the incoming rays, and keep changing the aperture size (which is just a real number), the output results don’t really change unless I exclude or include some of the beams that OSLO is using for calculation – thus the result is changing in steps, and hard to trust. I guess there should be a setting somewhere to change the number of rays used, but still trying to find it.

It is one clever program though. The input and output parameters of a lightbeam are really connected by the system, since practically speaking, the optics are just transforming some parameters of the beam into other values, thus if I set the input parameters, the output is given. With OSLO they do this in a way, that the code just follows what parameters you have changed last time (input beam size? output image angles?…) and that will be a fixed variable, while everything else is adjusted. Some parameters disappear when others take certain values (set your object very far away? then we don’t have to let you set your object size, since it’s not important, instead give you a viewfield angle). It’s funky when I figured it out, but took a while…

I don’t know how much longer I’ll have to use it at work, but this looks something that really tickled my interest learning optical design much more, and once I have the hang of the quirks, it’s pretty neat. If anyone’s interested, there’s a pretty useful mailing list.

Now if only there was an open source solution… : )  (please drop me a line if there’s a good one)

Categories
Admin

Fighting forum spam

As one of the managers of Ignite Taipei, I’m trying to come up with new ways to let the community communicate, new ways to share information, advice and all. A while ago I have set up a forum at http://bbs.ignitetaipei.tw/ and I thought that will be an interesting experiment. Well, so far it is useless for communication, but turned out to be a very interesting experience from the sysadmin point of view.

I used FluxBB, because it looked simple enough, seemed to be quite fast (for low traffic volume at least), and well configurable. Except that within a very short time I run into a spam problem, so many fake users registered, and lots of algorithmically generated garbage text with a bit of advertisement here and there.

First I looked into FluxBB’s own solutions, and looks like it might not have been a great choice, because many of the spam-fighting plugins are out of date, or not supported anymore, or just a real pain to set up. The immediate practical step I could take was updating my security questions, roll my own version of “written with words, how much is 5 + 4?”, the regular low-tech captcha on FluxBB. Looks like the original answers are already in the database everywhere, so had to write my own set, which seemed to work for a while, cutting down on red-flagged registrations. But it’s not ideal, since I want to make this a dual-language forum (Ignite Taipei has both English & Chinese as official language).

Instead I turned on email confirmation. When someone registers, the password is sent to their email and have to use that to sign in. It was okay for a tiny bit, then crazy registration boom happened. I think I might be the only one real member of the board (I said that it is a failure so far for communication:) and there are 500 other spam members. Looking at their email addresses, it seems all of them have Hotmail. That kinda suggests a giant failure at Hotmail to restrict automatic registration, which is probably a problem overall. I cannot just throw out Hotmail addresses either, because it’s a popular mail provider here in Taiwan too (my first email was Hotmail too, but that was a looooong time ago, before it was Microsoft property).

So captcha don’t work, email don’t work. What to do instead? At the time I was playing around with Cloudflare, to act as an easy to use CDN. I tried it before for our Ignite Taipei blog, which is hosted on Tumblr, and that doesn’t play well with Cloudflare unfortunately. Couldn’t use it for this blog before because of my DNS provider, but now I switched, so started playing with it again.

The dashboard of the Cloudflare interface
Cloudflare stats snapshot (parts of it)

Instead of enabling Cloudlfare for the entire ignitetaipei.tw domain, just turned it on for the forum, since it’s hosted elsewhere. And that totally did it. Spam stopped that very moment, and haven’t returned since. I think what happens is that Cloudflare knows globally a lot of web/forum/email span hosts, and can challenge them or generally ignore them. Can even see where those spammers are coming from.

List of captured threats on the Cloudflare threats console
Cloudflare Threats Console

One weird (but actually not that surprising) thing is that the most active web crawler on the site (Cloudflare gives that info as well) was Baidu by far, so I guess more people knew about the site in China than elsewhere. Why’s that? Some forums that share vulnerable sites, or something like that? I barely had any Chinese content at that time, so it cannot be that. And since I turned on the threat control part, Baidu seem to have dropped quite a bit (submitted the site to Google so now that’s the busiest crawler).

All in all, Cloudflare is an interesting experiment. I can really mess up my DNS with it, and could blocked my own site for several hours, but in general it worth it. Just have to be careful. For example when testing, use their own name servers to check the information, and maybe instead if “automatic” time-to-live, set some very short time first. I usually use Google’s 8.8.8.8, and they pick up the first wrong setting really quickly, then it takes hours to pick up the correction I made just minutes after the first one.

After a bit of playing around, at least I have no spam anymore (keep fingers crossed). Now just have to get people to use the forums. :)