I like advent calendars a lot. They can bring a lot of surprise, preparation, focus, and joy. They can come in many shapes and forms, and they encourage DIY – make your own calendar, count the things that are important.
This year, I got to play with a very interesting “advent calendar”, called 24PullRequests. It is the kind of thing that I don’t understand why people haven’t done it before. The mission: help out open source projects by submitting enhancements and fixes (i.e. “pull requests“), and do that for 24 days counting down to Christmas.
my 24 Pull Requests calendar
I had to take part in that one, and while the result wasn’t as successful as I wanted, it was so far my best contribution to open source.
My pull requests
Instead of 24, I managed to make 4 enhancements that were ready to be sent off. That’s not consoling me that it seems nobody managed 24, but never mind. Here are the things I made:
SmoothieCharts: make the charts use newer browser animation technologies that have better performance, and save on battery as well. This one was prepared somewhat earlier than December, but the final version was pushed within the right time frame. Being tested, not merged yet.
OpenHack: I’m organizing the event in Taipei, and noticed that some other place has broken image link. Hunted down the same pic from Google Cache, and set it up again.
Python Guide: added some info about installing certain Python packages in Arch Linux and Ubuntu. This is embarrassingly tiny fix, there’s so much more to do here
AngularJS: this is fixing that one couldn’t run the build script if the system Java can’t run in 32bit mode. I didn’t know that this was a Google project, until they sent me a request to sign some contributor agreement. I feel strangely humbled.
Lessons learned
Four contributions were already a lot of experience, because all of them were so different. Here are some lessons learned:
Write good pull requests – that starts with writing good commit message! People keep saying that, but seriously, no excuse not to do that.
When the changes have been sent in, don’t mind that they are not accepted yet. Every project have their own pace. Keep working on whatever you like
I was looking for ow hanging fruit, but one has to go in there still to make some meaningful contribution.
The issue tracker is a good start to see what to fix, but not always helpful, as it can be difficult to understand what propblem the others try to describe, if you are new to the project. On the other hand, try to use the code, I’m sure you’ll find some pain points right away (that was AngularJS). Also, the busiest issue trackers are not the best, they are full of things that would side-track you for a long time. Projects with a medium count are good for such an improve-and-run contribution.
Don’t be afraid to do things, but still do them the best you can. Your contribution doesn’t always feel meaningful, but still a little improvement is more than most people do. (just like PythonGuide was)
Keep things simple – easier to do, easier to pull. Even if sometimes that takes longer to write (the AngularJS contribution srunk to the quarter of its size while I was trying to figure out the simplest way to achieve what I wanted)
If interested, don’t worry if the project uses programming language you don’t know. You can pick up new things easier than it seems. Also, many projects give you feedback on your contribution, to help you improve it.
This project don’t encourage to work on your own stuff, but that doesn’t matter, there are another 11 months for that, or every day after these contributions are done
How to do this for the whole year? Bug squashing day in general? Still need to get deeper in projects, but go and explore. Can also see CodeTriage and ContribHub, linked from 24PullRequests
If stuck in the fixing, but the problem is interesting, don’t worry if it doesn’t fit in the 24 days. Keep working on it, the recipients will be happy any time (I have have 1 or 2 such patches)
Google’s suggestions for what meetings are according to the Internet
The results were really for my liking, and I was thinking whether I can use this to get some insights into fields of knowledge, for example what does the Internet think about different programming languages?
Started with LabView, since that’s the one I’m struggling at work at the moment (that’s for some other stories). I don”t want to overly generalize, and I have a lot to learn about it, but still, I am not that surprised by the Number 1 spell correction:
Labview is….. crap, according to the Internet
So let’s see how the other languages and software engineering keywords come are treated?
Positive sentiments
There were some keywords with positive sentiments, never really among the top suggestions and never too many but there were
There are some languages, that come up only with language related corrections, I guess that’s mostly because people don’t have strong negative or positive feelings, or maybe it’s more confusing then infuriating? Or people just don’t know much about some of these?
I think my favorites are “worse than crap” and “an exceptionally bad idea”, with so explicit phrases, people must feel very strongly about it. Also, I’m surprised that with this many “dead” languages there’s still any programming going on! Of course, this is not an exhaustive list, would love to see more examples, and add to the list
Anyways, let’s head back to coding now that I have cheered myself up. Programming is hard, and the Internet knows that too.
Looks like that one of my specialty as a physicist, and contribution to the labs where I have worked so far, is bringing different kinds of programming techniques, and technologies to the table. I’m not saying I’m any better than many of the professors, post-docs, and students I’ve met so far (there are plenty of ingenious ones), it’s more like I experiment with different tools, have tried more of the cutting edge or recent technologies, did some web programming and could whip up something quick – that might not work very well at first, but does broaden the horizon for the rest of the people.
Also, I’m a lazy person, so want to automate as much as possible. That was on my mind recently when we have been preparing to do a vacuum-system bake-out. It’s essentially a procedure to have a delicate experimental system, mostly made up of steel, glass, and stuff like that, closed up from the atmosphere, all the air pumped out, then heated up to high temperature (~150-300°C). One has to be careful, because things can break, there are temperature limitations for some materials, also on how quickly that temperature can change, requiring careful monitoring of the status of the system. And the whole thing takes something like two weeks or more. Perfect setting for automation.
Set up the electronics
The pressure measurements are done by some expensive other equipment so didn’t have to bother with that one yet, so set to work first on the temperature monitoring. Before it was a bunch of thermocouples and multimeters, requiring manual intervention and lots of labour. Instead, got some inspiration from Adafruit’s Thermocouple Breakout Board, using the MAX31855 chip, and also from the Thermocouple Multiplexer Shield. It can handle only one channel, but can use some other chip together with it to switch between the different thermocouples, and so we can read it out one-by-one. The Adafruit board could only handle 1 channel, and the multiplexer shield was using an older chip for the measurement that I could not buy anymore. In the end, found a good analog multiplexer that one that is sold in the computer market here in Taipei, the CD4067B, and it works pretty well.
Breadboard setup for temperature monitoring with Arduino
Of course, setting it all up was quite a bit of fun times, as there were way too many gotchas along the way.
MAX31855 is a surface-mount component, and haven’t worked with it before. Not too bad, and can be much neater, just takes some plactice
MAX31855 is a 3.3V circuit, so the CMOS voltage levels used by my Arduino Mega ADK had to be level shifted
Unlike the older chip, MAX31855 really needs differential input, and it’s much more sensitive to the environment. This required different kind of analog multiplexer than that board had
The Arduino Mega is a new model for me, and had some strange behaviour in terms of the serial communication
Surprisingly there are not too many options for 3.3V voltage regulators over here, just the LM1117, which is different from what others are using elsewhere
Lots of noise and stability issues until figured out what should be how. For example under no circumstance should touch the thermocouple to conducting surfaces, and avoid ground loops
While MAX31855 says it’s “cold-point compensated”, meaning that it accounts for the chip-s local temperature when measuring the thermocouple, it doesn’t appear completely compensated, meaning that we can have unexpected measurement change because the chip is heating up for example by being in a closed box.
Figuring out the right amount of time to wait between switching channels (375ms seems to be good enough, 500ms is totally fine)
In the end, though, we did have a nice 16 channel thermocouple multiplexer, sending off the measurements onto an LCD screen and to the computer over an USB cable.
Temperature monitoring board in it’s lab setting with 16 thermocouple channels
This is then saved in a database, and can be accessed from elsewhere.
Visualize!
The thing that my co-workers were most amazed by wasn’t the electronics. Sure, they haven’t worked with Arduinos, but did do similar stuff. Instead they liked the monitoring interface much more, this is the one on the picture right here (can click to enlarge)
Bakeout Monitor interface (click image for full view)
It’s the schematic layout of our equipment, with the temperatures positioned where the actual sensors are. Also, the change of the measured values in time are also displayed with live scrolling.
I’m not saying it’s great. Thinking about it, the major insight that made it good for the rest of the people is that I realized how much more people understand visual data: the placement of the values to the corresponding locations on the schematics. That’s the only thing.
It was actually quite fun to write it all, and the gradual improvements, trying the new tech, trying not to lose to much data, amazed how well it works. Especially had a good time learning about the database, scaling, fault tolerance, performance…
Of course there could be room for a lot more improvements.
My failover-restart bash scripts are awful, though they do seem to work more or less and counteract the USB unreliablilities
There were some changes to Smoothie Charts that I could improve on: logarithmic plotting, some display enhancements, wonder if it can be more optimized for performance
More efficient data loading. 12h data is about 30Mb in JSON format, that I send compressed, apparently it gets down to ~5% in size, but it still takes quite a bit of time to process on the frontend
The layout now can be changed from config files if the sensors change, so co-workers can do that without programming knowledge. I wonder if that can be simplified even more
Of course, I’m a person who generally overengineers stuff, so maybe it’s good to stop somewhere. And the somewhere might be when I got to the point to use my Kindle for monitoring (craps out on 1h data already, but some real time things are good enough).
Bakeout Monitor on running on Kindle 3, not perfect but does work
Get on with it
I did learn a lot along the way, and I’m sure that with this experience I will be let to do a little bit more in the lab in terms of programming ideas. I don’t like that the rest of the system is currently forced to be LabView, but that’s for another post, and there are so many things that can be improved in general as well. Let’s just go and do that.
I was out with a few friends the other day, they were forming a team to go to StartupWeekend Taiwan Hardware next month. I have been to one or two previous StartupWeeekends and they are good fun. Haven’t made up my mind about the hardware one, but since they asked me could they prepare for it, since it’s their first experience, I did try to gather some advice. Not sure how better they are off with it, but I hope at least a little. Looking at the previous events, a little bit of experience and knowledge can put people way ahead, because Taiwan is just learning startups, every experience is golden.
Later, though I started to think what I have told them. One particular advice I had to examine: don’t start with a tech that you find interesting, start with a story instead and choose your tech for that.
Now I’m not that sure that this is a good advice for every occasion, especially because I was brainstorming about one particular tech that really excites me, and if I was right earlier, then I’m wrong now. And the more I thought about it, the more I felt, that there can be a case (especially for StartupWeekends) for starting with a tech, even if it is probably the harder way to get something awesome out of the process. Still, given that limitations make one more creative, it worth doing at least some proper brainstorming about it.
Lilypad Arduino
The tech I was thinking about that time was the Lilypad Arduino. I haven’t got one (it’s sitting in my Sparkfun basket, ready to be ordered), but really want to make a project using it.
Lilypad Arduino Ebroidery by Bekathwia (click picture for Flickr link)
So now the question is, what kind of projects I could come up with that could use this. In the last few days of brainstorming (and now writing this), I came up with a couple of opportunities, not sure if any of them has been done. Not sure about the quality either.
Project ideas
Smart bag: figure out how to make some simple/small way of communicating with a little sensor that can be attached to items that one does want to always bring along: keys, wallet, phone…. The bag would sense it if they are out of its range and warn the user. Never leave stuff behind at home or at a cafe.
Visual turn-by-turn navigation for cycles: use a smartphone to get navigation directions to where you want to go with your bicycle or motorcycle. A jacket is outfitted with lights on both arm, and would have a Lilypad to communicate with the smartphone, signaling which way to turn and when.
Movement direction clothes: smart clothes that would detect the position and posture of the person wearing them, and use light or vibration to signal what movement is the next one. Could maybe correct choreography or teach karate katas.
Communicating clothes: this could be done in multiple ways, wireless, infrared (like the TV remote control, just two way), consciously controlled or in the background. Ultimate spy clothes, send messages between people in the crowd without them actually doing anything. Could local business send out signals that are received the clothes could prompt you with a deal or order ahead your favorite if you are a regular (though, this can really easily be very spammy). Also, the different units can synchronize with each other – cue super visual flashmob.
Smart bedding: what if your bed could monitor your sleep and wake you up when the morning comes? Bit like Wakemate, but for the bed.
People tracking: build into work clothes for a factory and can log in and out people just by sensing them. Make it part of the ticket in an amusement park, and can see which rides are too popular, can communicate back to advise about waiting time and suggest other ways to pass the time. Finding people in a crowded place in emergency, see if anyone’s left behind. Though these can be very 1984 if done badly….
Health sensing: monitor vital body functions for people who are somehow at risk: older people, partying in town (drink responsively), doing sports…. Warn them when some critical situation is predicted.
Style guide: clothes having smart tags with “style” and “colour” and “pattern”, and either all of them collectively, or a central piece of clothing would check whether these thing you wear do match with each other. Could also make recommendations what to wear. “I have this trousers, shoes, and so on – which shirt should I wear to that?” – press a button, the right choice lights up or vibrates in the closet.
Well, that’s for now. Most of these, I realize, are quite a bit tentative, many of them seem to miss some key ingredient, or have a (technical) problem to solve before it would work. Which one would I bring to a StartupWeekend if I’d go, which one is ready to be made and could be made in 54 hours? Tough one…
Found anything interesting here, or have some more ideas? I’d love to hear them, let me know in the comments!
It is very much summer now here in Taiwan, and if one thing the Tropics is good at, then it’s being hot and humid. That’s sometimes fun, when there’s enough chance to hang out by a pool or on the seaside, but most of the time not that much. That’s why there are so many air conditioners, and other clima controls.
As I really like data, and this place is great for electronics, I thought I can just combine the two things, and try to learn more about my environment. Temperature is relatively easy, but for a long time I was looking for a humidity sensor. At the local electronics market I found a few different models the other day, and got one of them – pretty much the cheapest one, because all the manuals were in Chinese anyway (which I don’t yet read), and they were also all packed up, so there was not much peeking anyway. There was a combined temperature/humidity sensor board (two sensors already packed up together), but at 450NT it felt pricey and needed to buy some connectors for it as well. There was a larger round capacitive sensor, but it was too large for my liking.
I ended up with a “CM-R Resistive Humidity Sensor” (here’s an English spec scheet for it). It was 250NT (~5 quid or 8 bucks) which makes it relatively expensive equipment for me, though maybe because I was saving my cash in my wallet for going to Mobile Monday Taipei later that night. Still, if it works then it does worth it, and might get a few more pieces.
Wiring it up
I never actually knew how the electronic humidity sensors work, so this is a good primer for that. Nevertheless, the plan was afoot to hook it up to an Arduino and measure some RH% (relative humidity). It was quite tricky for me to figure out the wiring, though, because I never worked with resistive element that could handle only AC voltage, and gets destroyed by DC.
That white rectangle with black spots, the humidity sensor at work, all plugged in with Arduino
Fortunately there were a couple of people (indeed just 2 or 3 that I could find) who were working with this, and had some good information. The one I went with was this post from the Arduino forums. Was quite informative, though I think the final formula is faulty – not because I understand it completely (my electronics theory is spotty at best) but because experimentally I figured out what works instead (being an experimental physicist, that’s pretty much what I do all day anyway).
Finally it is hooked up something like what is shown in this hideous schematics (really got to find some better way to draw this):
Humidity sensor schematics, done with Fritzing
After this wiring, I have to switch the digital output (DO) pin between High and Low at about 1kHz with 50% duty cycle. When the pin is on High value, every now and then I measure the voltage with the analog input pin. From the measured voltage I can deduce the resistance of the humidity sensor as
RH = R1 / ((Vref - Vm) / (Vin - Vm) - 1)
where Vref is the reference voltage (5V of the digital pin), Vin is the measured voltage, Vm is in this case Vref/2=2.5V, the voltage that the capacitors are charged to during all this switching, and R1 is the reference resistance. It works very well, I tried to measure some fixed resistances, and the results are pretty accurate.
When we have the resistance value, the spec sheet has a look-up table for translating the resistance into relative humidity (RH%). It’s quite cumbersome, because will have to interpolate between values, and it’s also not that accurate if the temperature is not known. On the other hand, the values from 25℃ can be used pretty well for all normal temperature ranges here in the summer (let’s say air conditioned 21℃ to outside summery 33℃) with maybe ±5% error in the humidity measurement, that’s not that bad.
The sensor itself feels really sensitive, just having it in the lab, could sense changes in RH as others were doing water-related things the next room. Even if reading is not that accurate, it is still quite precise, and for most of the things it is still okay. I could try in different environments, like our “dry box”, which maintains ~20% humidity, but that means quite high resistance of the sensor, MOhm instead of tens of kOhm, so the voltage reading just moving around near 5V, making it not that precise and noisy as well. For 30-90% humidity it works pretty well, though.
To monitor the readings, I wrote up a quick web-app from some reused code with NodeJS, Express, Socket.io, node-serialport2 and Flot. There’s no kill like overkill, but at least I can already monitor it from any other computer as well, it’s real-time, and the GUI is much easier to adapt than most other ways I know.
All the code for the Arduino and the monitoring server is in my Weatherstation repo on Github. It’s pretty buggy and not documented, will clean that up later when I’ll have a good working version.
Deployment
Now let’s get down to some real action. Being at home, I often feel that my flat is way too humid. Firing this circuit up, I got a reading of 80-85% relative humidity, that’s not something to sneeze at. Then turned on the air conditioning in dehumidifier mode, and watched what happened.
First there was an increase of the relative humidity, because the drop in temperature is quicker and the drop in water content. After chilling for a while, this is how it looked like.
Humidity logging while running the AC’s built in dehumidifier (RH% vs time in minutes)
Looks like some funky periodic behaviour, but empirically I felt much better, even if the humidity didn’t drop much. Maybe the extra comfort comes from the temperature drop as well, so will try to do the same thing in the usual temperature-control-only mode as well, see what does that one look like? I haven’t had neither my laptop battery charger, nor a spare battery with me, so I couldn’t run this too long, I wonder what level it would reach over a longer time. Looks like it has a downward slope, but it’s hard to tell.
Also, the wobbles are likely because of some feedback control that is turning the dehumidifier inside the AC on and off, and that controller has a huge hysteresis. It is not totally surprising, though: as much as I know, dehumidifiers are just big cold surfaces and the water condenses on them from the passing air. For that you have to keep a large surface pretty cold, and it is likely some time to cool a surface or let it warm back up again, thus the controller likely overshoots in both the de- and the re-humidifying parts of the process.
From what I have tried, it feels like a pretty good sensitive sensor, and coupled with
Plans
I’ve seen another way to wire up the sensor in a tutorial, which is basically a resonant circuit and the chance of the sensor’s resistance will change the output frequency. That frequency can be readily measured with Arduino already, so if I get the few extra components, might give it a try. It seems that potentially it can have better noise characteristics then the method I’m using.
Want to add some real logging function to it, maybe saving readings and timestamps into an SQLite database, which I can analyze properly later with Numpy. Or maybe store it in MongoDB, on which I can still interface and do some concurrent analytics even when the sensor is running. Or maybe I consider that just because I use Mongo for Friendcare?
Would like to add a 10-LED light bar, which could display the humidity decades without any computer, thus making the whole setup so much more portable. Though for that I will definitely have to revive a 74HC595STP16C596 Serial-in/Parallel-out shift register, because then I’ll need one resistor and a single DO channel to control those LEDs, instead of 10 DO channels and 10 resistors.
Could also add an LCD screen like this SparkFun SerLCD. The only problem is that it interferes with the serial connection, so might not be able to use it together with the computer monitoring. Might worth it, though, that LCD I have (white-on-black with backlight) looks pretty darn funky.
Could also add some long-term monitoring and some other sensors, to build up a real home weather station. I have my eyes on a barometric pressure sensor, but that’s quite expensive.
Update (2012/11/6):
Been using this in our laboratory for almost two months now, it works quite well, though still thinking of there’s any better way to measure the resistance of the sensor. Brushing up on RC circuits is in order?
Looks like I’m not the only one thinking about humidity sensors and data logging: there’s also another post from the awesome Tom Igoe.
