Computers Lab

Dual Satellite NTP server with Navspark

A friend from NIST recently told me about a Raspberry Pi Stratum-1 NTP server project, and that reminded me of the experiments I did with the Navspark dual GPS+Beidou receiver module. Navspark is a small, Arduino-compatible module that besides GPS can also receive data from China’s Beidou 北斗 satellite navigation system , that is currently being built. I thought it would be fun to build a Beidou-powered Stratum-1 NTP server to see how does it compare to GPS.


To have a really good really good, satellite-powered reference clock, I have to have access to a 1-pulse-per-second (1PPS) signal from the receiver. The pure USB-connected receivers don’t really seem to do that yet (looks like plenty of opportunities there!), instead I have to use separate hardware for it.

The Navspark module has a 1PPS pin (GPO3 below), and the only other pin I’ll really need is a serial pin to receive the NMEA stream of the satellite lock data (TXD1 below).

Detailed Navspark pinout with pin functions
Navspark pinout from the User Manual
Computers Lab Programming

OSLO is a strange one

Experimental physicists have to be the jack-of-all-trades, even on a good day, but building a new laboratory, on a budget (for any definition of “budget”) and with a limited team, is an exercise that would test anyone.

For my field, atomic physics, one needs a lot of different expertise: vacuum systems, computers, electronics, machining, optics are among the ones first to my mind. Today I’m looking at one of the tools that we are using for the last topics: optics.

In some ways, optics is pretty straightforward (both geometrical and diffraction optics), many of the calculations one could write in Python/Numpy just as well, as any software can do it. They are just pretty tedious, and there are a lot of them, optics is not a new profession. Some optics design software is very useful then, and there are not that many of them, lots of work goes into the ones that exist, thus they can be extremely expensive. One of the industry standard seem to be Zemax, and many lens manufacturers provide the specs of their lenses in a Zemax format – though it is fortunately just a relatively simple text file. On the other hand, its cheapest edition is $2500, and goes up to $9500…

Another serious competitor is OSLO (Optics Software for Layout and Optimization) which is still very well known, and fortunately for us, has a dumbed down (limited number of surfaces, not all the functions available), EDU version, for free. It even runs on Linux with Wine.

I instinctively resist any software in science & research that is not free & open source, because I feel that it closes some people out, though had to use something, and in the meantime I have grown to OSLO quite a bit.

OSLO ray graph of an imaging system
One of our imaging system design

Using OSLO, I have found that one big advantage of these software is their catalog: the glasses (oh-very-important for optics) and stock optics of different companies. This alone makes everything much easier.

What does the lens designer software do? You enter the parameters of some lenses by defining different surfaces and materials & distances between those surfaces, give it some lightbeams, and see what comes out of it. Most of this happens in this window:

The window to enter the system parameters
Lens spreadsheet

After these settings, one can run a number of different analytics code, checking the performance and behaviour of the system by calculating a bunch of physics parameters, or can modify the system to optimize that performance one way or another.

This is where the number 1 strangeness – and big bonus – comes in: OSLO can be scripted in a language called Compiled Command Language (CCL), which is a subset of C; and not just scripted, but the whole program is written on top of the CCL compiler (as I understand)! The result of this is that:

  • Your scripting can be just as powerful as the entire program, since both sides have the same functions accessible
  • There are a bunch of scripts included in the main version and even if CCL itself feels poorly documented, there are plenty of examples to learn from
  • Once your script is compiled for OSLO, it can be totally part of the system, your own menuitems, options, defaults, everything
  • Scripting is not just bolted on later, but the whole software feels like a result of “eating your own dogfood“, resulting in a much better experience.

Optics design is tedious, so once I figured out how to do the scripting, everything just become nicer. Haven’t had time to do a lot of things, but the script I have so far is open source, and hope to add other things later.

From the programming point of view, there’s one more strangeness. Starts with that the output of numerical calculations looks like this:

Window showing numerical results of a lens' calculation
Looking at some numerical results

Those are all results from different functions, calculating spot sizes and wavefront distortions and what not, and those can be run from your own script – but the return variables are just printed. And they are a table. Those are table values that have to be accessed, so for example if I want to have the Strehl Ratio there, I would have to assign the value of something like “c1” to my internal variable – c for 3rd column, 1 for first row. Though to be this simple my code would have to clear the existing table before running the wavefront function correctly. This results in a lot of counting on the screen to see what row/column output of a function I want to actually use.

Internally, I think most of the calculations are following a finite number of rays. Finite but very high (~1000). The big problem, that could get the unsuspecting people (like I was), that what looks like a continuous calculation, is actually digital. For example if I set an aperture within the system that cuts into the incoming rays, and keep changing the aperture size (which is just a real number), the output results don’t really change unless I exclude or include some of the beams that OSLO is using for calculation – thus the result is changing in steps, and hard to trust. I guess there should be a setting somewhere to change the number of rays used, but still trying to find it.

It is one clever program though. The input and output parameters of a lightbeam are really connected by the system, since practically speaking, the optics are just transforming some parameters of the beam into other values, thus if I set the input parameters, the output is given. With OSLO they do this in a way, that the code just follows what parameters you have changed last time (input beam size? output image angles?…) and that will be a fixed variable, while everything else is adjusted. Some parameters disappear when others take certain values (set your object very far away? then we don’t have to let you set your object size, since it’s not important, instead give you a viewfield angle). It’s funky when I figured it out, but took a while…

I don’t know how much longer I’ll have to use it at work, but this looks something that really tickled my interest learning optical design much more, and once I have the hang of the quirks, it’s pretty neat. If anyone’s interested, there’s a pretty useful mailing list.

Now if only there was an open source solution… : )  (please drop me a line if there’s a good one)

Admin Lab Programming

Laboratory 2.0 – a monitoring system

Looks like that one of my specialty as a physicist, and contribution to the labs where I have worked so far, is bringing different kinds of programming techniques, and technologies to the table. I’m not saying I’m any better than many of the professors, post-docs, and students I’ve met so far (there are plenty of ingenious ones), it’s more like I experiment with different tools, have tried more of the cutting edge or recent technologies, did some web programming and could whip up something quick – that might not work very well at first, but does broaden the horizon for the rest of the people.

Also, I’m a lazy person, so want to automate as much as possible. That was on my mind recently when we have been preparing to do a vacuum-system bake-out. It’s essentially a procedure to have a delicate experimental system, mostly made up of steel, glass, and stuff like that, closed up from the atmosphere, all the air pumped out, then heated up to high temperature (~150-300°C). One has to be careful, because things can break, there are temperature limitations for some materials, also on how quickly that temperature can change, requiring careful monitoring of the status of the system. And the whole thing takes something like two weeks or more. Perfect setting for automation.

Set up the electronics

The pressure measurements are done by some expensive other equipment so didn’t have to bother with that one yet, so set to work first on the temperature monitoring. Before it was a bunch of thermocouples and multimeters, requiring manual intervention and lots of labour. Instead, got some inspiration from Adafruit’s Thermocouple Breakout Board, using the MAX31855 chip, and also from the Thermocouple Multiplexer Shield. It can handle only one channel, but can use some other chip together with it to switch between the different thermocouples, and so we can read it out one-by-one. The Adafruit board could only handle 1 channel, and the multiplexer shield was using an older chip for the measurement that I could not buy anymore. In the end, found a good analog multiplexer that one that is sold in the computer market here in Taipei, the CD4067B, and it works pretty well.

Breadboard setup for temperature monitoring Arduino
Breadboard setup for temperature monitoring with Arduino

Of course, setting it all up was quite a bit of fun times, as there were way too many gotchas along the way.

  • MAX31855 is a surface-mount component, and haven’t worked with it before. Not too bad, and can be much neater, just takes some plactice
  • MAX31855 is a 3.3V circuit, so the CMOS voltage levels used by my Arduino Mega ADK had to be level shifted
  • Unlike the older chip, MAX31855 really needs differential input, and it’s much more sensitive to the environment. This required different kind of analog multiplexer than that board had
  • The Arduino Mega is a new model for me, and had some strange behaviour in terms of the serial communication
  • Surprisingly there are not too many options for 3.3V voltage regulators over here, just the LM1117, which is different from what others are using elsewhere
  • Lots of noise and stability issues until figured out what should be how. For example under no circumstance should touch the thermocouple to conducting surfaces, and avoid ground loops
  • While MAX31855 says it’s “cold-point compensated”, meaning that it accounts for the chip-s local temperature when measuring the thermocouple, it doesn’t appear completely compensated, meaning that we can have unexpected measurement change because the chip is heating up for example by being in a closed box.
  • Figuring out the right amount of time to wait between switching channels (375ms seems to be good enough, 500ms is totally fine)
In the end, though, we did have a nice 16 channel thermocouple multiplexer, sending off the measurements onto an LCD screen and to the computer over an USB cable.
Temperature monitoring board soldered
Temperature monitoring board in it’s lab setting with 16 thermocouple channels

This is then saved in a database, and can be accessed from elsewhere.


The thing that my co-workers were most amazed by wasn’t the electronics. Sure, they haven’t worked with Arduinos, but did do similar stuff. Instead they liked the monitoring interface much more, this is the one on the picture right here (can click to enlarge)

Bakeout Monitor  interface showing the vacuum system, temperatures, pressures and long term graphs
Bakeout Monitor interface (click image for full view)

It’s the schematic layout of our equipment, with the temperatures positioned where the actual sensors are. Also, the change of the measured values in time are also displayed with live scrolling.

I’m not saying it’s great. Thinking about it, the major insight that made it good for the rest of the people is that I realized how much more people understand visual data: the placement of the values to the corresponding locations on the schematics. That’s the only thing.

So inside it’s a MongoDB database (learned from previous mistakes, using a replica-set at least), with Python scripts talking to the sensors and saving the data, NodeJS / Smoothie Charts for visualization (and plain old CSS positioning of <input> tags for the reading display), nginx‘s upstream module for running two monitoring servers just in case. It’s mostly in the Github repo of the monitoring code, as well as the Arduino sketch for talking to the electronics.

It was actually quite fun to write it all, and the gradual improvements, trying the new tech, trying not to lose to much data, amazed how well it works. Especially had a good time learning about the database, scaling, fault tolerance, performance…

Of course there could be room for a lot more improvements.

  • My failover-restart bash scripts are awful, though they do seem to work more or less and counteract the USB unreliablilities
  • There were some changes to Smoothie Charts that I could improve on: logarithmic plotting, some display enhancements, wonder if it can be more optimized for performance
  • More efficient data loading. 12h data is about 30Mb in JSON format, that I send compressed, apparently it gets down to ~5% in size, but it still takes quite a bit of time to process on the frontend
  • The layout now can be changed from config files if the sensors change, so co-workers can do that without programming knowledge. I wonder if that can be simplified even more

Of course, I’m a person who generally overengineers stuff, so maybe it’s good to stop somewhere. And the somewhere might be when I got to the point to use my Kindle for monitoring (craps out on 1h data already, but some real time things are good enough).

Bakeout Monitor interface running on Kindle
Bakeout Monitor on running on Kindle 3, not perfect but does work

Get on with it

I did learn a lot along the way, and I’m sure that with this experience I will be let to do a little bit more in the lab in terms of programming ideas. I don’t like that the rest of the system is currently forced to be LabView, but that’s for another post, and there are so many things that can be improved in general as well. Let’s just go and do that.

Physics Thinking

The Two Things about Physics

Yesterday I’ve read an article that  about every topic, one has to know only Two Things.

For every subject, there are really only two things you really need to know. Everything else is the application of those two things, or just not important.

Of course it is a fascinating idea, and I started to think about my profession, physics. If I simplify my experience and knowledge down to this minimal level, what would be the two things I get to? I do think it is not a straightforward stuff, and one can only get to the bottom of this, can find the hidden truth below, if one spends a lot of time with the subject, gets to know it inside and out. I feel that I’m still just scratching the surface of the wast knowledge of the universe (even after being a physicist for about 12 years now). Still this doesn’t stop me from trying.

Drawing in my Wreck This Journal
At Geek Dinner Taipei someone contributed this drawing of the Einstein Field Equations to my Wreck This Journal. Of course, this is ignoring a possible non-zero cosmological constant.

I’m an experimental physicist, worked in Solid State Physics first, now mostly in Atomic & Laser Physics – all kind of fun stuff. I had very good professors, great inspiration and I have learnt a lot from them. If I think back all the things I’ve learned there are still things that come to me as my first thoughts, and usually those are the right guesses, for whatever intuition I have.

My Two Things about Physics:

  1. Pure math can take you very far along the way, though in the end still need experiment to see whether the results describe something real.
  2. Everything is an approximation, but that’s fine. Just pick your approximations carefully.

Maybe it is worth expanding a little bit on that these:

Pure math can be used to describe things extremely well. It’s maybe even too good at that, which got other, much cleverer people to think as well, like a fellow Hungarian physicist, Wigner Jenő (or Eugen Wigner), writing about The Unreasonable Effectiveness of Mathematics in the Natural Sciences. I frequently got myself really excited when after doing some complicated calculation to predict the behaviour of a physical system, the experiment matches up to all of its nuances. Let it be atomic spectroscopy, polarization of light reflected from metallic mirrors, magnetic field of a Zeeman-slower, I had a lot of fun figuring out the physical theories of different phenomena then matching it up with what happens in the lab. Of course, there were loads of times when they didn’t match, but it turns out my math was off. Really, if I want to understand the world, math is one of the most useful and versatile tool to have.

Having that tool is of course not enough. When people come up with new and interesting math trick, it often turns out (to many people’s amusement) that those tricks have some physical meaning, they often also give some insight into the world around us. It’s often, but not always. Mathematics can really easy take one to very strange places and give a result, which is completely aphysical. To complicate things even further, those aphysical results sometimes turn out to be actually correct and predicting real but insofar unobserved things. An example of this is the Dirac equation which gives two solution, one for electron, and one particle with negative mass that first people dismissed, but later it was understood as the representation of positron. How to distinguish between really wrong solution and “wrong as current understanding”, that’s a whole different level of problems.

The second point was really a revelation for me. Whatever equations we have, they all just describe things to a certain level. If we can look closer, we often find differences from the theory, that are harder and harder to explain as we get closer. On the other hand, using intuition and physical understanding, people often choose to ignore certain parts of the situation, or certain features of the problem since it cannot affect the results to a level that would be observed in the given experiment. This makes everything solvable, and once solved, one can advance on top of the new understanding even deeper into the problem. Finding the good approximations is almost as valuable as finding the right theory, that’s why often these approximations are named after the people who came up with them, or given other shorthand names so everyone can quickly recall and use them.

There’s a whole methodology built to help come up with approximations and handle them, called perturbation theory: if the given problem is very similar to a simpler, already solved problem, then treat it as the simple one plus some small effect that changes relatively little on the behaviour of the system. Not everything can be handled like this, but surprisingly many problems fit very well.

Others’ Two Things about Physics

On the original site there were other people’s Two Things as well:

1. Energy is conserved.
2. Photons (and everything else) behave like both waves and particles.
-Tim Lee

1.  Draw a diagram.
2.  Get the dimensions straight.
-Eric Schafer

I personally don’t like these that much. The first one is just stating two theories that can be superseded in the future, and right now they kind of limit instead of enable. The second one is good advice, but can’t say that’s the only thing there about physics. Having said that, I have more adventures with incorrect dimensions and units than I’d prefer to have.

What other Two Things choices one can make, in physics or in other sciences, other topics?


Rules of the garage, I mean, lab

I was catching up on StartupBus, an awesome (crazy? probably both) project that I really wish I could take part in. One part was watching the videos they shared in Youtube, especially one series by Phil McKinney (Chief Technology Officer at HP), where among other things he talks about the Rules of the Garage (the garage where HP was started). It is very inspiring I think those rules should apply not just in business but many other creative team effort. Such creative team effort is, for example, a physics lab, just like the one I’m working now. And indeed this group would need some philosophy infusion like this.

  1. Believe you can change the world.
  2. Work quickly, keep the tools unlocked, work whenever.
  3. Know when to work alone and when to work together.
  4. Share tools, ideas. Trust your colleagues.
  5. No Politics. No bureaucracy. (These are ridiculous in a [lab]).
  6. The customer defines a job well done. [?]
  7. Radical ideas are not bad ideas.
  8. Invent different ways of working.
  9. Make a contribution every day.
  10. If it doesn’t contribute, it doesn’t leave the [lab].
  11. Believe that together we can do anything.
  12. Invent.

These are so well formulated that they give more of an “aha!” feeling instead of a question that “yeah? why’s that?”. Researchers should have all of this drilled in. Or rather everyone should have.

The only one that I’m not satisfied by is #6: what constitutes a “job well done” in academia? No customers so no direct measurable response to the effort. The commonly used metrics are amount of grants won, number of members who won tenures, number of publications and/or citation, conference invitations, and so on, none of which really captures what science is about and why people should be doing it. But then what is the metric? Maybe this is a problem in general, no clear aim makes unfocused effort. Or is it a strength as people can define their own measures of success? I don’t think that’s the right way either. I usually go by the “amount of curiosity satisfied and new things learned”. But since I feel I have learned an awful lot lately but still barely going forward, maybe I’m not the example of someone who should be followed on this.

Anyway, I wanted to post this, because it needs a bit more thinking and have to start somewhere. And indeed there’s a lot to do…

Optical table
Just a random landscape on the optical table

Now back to work. But better this time, there’s a world to change.