Categories
Computers

Tech setup for an Ignite

Recently I was co-organizing our Ignite Taipei #2 event (see the pictures and watch the talks – this latter if you speak Chinese…) I try to be the self-proclaimed Chief Technology Officer (CTO) of the event. Either because I hope the tech side of things goes down well, or more likely if things fail then there’s no-one else to blame just myself. And things do indeed fail all the time all different ways.

So as a CTO I try to make sure that everything computer related runs well and I did collect some useful scripts (for that mostly command-line driven way I do things). I thought it would be useful to write them all up, mostly for me to remember, not just the scripts themselves but the rationale of some choices made along the way.

As Ignite’s motto is “Inspire us but make it quick”, I found that that the Ignite organizer’s motto can be “Everything will be just fine.”

The computers I used for getting things ready for the show
The computers I used for getting things ready for the show

Intro

Ignite in short is an evening of quick presentations, each one of them exactly 5 minutes, 20 slides that auto-advance every 15 seconds. Altogether there are 10-16 talks on evening, It aims to be an event for inspiring people, and something that should be relatively straightforward to organize.

Computer setup

I had 2 computers for the event, but those were actually 3 systems:

  • A Linux system (Arch Linux), my own computer that I knew very well and run the presentation off.
  • A Linux/Windows dual boot system that I borrowed, Windows (Vista of all things) for PPT-PDF conversion and Ubuntu for live streaming.
If I could, I’d ditch the Windows part altogether, but if can’t it would probably be better to run it in a virtual machine (so I don’t have to reboot, more on this later) or on a separate (third) computer.

Pre-event

Most of the organization was done on Google Docs with a shared document, keeping tabs of who we have as speakers, what needs to be done.

Shameless plug: next time it will be even easier with WatchDoc, a Chrome extension I wrote after Ignite to get notification when shared documents change :)

Besides keeping up with what to do, I had to take care of the presentations that the speakers sent to me. Since I’m not a fan of PowerPoint and its unreliability to display things the same way on different computers (and LibreOffice not being ready to handle .ppt/.pptx as well as I’d like it to), I had to convert everything into PDF. People with Mac, using Keynote were easy, they have PDF export. Most people using Windows/MS Office didn’t have. First I installed a “print to PDF” plugin but that was just terrible, awful quality photo conversion and all. In the end I had to get an Office 2007 just for this occasion and use their own Save to PDF add-in, for great good. Actually, it worked like a charm, I just wonder why they didn’t make the same thing for Office 2003 that seems to be much more abundant (I know, 8 years old software has no love). Finally I had everyone’s thing in the same, reliable format, and uploaded to Google Docs and Dropbox, so can transfer it between computers easily.

One more thing I had to do the the slides: add an empty slide on the end of the 20 slides of each presentation, so it’s easy to see when they are finished. Used pdfmanipulate for that. Prepare an empty slide, can use e.g. LibreOffice, export it as empty.pdf, then having all the talk PDFs in a a sub-directory called “original, run a script like this:

#!/bin/bash
DIR=original

for f in $DIR/*.pdf
do
    remf="${f%.pdf}"
    newf="${remf##*/}"
    echo "${newf}"
    pdfmanipulate merge --output="${newf}_extra.pdf" "$f" empty.pdf

Slides

The tech setup in the front; From Ignite Taipei #2

There were two kinds of slides to take care of: one with the speakers’ names and their talks titles, the other one is the talks themselves, prepares as described earlier.

The speaker intro slides were done in LibreOffice. Had a little problem with that one as well, as if Chinese characters were set to bold text, they showed up blurred. Not good looking, lot even legible sometimes. So just before start I was scrambling to change all text from bold to normal. It’s just a pain.

The talk slides were shown using Impressive, a command line presentation software written in Python. It is pretty good, quite flexible and easy to set the slide show times, transitions and total time progress bar.

Use the following script saved as e.g. present.sh to show the talk as present.sh nextalk_extra.pdf:

#!/bin/bash
# Run the presentation

PRESENTATION="$1"
SLIDETIME=15
TOTALTIME=300

./impressive.py -D 1 \
                -a ${SLIDETIME} \
                -d ${TOTALTIME} \
                -T 350 \
                -t CrossFade \
                -c persistent \
                "${PRESENTATION}"

There are some problems with it, though. Sometimes it took quite a while to show a slide that had a good quality photo, which messed up the timing. The total time didn’t quite work out the way I expected, I think every presentation was a bit longer than 5 minutes overall. Not too big problem., but if I can do right, then I could.

Previously used the Adobe Acrobat Reader’s auto-advance when full screen mode, that was just fine, might go back there next time if I cannot fix Impressive.

Web streaming + recording

A common thing to live-stream Ignite so other people can watch it to. I found it easiest to do with VLC + Justin.TV, they already have some software done that seems to work quite well. The first time I had some sound sync issues, this time managed to fix that, but in the end it didn’t matter.

To do the stream using my Logitech QuickCam S7500 and sound on the line-in from the venue’s sound system, start VLC with the following script:

#!/bin/bash
# http://community.justin.tv/forums/showthread.php?t=7081
# Using webcam + line-in audio
# Display + Transcode (Save + Stream)
# needs jtvlc to get it out

vlc v4l2:///dev/video0 \
     :input-slave=alsa://plughw:0,0 \
     --sout='#duplicate{dst=display,dst="transcode{venc=x264{keyint=60,idrint=2},vcodec=h264,vb=600,acodec=mp4a,ab=128,channels=1,deinterlaceaudio-sync}:duplicate{dst=standard{access=file,mux=mp4,dst=/home/user/ignite.mpg},dst=rtp{dst=127.0.0.1,port=1234,caching=2000,rtcp-mux,sdp=file:///tmp/vlc.sdp}}"}'

It as some mighty long line and not sure if it can really be wrapped anywhere. Basically it displays the webcam’s picture, transcodes it to x264 and sends it to an rtp socket and saves into a file in the same time. Might want to be careful with this, as the saved file will be overwritten every time the script is run.

The most curious thing is that in the copy that is displayed, the sound is not in sync, but actually the transcoded video is okay.

Next step is sending the stream out to Justin.tv, using jtvlc:

#!/bin/bash
./jtvlc-lin-0.41/jtvlc ${JUSTINTV_USERNAME} ${JUSTINTV_STREAM_KEY} "/tmp/vlc.sdp" -d

Here the username and stream key has to be filled in with your values. If everything’s fine then there’s a big stream of debug information on the screen and on the website the channel goes online.

At our event I had a problem that after reboot somehow the sound input is borked and all out video was without sound, the final saved file couldn’t even be opened. Never mind, fortunately we had recording from a proper camera.

HD recording + postprocessing

It is also essential to have some good recording of the show, since that can make it really reach a wider audience, and it’s good to look back at it later as well.

This time we had a cameraman helping out, and after the talks finished I got the videos from him. He’s using a Sony camera with a FAT32-formatted SD card, which was a nice pain to manage. Last the video we had was straightforward mp4 format, while this time 2Gb chunks of AVHCD, a proprietary format. Had to convert that into something I can manage.

It took a wile to figure out, but in the end the result was acceptable.

1) join the spilt parts with tsMuxeR. Had to make sure I’m using joins, not just listing all the files and saving them as one (sounds like the same thing but it isn’t). In the end I had some “.m2ts” files

2) Next I had to convert those into mp4. The m2ts files had actually h264 video that is good for mp4, so ended up just copying it. The audio had to be transcoded to aac, because the original was ac3 while that’s not accepted audio codec for mp4. Fortunately ffmpeg could handle AVHCD files now. The only really tricky part was that the original video was 1080i – interlaced. Next time have to make sure that the cameraman sets things to progressive, makes life so much easier.

#!/bin/bash
INFILE=$1
OUTFILE=$2
# Start recreating video
ffmpeg -deinterlace \
       -i ${INFILE} \
       -f mp4 \
       -vcodec copy \
       -strict experimental \
       -acodec aac \
       -ab 128k \
       -y ${OUTFILE}

This took quite short time, which is a relief, before I figured out that things can be this simple I had transcoded files in hundreds of giga-bytes, and things took hours.

3) Spilt the video for uploading to YouTube. Used Avidemux for that, which can – barely – handle h264 encoded video. Somehow it couldn’t find the key-frames so it decided that every 30th frame must be that and I can only split things there. This resulted in every video having some strange pictures in the beginning (duh, missing keyframe). Might be better with re-encoding, or rather finding a better program to handle h264 video.

Future

Fixing stuff at halftime; From Ignite Taipei #2

Of course there are some things to improve next time:

  • Better video recording that eases post-processing. Preferably have our own camera that we learn how to use and not have to figure out something new every time
  • One a computer is set up and tested, no more reboots
  • Check the timing of the slides to make sure they are really 15 seconds each
  • Improve Impressive, maybe prepare some patches: exit when presentation finished, show empty screen after last slide, timing issues
  • Switch to scripted intro slides so I don’t have to edit each of them to make sure they look the same. Maybe use one of the Javascript web presentation frameworks and a full-screen Chrome window…
  • Set up our own website for Ignite. This one is the big one. Should allow us to do a lot of interesting things, but I try get myself to make it first with a few features only, not with all bells and whistles. A WordPress page with some plugins or a full blown Django site or something else? Time will tell…
Categories
Programming

Language of the Month: Prolog, part 2

If something, then this is a belated post. Prolog was the language of July and now it’s September. Anyway, before I completely fail, let’s just wrap it up and go on the next language with this one month hiatus in August.

I really enjoyed the language and one month is indeed barely enough to start doing something useful. So I have to come back to it again and maybe keep reading about it in the meantime. It’s actually quite interesting to try to figure out Prolog code on paper, without actually running something. I think one of the books I was reading had plenty of exercises like this: without running the code, can you guess what is it doing? Needless to say, there were plenty of tricky bits/

From the comments

During two months, some additional resources did show up. A dear commenter on the previous post recommended me the following book: Richard O’Keefe: The Craft of Prolog. I got about a third, maybe halfway through it and it’s interesting, written long time ago in a style that since gone out of fashion, unfortunately. It’s a programming book that is fun to read and one can see that the author is very knowledgeable. Some aspects of the book didn’t age very well, though. The author keeps comparing Prolog to other languages – many of them are not very widely available either. Also, some of the language features are specific to his version of Prolog, that wasn’t the same one I was using. Would recommend, though!

This last part, the different implementations of the same language, can be a real problem. Of the three compilers available for me, all of them had specific strengths and weaknesses. Guess they are converging, but not quite yet. So far I was mostly using SWI Prolog, but this might change in the future.

The said commenter also recommended a cross-compiler, doing Prolog-to-C magic, for portability and other goodness. Can grab it from here: http://ftp.shaw.ca/irvinsh/prolog.tar.gz I haven’t had time to try it yet, but once I have, I’ll do a comparison.

From the web

From a friend’s recommendation I was checking out a site with free textbooks. They are all advertisement-sponsored, which is an intriguing idea (for another post). The IT section had not one but two books on Prolog: Prolog Techniques and Applications of Prolog.

Two prolog book covers
Free textbooks on Prolog from Bookboon

This latter one has a Hungarian author so I’m even more intrigued, we used to have great computer scientists (John von Neumann / Neumann János, anyone?), so I hope we keep up that tradition. (Oh, yeah, and had great physicists as well, maybe I can do more on that front later).

I was only skimmed them a little bit, but looks like these will be good addition for my “programming for fun and efficiency” library.

Will update the original LotM:Prolog page with these links as well. Now onto September’s Language, fortunately I have idea what I want to learn for the next ~3.5 weeks. October will be something Artificial Intelligence related since I signed up for the AI-class.

Categories
Programming

Language of the Month: Prolog

New month, new programming language to learn, the 3rd one in this series. So the repertoire so far contains:

Prolog

It is again a very different choice, logic programming. Been playing with it for the last two weeks or so, and it really makes me think differently about programming and programs. Logic and complex thinking was always a favorite past-time of me (e.g. solving puzzles and such), but only now I realized that I do have a lot to practice in this area.

Prolog is also one of the older languages (feels like a “classmate” of Lisp and Fortran) so it was the first language in the series where I could actually go to a library, take out some books to learn it, and that book wasn’t hopelessly out of date (try to do the same thing with Python or Ruby…). Since these books were also written by academics and not necessarily computer scientists, their whole approach is different, in a way more curious, though probably less practical.

In the end, I think what I would like to gain is a tool that I can use to attack problems that are intractable or at least very difficult in other languages.

Start

One of the first thing I found hard to figure out was how to actually run a program. So in a nutshell: even if most of the interaction is within an interpreter-like environment, all the basics have to be prepared in a file and loaded.

E.g. I prepare a file like this (based on Ender’s Game), call it ender.pl:

%% Genealogy of Ender's Game.
% Facts
male(ender).
male(peter).
male('john paul').
female(valentine).
female(theresa).

parent('john paul', ender).
parent('john paul', peter).
parent('john paul', valentine).
parent(theresa, ender).
parent(theresa, peter).
parent(theresa, valentine).

% Predicates
father(X, Y) :-
	male(X),
	parent(X, Y).
mother(X, Y) :-
	female(X),
	parent(X, Y).

sibling(X, Y) :-
	father(F, X), mother(M, X),
	father(F, Y), mother(M, Y),
	X \= Y.
brother(X, Y) :-
	male(X),
	sibling(X, Y).
sister(X, Y) :-
	female(X),
	sibling(X, Y).

Then starting Prolog I can load the file with the [‘ender.pl’]. form, and ask questions like who is Ender’s sister? Who are Theresa’s kids? Who are Peter’s siblings?

?- ['ender.pl'].
% ender.pl compiled 0.00 sec, 144 bytes
true.

?- sister(Sis, ender).
Sis = valentine ;
false.

?- mother(theresa, Kid).
Kid = ender ;
Kid = peter ;
Kid = valentine.

?- sibling(peter, X).
X = ender ;
X = valentine ;
false.

Well, this is laughably simple, and I’m beyond this already, but it’s good for a first illustration.

By the way, it looks like 90% of elementary Prolog examples use family trees (Nordic or Greek gods, literary, real families…)

Now let’s get in there and learn some more stuff…

Links

Tutorials & Info

Books

Compilers

Code

Categories
Programming

Language of the Month: Lua, part 2

I’m a few days behind schedule since Lua was June’s language, but here it comes, some summary of my experience, after setting things up in the first part.

A Trip to the Moon / Le Voyage dans la Lune
Lua is not this painful, indeed, just the opposite

Overview

I really liked this language, and I’m sure I’ll keep learning it. Very useful and lots of interesting concepts. I haven’t seen an embedded language before, and some parts

Most of what I picked up was from Beginning Lua Programming and Programming in LuaLua Missions was another very useful experience: a “fill in the blank” type of source code collection, which drilled my understanding of the Lua concepts. Some sections felt a little short, so maybe later I’ll think whether there are any more questions I can come up with. Very well worth the time!

There are two things that stand out: tables and local environments.

Tables

If I have to refer to my knowledge of Python, then Lua tables are like Python lists and dicts rolled into one and then some. Here’s one such table:

x = {1, 2, 3, y = "something", ["or else"]="what"}

The first three elements are giving an array (and one can loop over them), then the last two are key-value pairs. One tricky thing is that the “y” there is actually a string (the same as [“y”]), just made it more readable with syntactic sugar. Since variables (references to variables) can be keys as well, this can get confusing if one has a variable called “y”… There are quite a few things about tables that are confusing, but actually pretty consistent so my fuses are not blowing that much as they were in the beginning (for example: how the length is looked up, how to iterate through the different values) .

They are very complex animals so some things are quite complex to do, which in turn enables quite complex things to be done. Pretty much any kind of data structure can be built upon them. So all in all, I’m enjoying them with all their quirks. Also, “weak tables” are an interesting concept (where certain table entries can garbage collected).

Scopes

In Python, the name spaces and environments seem to be on the sidelines (at least what I’ve seen so far): people need it and have to us it, but nothing more than that. In Lua this is taken to a whole new level: scopes can be essential parts of the way algorithms are implemented and neat tricks can be played.

I tried to come up with some good example and failed, here’s a “duh” one:

function newCounter ()
    local i = 0
    return function ()   -- anonymous function
        i = i + 1
        return i
    end
end

c1 = newCounter()
print(c1())  -- 1
print(c1())  -- 2

This is a bit like Python’s “yield” type of generator, without anything special.

Also, in Python there are only two kinds of variables (I think): local and global: either just for you or for everyone. Here they have one more: upvalues, where the variable is local to a parent, so children environment can access them but the parent’s parents cannot. They are strange beasts as well, e.g. the code itself cannot really know if a variable is upvalue or global, only that it can access that.

And of course these scopes help embedding to not just be possible but straightforward and powerful. Local environments can be restricted, some things selectively enabled, and all this transparently, so the script running might not even know that it’s being in the Matrix.

The Lua Experience

Good

  • Fast, both start up and running
  • Language itself is quite small, possible to wrap one’s head around quite easily
  • Written in ANSI C, so basically on every architecture where one can run C, one can have Lua as well
  • Quite a few program has Lua interpreter or interface built in, see the list of these at Wikipedia
  • Tables are great and once one mastered them, are really Swiss Army Knives
  • Built in interpreter (though could use a little love like iPython)
  • The “…” syntax for variable number of arguments
  • Astrological naming convention – modules are rocks, the exercises I linked previously are missions, web framework is Kepler

Bad

  • There seem to be a lot of reserved names (e.g. in metatables) that would be pretty much impossible to figure out with a book (and/or an iPython-like helpful environment), though this is a problem probably only when starting out with Lua

Ugly

  • While some complex things can be solved, some simple things can require complex coding (e.g. in case of the tables when using “__index” and “__newindex“) and there are things that are not even possible. Or I just misunderstood some sections of the book.
  • Using a lot of syntactic sugar (especially in the object oriented section), that I feel need some more practice to pick up, as things can be done in multiple ways
  • The tables’ syntactic sugar covered way of handling string keys, convenient when things go well, mighty confusing at even the slightest problem
  • Too few projects using Lua, too few sources of information (this is not the language’s fault, though)

Project

Originally I wanted to do some useful programming in every language I learn during the Language of the Month series. I failed with Scala (lazy), and now failed with Lua too (underestimating the difficulty). I’m using VLC a lot and it has Lua scripting enabled, so I wanted to do a plugin that can be useful for me later as well. This didn’t work out very well, mainly because it feels the VLC Lua interface is not very well documented, one mostly have to find some examples that exist already and use them. I ended up wasting a lot of time and still don’t quite understand how to do certain things I would like for my code, and don’t even know if it’s possible to do or not. Will come back to it later, until that here are a few links that helped me to get started:

  • VLC Lua FAQ – detailed list of available API functions, but almost no context and examples
  • Lua extensions in the VLC git repo – one can pick up something by checking out those examples
  • Example: Streaming Radio Player Extension – someone who have gone further down this road than me and has a working extension. Very educational, especially to see how concise and straightforward it can be, because the VLC Lua API seems to be thought out well.

Again, hopefully pick this up later (I mean it).

Categories
Programming

Language of the Month: Lua

New month, new language. So far in this series:

I think I do want to amend the original rules set up for this Language of the Month series. At first I though I can write a new project in any language I learn. That is probably too ambitious. So new rules: every month write a new project in the given language OR contribute to an open-source project in that language. This should take away most of the stress and add some social aspect as well. :)

Lua

I’m intrigued by this language because of it’s niche: scripting language within other software. How one sets out to do something like this? What are the requirements for the language, in terms of design, syntax and so on?

Lua programming language logo
See, the moon!

First impressions

So far I was trying out some code snippets and example scripts. First thing to notice was that Lua is pretty darn quick to start. And pretty quick to run as well, though I haven’t used it for very heavy computation yet. E.g. the “get_all_factors” code at the Lua Crash Course (using a larger example number, 1029384756) is timed at ~7ms, wheres the same version on Python (the language that I probably know best and compare other languages to) runs ~10x slower. The snappiness of a scripting language is a surprisingly happy feeling. :)

Tables look very interesting, how the same thing can implement several data structures at once (it’s like Python’s dict, list and kinda-struct at the same time).

Quite intriguing that the Lua Virtual Machine is stack based, probably that’s the reason the language is so embeddable. A while back I was looking for a language with small resource (especially memory) requirements. I had a few suggestions (Dalvik, Forth) that were interesting, but maybe Lua is the one?

The “local” keyword (and the reasons for having it) seems to be a possible source of many harder to debug scenarios. Got to keep this in mind, again not being in Pythonland anymore.

Interesting how the “function” can be defined inline, so looks like there’s essentially no need for a special “lambda” keyword, at the expense of tying a few characters more…

The interpreter is quite well done (with it’s intelligent indentation probably even better then ipython, though would love to see more colours ^^)

As an exercise, here’s a version of FizzBuzz:

function fizzbuzz(num)
   --[[--
   The usual FizzBuzz from Jeff Atwood
   Doing it a bit roundabout to try some Lua language features
   --]]--
   local answer = ""

   if (num % 5) == 0 then
      answer = "Fizz"
   end
   if (num % 3) == 0 then
      answer = answer .. "Buzz"
   end

   if answer ~= "" then
      print(answer)
   else
      print(num)
   end
end

-- Run the thing
for num=1,100,1 do
   fizzbuzz(num)
end

Links

This section will be updated gradually as I find more information.

(Edit: added the second part of this experiment.)

Install

Tutorials

Books

Articles and info

Source of code

Projects using Lua