ClickedyClick

Fighting forum spam

As one of the managers of Ignite Taipei, I’m trying to come up with new ways to let the community communicate, new ways to share information, advice and all. A while ago I have set up a forum at http://bbs.ignitetaipei.tw/ and I thought that will be an interesting experiment. Well, so far it is useless for communication, but turned out to be a very interesting experience from the sysadmin point of view.

I used FluxBB, because it looked simple enough, seemed to be quite fast (for low traffic volume at least), and well configurable. Except that within a very short time I run into a spam problem, so many fake users registered, and lots of algorithmically generated garbage text with a bit of advertisement here and there.

First I looked into FluxBB’s own solutions, and looks like it might not have been a great choice, because many of the spam-fighting plugins are out of date, or not supported anymore, or just a real pain to set up. The immediate practical step I could take was updating my security questions, roll my own version of “written with words, how much is 5 + 4?”, the regular low-tech captcha on FluxBB. Looks like the original answers are already in the database everywhere, so had to write my own set, which seemed to work for a while, cutting down on red-flagged registrations. But it’s not ideal, since I want to make this a dual-language forum (Ignite Taipei has both English & Chinese as official language).

Instead I turned on email confirmation. When someone registers, the password is sent to their email and have to use that to sign in. It was okay for a tiny bit, then crazy registration boom happened. I think I might be the only one real member of the board (I said that it is a failure so far for communication:) and there are 500 other spam members. Looking at their email addresses, it seems all of them have Hotmail. That kinda suggests a giant failure at Hotmail to restrict automatic registration, which is probably a problem overall. I cannot just throw out Hotmail addresses either, because it’s a popular mail provider here in Taiwan too (my first email was Hotmail too, but that was a looooong time ago, before it was Microsoft property).

So captcha don’t work, email don’t work. What to do instead? At the time I was playing around with Cloudflare, to act as an easy to use CDN. I tried it before for our Ignite Taipei blog, which is hosted on Tumblr, and that doesn’t play well with Cloudflare unfortunately. Couldn’t use it for this blog before because of my DNS provider, but now I switched, so started playing with it again.

The dashboard of the Cloudflare interface — Cloudflare stats snapshot (parts of it)

Instead of enabling Cloudlfare for the entire ignitetaipei.tw domain, just turned it on for the forum, since it’s hosted elsewhere. And that totally did it. Spam stopped that very moment, and haven’t returned since. I think what happens is that Cloudflare knows globally a lot of web/forum/email span hosts, and can challenge them or generally ignore them. Can even see where those spammers are coming from.

List of captured threats on the Cloudflare threats console — Cloudflare Threats Console

One weird (but actually not that surprising) thing is that the most active web crawler on the site (Cloudflare gives that info as well) was Baidu by far, so I guess more people knew about the site in China than elsewhere. Why’s that? Some forums that share vulnerable sites, or something like that? I barely had any Chinese content at that time, so it cannot be that. And since I turned on the threat control part, Baidu seem to have dropped quite a bit (submitted the site to Google so now that’s the busiest crawler).

All in all, Cloudflare is an interesting experiment. I can really mess up my DNS with it, and could blocked my own site for several hours, but in general it worth it. Just have to be careful. For example when testing, use their own name servers to check the information, and maybe instead if “automatic” time-to-live, set some very short time first. I usually use Google’s 8.8.8.8, and they pick up the first wrong setting really quickly, then it takes hours to pick up the correction I made just minutes after the first one.

After a bit of playing around, at least I have no spam anymore (keep fingers crossed). Now just have to get people to use the forums. :)

Tags Cloudflare, FluxBB, ignite

Computers

Editing talk videos for Ignite Taipei

Post author By Gergely Imreh
Post date 2013-01-11
1 Comment on Editing talk videos for Ignite Taipei

As an organizer for Ignite Taipei, there are plenty of technical things that I need to take care of. Managing people’s slides (plenty of fun there), runing the presentations (it needs the Ignite type 15s autoadvance), setting up web stream (where everything can go wrong), recording the talks themselves. One thing I haven’t done yet, is editing the videos.

For all previous events we got some more experienced people to help out. The results were okay, nothing surprisingly great but plenty usable. It still did take a lot of time to do that, and I don’t like to ask people favours. Instead this time, for Ignite Taipei #7, after almost a month of procrastination I decided to do it myself. Ignite is an experimental ground for me, where I try a lot of things I think about startups (even if it is a non-profit organization, it can have a startup like “organization”), and I thought if I was a CEO I would make it such that “the buck stops” really here with me. Have to experience the work myself to be able to find better people doing it later. And I like video editing (from the good-old DVD ripping days, that came really handy this time as well).

Recording

The recording was done with my Canon EOS550D. When I got the camera, I could have gotten a 600D, but I thought “I won’t be using this for videos!” Duh! Now I have to put up with freaking large files, because 550D cannot compress them on the fly. Also, need to check whether I can get better SD cards, because at 1080p it broke the recording last time (that’s why the Ignite #6 videos are still not edited). 720p (at 50fps) seemed to work somehow, and looks good too. Will try to fix that still in the future. Now I have a GoPro Hero3, that can do 1080p without even breaking sweat, and I plan to use that one for the next Ignite.

Title screen

A video starts with good title screen. Previous editors could manage animations and all the works, which I didn’t have any idea how to do this time, so just keep it simple: a static page. But still have to add my own spice: not just a simple title and header, but something good-looking, and expressing the spirit of our organization. So got a cool HDR photo of Taipei City (well, actually Zhonghe, the suburbs to the south) from my friend, Chia-Da Hsu, tuned it a bit to be a background image, and plugged it into Inkscape.

Editing window of Inkscape — Creating the video titles in Inkscape

I like Inkscape a lot, open source, capable, fun.

Put in the background, position the text. The font used by Ignite, and thus for most of our own text is Insignia LT. To make it more readable (20% gray setting), make a black outline to the text: Copy/Paste in Place/Stroke paint to black/Make it thick (20px thickness for us), and blur with a suitable parameter value (was 2 for us), then move the layer one below. Voila, outlined text, drapper. Just had to make one, then replace the text (and recreate the outline).

Editing

I needed to put together the video with the slides, side by side, with synchronized timing, so it would look similar to the original talk. First, I needed to get the slides in image format, that I keep doing with ImageMagick, whit this script:

Unfortunately, the ImageMagick on my ArchLinux seem to be chocking on this script, that used to work so well 6 months ago. Looks like it outputs only the first slide. I had another computer with an older version of ImageMagick and it worked like a charm. Got to investigate it – or wait until the next Ignite and hope the problem goes away.

Next I had to find a suitable software for editing. As much as I looked, Linux didn’t have many good ones, or even many of any kind. Avidemux don’t like MP4, and it has bitten me in the butt before, Cinelerra didn’t work for some other reason, and these are the two in the official ArchLinux repo that I could find. Had to look more to get to Flowblade, an editor written in Python. I got to say, it’s such an amazing bugfest, that it drove me nuts sometimes, but got the work done, and that’s more than I can say about the rest of the candidates.

Combining the video on one channel, and the slide images on the other channel, translation to move them side by side, then finally a blending compositor so that the layers don’t overlap.

Flowblade window during editing — Combining the video and slides in Flowblade

Looks quite good, was a rocky start to figure out how it went, but when it went well, then I got one talk done in about 10 minutes, which is great! It helps that the Ignite format is predictable (15s each, could just add an image, and found the new insertion point for the next one in a very short time. Though now I see that Acrobat Reader that I use to display the slides, don’t do accurate timing usually.

Then it came the rendering. Originally I used a pre-compressed version of the videos (from 2Gb for 5 minutes to about 200MB), but the quality just wasn’t that good after the joint rendering. Went back to the original source, and tweaked the parameters. In the end, discarded the entire stock parameter setting, and used these ones manually. To get a good result, the important ones are preset (from ultrafast to veryslow, increasing quality) and crf (lower number means better quality, 18 is good, 17 a bit overkill because I like that, about 20% larger file than 18):

Too bad that Flowblade cannot do 2-pass encoding, maybe can do that another time differently, for better quality and smaller filesize. Also, had to make sure that I marked the beginning and the end of the video well and just rendered that part, otherwise it crashed right at the end of the job. Not good, considering that it took about 40 minutes each to be done.

Flowblade is unfortunately using absolute paths to remember things, and cannot edit the settings very easily, so couldn’t move the rendering onto a more powerful machine. Maybe could make a tool for save-file conversion, it’s all Python anyways…

When that done, I remembered from listening to a lot of Youtube videos, that the sound had something to desire, and all I had was puny DSLR microphone recording, so I run it through a normalization to tweak the level of the audio.

Upload

Time to upload to Youtube. Fortunately the network is great here in Taiwan, so it was a very short time, sometimes the postprocessing took longer than the actual upload.

The Youtube video info editing window — Editing all the options for the uploaded video on Youtube

From the settings I always go for Creative Commons, not that anyone had remixed our videos yet. Also set some tags, maybe that can improve our visibility in search, as well as the recording date and location. The description writing is tricky. Most of our videos are found through recommendation on the Youtube website, so it should be good. I wonder what would make it work better?

Ready to share

All that done, just created a playlist to tie the videos together, and found this new (to me at least) look, which is not too bad:

Youtube playlist window with all the uploaded videos — Finally, got the playlist ready

Now, after 2 days we have over 200 views, which is not bad considering that I could barely advertise yet, only friends sharing on Facebook and such (and you know how Facebook hides everyone’s precious updates, so it’s a surprise we even have this many).

Finally, the results can be all enjoyed.

Tags Flowblade, ignite, Inkscape, video editing, Youtube

Life

Counting down to the next

Post author By Gergely Imreh
Post date 2012-12-31
No Comments on Counting down to the next

Since living in Taiwan, the Western style new year feels a bit arbitrary for me. The Lunar New Year is at least tied to something, it is easy to define, even if it’s more complicated in its own traditions. Nevertheless, I like the New Year, and it’s good to have some periodic semi-forced review time. When it’s cold, it’s even better (sorry Southern Hemisphere).

This year felt a bit like growing up. I was grown-up before (can’t afford not to be at the age of 31), but my way of thinking have matured a lot, and it feels like that happened because of all the limitations and stress and projects and winding path I took on.

In 2012 somehow extended on the projects I worked on, like building a certain portfolio. Ignite Taipei is still there (maybe the main drive behind my experimentation), while it got company in the form of Future Shorts Taipei, the continuation of the Geek Dinners, later OpenHack Taipei and the (preliminary) Taipei Hackerspace. They seem to have a common theme – building local events related to technology, innovation and art. I hope I can continue with this theme, and see where it takes me. Kickstarter, social networks helped a lot to expand my world in these directions. Not sure if all these projects will survive next year (at least not with me at the wheel), but will definitely learn from all of them.

There were some really good books this year, discovered Pynchon, Tolstoy, and Calvino, as well as many smaller ones. Keep rediscovering Tolkien, Asimov, Murakami, Vonnegut, Beckett. Had a lot of brainy-books too. My shelf is still totally full, I can be sure, that the next year will be just as busy in terms of reading and just as mind-blowing.

In research I wasn’t that successful, though I have learned a lot. Will see what can we do in the next year in the lab. We were building a new experiment and it is not a real surprise that the results are still ahead. Whether my true way is down on academia lane or not, that remains to be seen.

I have a lot of startups related influence around me, and for years I’ve been hatching plans without any results. Next year supposed to see me set some hard deadlines for decisions (different visa for Taiwan, that lets me do more, for example).

There were some great travels this year, a big European tour for the first time since I moved to Asia, and in Japan for the first time ever. No plans for travel for the next year yet, maybe more within the country and more purposeful choices of destinations? Though I like quick changes of plans and surprises. And visitor friends from countries are good too.

This time it feels strongly that I’m in “in-between”, with choices ahead. The Sun is already setting on 2012 here, let’s make 2013 count.

Tags projects, review, year in review

Programming

The Gift of Code

I like advent calendars a lot. They can bring a lot of surprise, preparation, focus, and joy. They can come in many shapes and forms, and they encourage DIY – make your own calendar, count the things that are important.

This year, I got to play with a very interesting “advent calendar”, called 24PullRequests. It is the kind of thing that I don’t understand why people haven’t done it before. The mission: help out open source projects by submitting enhancements and fixes (i.e. “pull requests“), and do that for 24 days counting down to Christmas.

I had to take part in that one, and while the result wasn’t as successful as I wanted, it was so far my best contribution to open source.

My pull requests

Instead of 24, I managed to make 4 enhancements that were ready to be sent off. That’s not consoling me that it seems nobody managed 24, but never mind. Here are the things I made:

SmoothieCharts: make the charts use newer browser animation technologies that have better performance, and save on battery as well. This one was prepared somewhat earlier than December, but the final version was pushed within the right time frame. Being tested, not merged yet.
OpenHack: I’m organizing the event in Taipei, and noticed that some other place has broken image link. Hunted down the same pic from Google Cache, and set it up again.
Python Guide: added some info about installing certain Python packages in Arch Linux and Ubuntu. This is embarrassingly tiny fix, there’s so much more to do here
AngularJS: this is fixing that one couldn’t run the build script if the system Java can’t run in 32bit mode. I didn’t know that this was a Google project, until they sent me a request to sign some contributor agreement. I feel strangely humbled.

Lessons learned

Four contributions were already a lot of experience, because all of them were so different. Here are some lessons learned:

Write good pull requests – that starts with writing good commit message! People keep saying that, but seriously, no excuse not to do that.
When the changes have been sent in, don’t mind that they are not accepted yet. Every project have their own pace. Keep working on whatever you like
I was looking for ow hanging fruit, but one has to go in there still to make some meaningful contribution.
The issue tracker is a good start to see what to fix, but not always helpful, as it can be difficult to understand what propblem the others try to describe, if you are new to the project. On the other hand, try to use the code, I’m sure you’ll find some pain points right away (that was AngularJS). Also, the busiest issue trackers are not the best, they are full of things that would side-track you for a long time. Projects with a medium count are good for such an improve-and-run contribution.
Don’t be afraid to do things, but still do them the best you can. Your contribution doesn’t always feel meaningful, but still a little improvement is more than most people do. (just like PythonGuide was)
Keep things simple – easier to do, easier to pull. Even if sometimes that takes longer to write (the AngularJS contribution srunk to the quarter of its size while I was trying to figure out the simplest way to achieve what I wanted)
If interested, don’t worry if the project uses programming language you don’t know. You can pick up new things easier than it seems. Also, many projects give you feedback on your contribution, to help you improve it.
This project don’t encourage to work on your own stuff, but that doesn’t matter, there are another 11 months for that, or every day after these contributions are done
How to do this for the whole year? Bug squashing day in general? Still need to get deeper in projects, but go and explore. Can also see CodeTriage and ContribHub, linked from 24PullRequests
If stuck in the fixing, but the problem is interesting, don’t worry if it doesn’t fit in the 24 days. Keep working on it, the recipients will be happy any time (I have have 1 or 2 such patches)

Now let’s be a better coder in 2013.

Tags github, open source

Life Thinking

My 40-by-40 plan

A while ago I was reading someone’s 30 by 30 plan, 30 things to accomplish before the author turns 30 years old. Since I always liked lists, it got me interested. Since I’ve just turned 32 a few days ago, that other ship has sailed a long time ago, but then I can set my even more ambitious list: 40 by 40.

It took a 2 or 3 brainstorming sessions, here are the results, not any particular order, just numbering it to keep track, and a bit of explanation here and there:

Travelling to: Chile, Africa (somewhere), Southern Hymalayas, a desert, Hokkaido (because seeing new places always open new dimensions)
Create a new programming language (programming so much, maybe can make something new, which would me to appreciate the other languages even more)
Become professor (that’s the science career path and working this long for other professors, I wanna see how I can be in their place)
Publish a book (reading so much, writing quite a bit, can I really write something others want to read?)
Start a company (the independence and creativity it would give me is hard to overestimate)
Travel to space (unlikely, but let’s just set the goals high)
Learn to play a musical instrument, current candidates: piano, trombone, or guzhen (I’m listening to so much music, want to be able to create something myself)
Have an art exhibition (creativity knows no bounds, and interactive art is awesome, would be great to collaborate too)
Be in a film (I’m a terrible actor, but still want to see what work goes into all the movies I like to watch so much)
Sail on the ocean (adventure time, real travel, real toughness)
Go skydiving (that must be a feeling impossible to describe, have to see it myself)
Send my parents travelling somewhere where they wanted to go (I was lucky to have travelled more than most of my family, let’s pay back some of that while possible)
Set up a foundation for some good cause (I was thinking about this quite a bit, I wonder how can I help the most?)
Learn to swim properly (being a frong is just not enough)
Run a marathon (I guess I’m already doing this wrong, but want to see what does it feel like)
Do a defensive driving course (haven’t driven for almost a decade, but the way people drive in Taiwan, this would be a very important skill to learn)
Write to my inspirational people (though first really have to figure out who are they, have some but need time to think)
Learn to meditate properly (have tried it a few times with guidance and it is simply life-altering experience, too bad single times fade, I want to have the habit)
Create a high traffic website (can’t read Hacker News day in and day out without having this ambition)
Become fluent in Chinese and one other language: Japanese, Spanish, French, or something else (languages are awesome, love to communicate with people)
Long distance bicycle trip to somewhere (many people do that around Taiwan, and used to cycle much more than these days, is a great way to get around)
Learn to brew really good coffee (these days I do drink a lot of coffee, but very few places make it such that it stands out, I want to see what it takes)
Learn the constellations and other important object on the sky, be able to navigate by night (used to do much more astronomy as a kid, have to rediscover the universe)
Get to know a traditional profession deeper, like carpentry or pottery. (there’s an awful lot to geek out about those)
Build a building, or renovate an old one (break the mystery of the places where I stay everyday)
Grow a tree (tried many times, and they have failed very early)
Learn to be a technical writer (that’s the influence of my advisers from Oxford)
Bring traditional Taiwanese food cooking back to Hungary, and Hungarian back here to Taiwan (cooking is a lot of fun, and dinner parties with people are even more)
Overcome helplessness about the issues I see around me everyday (stray animals, homeless people, poverty, all could be improved upon if I can face them)
Give away at least half of my stuff (I have too many things, need to simplify)
Get back in touch with my childhood, teenager and uni friends (I have burned so many bridges by inaction)
Learn to mix cocktails well – and come up with a new one that people actually like (cocktails are playground, and even though I can’t drink, I was reading a lot about them)
Attend the Nobel prize ceremony (preferably when someone I know won the prize)
Do something concrete for science education (also inspired by my high school friend who’s doing an amazing job at that)
Write up our family tree and as much of our history as possible (I love history, and very interested in my family)
Have a patent of an actually useful invention (then give it away)
Have a radio show (had some small stuff before, just feel like doing it)
Figure out how to donate blood again (they wouldn’t let me here because of being grown up in Europe)
Learn to fly helicopter (somehow much more tempting than aircraft)
Overcome the fear of heights (that would be very useful)

While making this list, though, I was thinking: this is all good and nice, but I care much less about it than I thought in the beginning. These would be nice, but I actually have 1-by-40 plan: be happy.

Let’s see how does that go. :)

Tags lifeplan