Short Status Review for Technical Blog

Markdown syntax is really helpful when you need to create structured text. I'm having a lot of problems in the WYSIWYG-Editors of all the blog portals when dealing with tags, escaping and code blocks. For example wanting to talk about the [sourcecode] tag and later having an actual [sourcecode] ... [/sourcecode] listing will not work in Wordpress. A nice article about the dilemma with WYSIWYG-Editors can be found here.

In Markdown you simply indent the sourcecode by 4 spaces and don't need to worry about escaping characters or tags. As the Markdown Reference puts it nicely:

This makes it very easy to include example HTML source code using Markdown. [...] Regular Markdown syntax is not processed within code blocks. E.g., asterisks are just literal asterisks within a code block. This means it’s also easy to use Markdown to write about Markdown’s own syntax.

The only downside is that in standard Markdown you cannot add the class for syntax highlighting on the <pre> that is generated. Instead of indenting you could also go with a "plain" HTML block:

<pre class="brush: javascript">
    for(var i=0; i<n; i++){
      bounds.extend(markers[i].getPosition());
    });
</pre>

Unfortunately this doesn't work as the less-sign is not automatically converted to its HTML entity because within a HTML block no markdown parsing is made.

In the simplest workaround you would simply replace all ampersands and brackets with the verbose HTML entities. But this kind of defeats Markdown's philosophy of being as easy-to-read and easy-to-write as feasible. So I will take a look at some of Markdown's dialects and extensions, like Maruku, as well as other lightweigt markup languages like Textile.

Chosing a Technical Blogging Portal

Recently I have thought of creating an extra blog for the more technical stuff I post, i.e. program listings and solutions to problems. Those kind of posts usually include some source code that I want to be displayed as preformated text and with syntax highlighting. So I looked at the source code posting features of three of the most popular blogging platforms:

• Blogger.com
• WordPress.com
• Posterous.com

What I did in Blogger so far (as you can see in my older posts) is to simply change the font type to Courier and decrease the font size. That kind of works for small listings, but doesn't look pretty. It get's really messy when you try to post longer lines of source code though. To improve things and get syntax highlighting working what you can do is modify the header part of your template to include a javascript highlighter. Alex Gorbatchev's SyntaxHighlighter seems to be a very popular choice. It is self contained, brings 23 Syntaxes with it and is used by big players like Apache, Yahoo and even Wordpress.

Following the installation instructions you must do the following to get it running reference the core javascript, css and theme files (3 lines) references all the syntaxes that you need (up to 23 lines) set blogger.com mode (and other default mode) and then call the syntax highlighter javascript function (>=2 lines) The large number of lines may seem intimidating at first, but it's pretty much just javascript and css references and you don't need all the files. The actual source code is then embedded in pre or javascript tags. Customization parameters can be passed class values on that tag.

As mentioned above WordPress.com includes Alex Gorbatchev's Syntax higlighter by default, which is nice; the usage is even simplified as instead of using the above HTML tags you can just wrap your source code in [sourcecode] tags where you also put some the options. Take a look at the support page. Example:

    [sourcecode]
    This is a short snippit of code with padlinenumbers set to 4.
    [/sourcecode]

Pretty neat and not as verbose as using pure HTML. Posterous goes a similar way, but seems to have problems with line spacing. And the HTMl editor is really not that usable, especially if you created a blog by E-Mail. But Posterous supports markdown syntax out of the box now, which is kinda neat; you just have to wrap the (markdown) content in tags. That let's you embed source code in "regular" markdown syntax by indenting it (pitfall: at least 4 spaces for posterous). Language is set up with a she-bang. So you would have:

    #!c++
    void main(){
      cout << "hello world!";
    }

As mentioned before, though, the HTML editor is a pain to use and won't work if you sent your markdown by E-Mail; I think it's a bug. Still having the markdown option built-in is nice. Blogger.com and WordPress.com won't allow you to write your posts in markdown.

In conclusion, SyntaxHighlighter seems to provide very nice looking code listings and gives you a lot of configuration options. If you don't want to mess with the HTML of your layout files go with WordPress.com that gives you easier access to it. If you want to write markdown online or in the email your only choice seems to be Posterous. But if you can live with writing your markdown and running it through a HTML converter you could use every blogging service here. Overall Wordpress.com seems to be the winner for maintaining a technical blog. So I probably will be looking into Wordpress for my technical blog that I plan to create. Opinions?

Colophon

I have added the above mentioned changes to my blogger template and used markdown to reformat this blog, parse it and then paste the HTML. My "source" markdown looks as follows:

<pre class="brush: c++">
    void main(){
      cout << "hello world!";
    }
</pre>

About blog writing and why it's important for me

I realize it has been almost 5 months since I wrote the last article on this blog. I also realize that it is not that easy to regularly write articles and that might be reason I haven't followed up until now.  Finally I have come to the conclusion that despite all of that I should continue writing articles and do so more often for the following reasons:

First, in the plethora of RSS-feeds and podcast I regularly listen to entrepreneurs emphasize how important good writing skills are, especially being able to write concisely. "Hire good writers", says 37 Signals in their book Getting Real because good writing skills will pay off. Or like the English proverb goes:  "The quill is mightier than the sword".  And it is only by writing that you will get better at writing. So writing regularly on your blog makes a good practice.

A big advantage of writing on your blog is that you practice the entire writing process. You will do  drafting, revising and fine-tuning before you publish your post. Why? Because virtually everyone might read the stuff you put on your blog. And if you go through the pain of keeping a blog it must be way better than any Twitter micro-content. Otherwise your blog wouldn't have the right to exist. It's almost the same phenomenon as with open source software where programmers pay attention to good design and style for similar reasons.

Second, by writing a blog you are effectively creating a personal brand. Your blog becomes a showcase of you and your work.  At the same time, however, you can leverage your blog for marketing your internet products. That can be a of real advantage if you need to push your internet site's traffic and ranking. It's like free SEO and advertisements together.

Lastly, I think that reviewing your old blog entries is real fun. Reading that one or two year old text that you ones wrote makes you kinda proud. In the end you're doing it for yourself so that writing a blog still makes sense if you don't have an audience.

For all of above reasons I have decided to create this blog entry and continue adding more in the future.  Surely I yet have to learn how to write concisely because this text could probably be half the size. I should practice more. But for now I'm going with Mark Twain's excuse: "I didn't have the time to write a short letter, so I wrote a long one instead."

Making VIM more TextMate like

TextMate is supposedly the text editor for MacOS to have. It's used by many Rails developers and I must admit it's quite slick and fast. It feels like a polished and better Emacs in my opinion and takes many of its keyboard shortcuts.  Unfortunately it's closed source, costs money and is only available on the MacOS platform.

I have converted back to VIM (I am using the MacVIM version). One of  features of TextMate that I am missing is the project drawer. Fortunately you can extend VIM with plugins. I have tried two plugins: Project and NERDTree. If you want something TextMate like go with NERDTree. You get a directory pane at the left, from which you can easily find and open files.

That's quite fine, but as a TextMate fan you will be missing the command line mate program to start with a directory "drawn out". I wrote a simple shell script to mimic that behavior and now typing 'vmate' behaves like its Mac OS counterpart:

function vmate(){

  if [ -d $1 ]; then

    mvim -c ":NERDTree $1"

  else

    mvim $1

  fi

}

To open a directory in project mode simply pass directory path to vmate. If you doing any Rails development be sure to check out the great rails.vim plugin.

How to get to Orly Airport and visit Chinatown

Last time when I went to Paris, my flight was departing from Orly Airport, the second largest airport in Paris. Many cheaper Airlines operate at this airport and it seems only be used for shorter distance flights, within the EU. Orly is located in the South of Paris and reachable by a number of different ways.

The cheapest way to get to the airport from the center of Paris is taking the subway all the way to  "Port de Choisy" station on line 7. You need one ticket T+ ticket for that. Then you get off and  take the 183 bus straight to the Orly airport with another T+ ticket. That's two tickets for a total of only 2.30 Euro! Be prepared that this bus ride will last another hour. AND BE SURE to take the right 183 bus! Not all the 183 buses go to all the way to the airport. There are two bus stops. Take the one on you left, but check the bus caption and timetable. There is an electronic display that shows you where the next two buses go. If unsure you better ask the bus driver before boarding. I didn't know that and almost missed my flight because the 183 bus to Orly Airport only departs every 30 minutes or so.  I  was too fascinated by the area around "Port de Choisy".

If you cross the street from where the bus stop is you land in Chinatown! I don't know if this district is  mentioned in any guide book; at least in mine it was not. I didn't expect to find so many Chinese shops, restaurants and Chinese people there: mothers with their children, elderly Chinese in the park, workers  coming home eating at restaurants or buying Chinese food in one of the many Chinese shops there. It's definitely worth a visit. And if you need to go to Orly Airport anyway, it makes up a good combination; have an authentic Chinese meal at one of the many restaurants or buy some snacks for your flight.

If I needed to go to Orly again, I would definitely do it in the same way. It may take a little bit longer than the train, but it's cheaper and you get to see some outer parts of the city that most tourist even don't know of. Just be sure too have enough time on hand.

Visit Pompidou

When I went to Paris last week, I made sure I go to the famous and historic Louvre and d'Orsay Museums. Both are stunning Museums and at least one should be on the list of your sightseing activities when you visit Paris. If I had to choose between one of those for time reasons I'd probably go with the later one because it is not as gigantic as the Louvre. I saw both, but was very selective in what I wanted to look at. I didn't spent more than 2 hours in each and pretty much headed right to main "attractions": the Mona Lisa in the Louvre, and the Van Gogh self Portrait in the d'Orsay.

But since I got myself a 2-day Museum Pass and the Centre Pompidou was on the list of its accessible Museums, I decided to go there too -  and I should spent more time in it than in the Louvre. The building itself is quite an attraction: all tubes and cables run outside the building; that is, there is no concrete that covers them, the building is see-through. Taking a transparent tube escalator upstairs you enter the museum and exhibition halls. It's modern art from the 20th century to the present: famous paintings from Piccasso, Mantisse, Leger, and abstract construction, installations, lots of naked skin and body art.  Maybe not necessarily the family or work-safe museum - but then after all, it's just art! Be sure not to miss it on your next Paris trip. Since entrance fee with 12 euros is quite steep, consider getting a Museum Pass if you visit another museum as well.

Photo from one of the exhibition rooms of Centre Pompidiou.

There are a lot of art students there practicing drawing.
(shot with my Canon A710is).

About taking photos and snapshots

Yesterday I came back from a Trip from to Paris. And like on every trip I go I took my camera with me. And it seems with every trip I am taking more and more pictures.

The ubiquitous availability of digital cameras, big and cheap memory cards and rechargeable batteries allow us to take endless pictures. Taking a digital picture is virtually free. The marginal price for taking another picture consists merely  of a fraction of the price for the power you need for charging your battery; and since you have a power outlet in any decent hotel or youth hostel, the marginal cost is really zero.

So if taking digital pictures is free, why shouldn't I take as many as I could? That is what most tourist seem to think nowadays. Every time there were many tourists taking picture of something, I just felt the urge to take pictures too. Well, that is snapshots. Snapshots are pictures that are taken out of the urge to take pictures, with no composition or thought about the motive. Snapshots are all right for your memories, but they all look the same.

And even though shooting digital pictures might cost no money, it consumes some of  your precious traveling time. Sometimes seeing, absorbing and understanding your environment makes more sense than a picture.  Or like Kevin Spacey says in a commerci al: "Point-and-shoot? That's a crime."

Korean Finger Food [Photo]

Korean Fast Food at the Franfurt Main Railway Station

(Shot with my 1MP Camera on my Motorola Razr and edited in iPhoto)

Some thoughts about the iPad

As I had correctly guessed, Apple enters the ebook market with its iPad introducing big competition to the Amazon Kindle. Unlike on the Kindle, though, users can consume more kind of contents than just eBooks: movies, music and applications. The iPad is coming out next Month, and I have thought about whether or not I'm getting one.

The iPad is advertised as a multi-purpose device for consuming and experiencing purchased content and the internet; Alex calls the iPad a "cynical .. digital consumption device". But as many 
have pointed out the iPad just seems like a bigger version of an iPhone/iPod touch (Have a look at this funny depiction). So I asked myself is it really worth the purchase? What is it that it can really do better than a notebook or a smartphone?

According to Steve there are many things that the iPad does really better; for me there is only the advantage that, due to its larger screen real estate, reading gets much more comfortable; this includes eBooks, as well as articles or documentation from the Internet. Other than that I am pretty sure, that a smart phone suffices my mobile connectivity needs, and a notebook my needs for getting some serious work done.

Unfortunately the battery supposedly lasts only (or until up to) 10 hours, which might be an annoyance if you use the device primarily for reading. So wouln't one be better of with a Kindle, if reading was one's primary use case. Unfortunately, the Kindle only has a monochrome, non-touch display and is a "consuming machine" itself. So reading your papers or PDFs is as far as I know not possible.  And the  Kindle is not really cheaper than the iPad.

Just looking at the technical and sexy attractiveness, the iPad clearly beats the Kindle hands down. But as a personal computer it lacks one quality trait: openness (see Alex). As on the iPhone there will be probably "root" hacks, that allow will allow me to tinkle around on the device. But doesn't change anything on the  Apple's iPad policy. I'm still not sure I wanna get an iPad when such great Apps like Stanza are "censored" maybe making me only buy books from Apple's own bookstore in the near future. But maybe the reading experience on the iPad just beats every other eBook reader. Let's see.

From hunt-and-pecking to browsing the WWW

Playing around with a lot of different Firefox plugins over the last few days, I have come to realize that the way we consume Internet content has been undergoing a change. What had been targeted fishing for small fish  in the huge information ocean, has now been transformed into whale like soaking. Instead of searching from a vantage point for well defined prey, we just swallow tons of water into us and hope to find some nutrition in it.

Streams of information are the product of social networks and recommender systems. What begins as a small creek, a twitter status update for instance, accumulates to streams, rivers and oceans. Trough the act of consuming those information streams our surfing behavior is changed. We click on links to sites we never would have visited, google for things that we never heart of, and subscribe to even more information streams we didn't even know existed.

Twitter is just one example. Take all this other stuff we are fed with every day: Facebook, Stumbler, Delicious, Last.Fm, Glued, ... All those applications generate endless information rugs that we wade through with our personal quivers. And at the end of the day we come home with new catches of internet addresses and content pieces.

Why did I mention the Firefox plugins? Having installed extensions for many of the social network sites mentioned I find myself more than often not using the address bar to navigate to a certain site. Instead I just follow the latest tweets or hit a key combination to stumble over new information; and if it's delicious I might put it in my Facebook for every one else to browse.

Goethe monument

The Goethe monument in Frankfurt, Goethe's birth city (Taken with my 1MP Motorola RAZR camera)

My Editorssey

My Editorssey
--------------------

I don't want to know how much time I have wasted finding the best editor for my needs. Have I found it yet? Well, not really. I am using still several editors on a regular basis depending what task I am working on or what system I am using. Was it worth the time spent to try and learn new editors? Not really. But it was fun though. Let me take you on a small journey.

Back in the days of Windows 95 I started with the built-in Notepad for tweaking system files like AUTOEXEC.BAT and CONFIG.SYS. When I booted in DOS only mode to tweak expanded memory for ancient games like Mortal Kombat and Destruction Derby I made use of the good old EDIT editor with its dazzling light letters on a just as dazzling light blue background. The basic editors were so basic that I wouldn't actually think about them at all. They were just necessary tools and came with the system.

Then in middle school I started programming. I bought the MSDN Visual Studio student edition - an Integrated Development Environment (IDE) that brought its own editor. Programming for me at that time was opening Visual Basic Studio, writing and compiling code in it. How could I know that I was just editing plain text files I could have edited with Notepad, EDIT or any other text editor as well.

After a while I realized that learning Visual Basic from a Dummy book wasn't really teaching me about the gits and guts of programming. I wanted to look behind the Draw-Your-program-Drag-and-Dropping curtain. I got started with C and C++ programming, wrote console programs and ,after I got The Petzold - the Windows API bible - for free from some stranger on the Usenet, even GUI programs. Since I edited all my code in the IDE instead of drawing them in Visual Basic I even got to understand the concepts of the Windows API.

At some point I even wanted to go more low level - assembler and hex level. I wanted to hack a latin program for vocab learning and I needed an editor that would allow me to edit binary files in Hex mode. I came across UtraEdit and I really liked it. It was way more advanced than the packaged Notepad, had syntax highlighting, a sleek and professional looking appearance to it. Using it I was eventually able to crack the time limit -- a huge feeling of satisfaction and power. I continued using UltraEdit for programming C/C++. I could tweak the editor to compile programs and even run them -- totally sufficient for the small command line programs I wrote at that time. "Who needs an IDE, anyway" I thought.

Soon I would get introduced to one again, though. In computer class in high school they made us use the Turbo C++ IDE, an DOS IDE. It reminded me of the EDIT editor having almost the same color scheme. To be honest it felt more like an editor than an IDE, maybe that was the reason why I liked it. I even used it to write and coordinate a Tank Wars game clone as president of the computer club.

I tried out Linux several times some teachers and friends swore by it. I gave up several times, either didn't get X up running or a mouse installed. But maybe the main reason was the sheer amount of text files that you had to edit and I didn't know how to use any of the unix editors at that time. Frankly, I didn't bother to learn them. I was too frustrated with Linux. It was only after Knoppix came out - a Debian based live CD that had all the drivers for my system and worked right after booting up, that I became again interested in Linux. I signed up for the LPIC exam and started with learning an editor; my book said there were two: Emacs and VI. I chose VI. How did I learn it? ":help tutor" was all it took and after around 30 minutes I mastered all the basic movement and editing commands. Linux started to be fun. Right after finishing the quick start guide I continued to read more of the manual and soon VIM turned out to be a powerful tool and captivating toy at the same time. It is not exaggerated when I say that I learned Linux by learning VIM.

When I came to university we were taught Pine and Emacs on Solaris machines; but since I already knew enough Unix, I cold-shouldered those programs and modified my environment to use mutt and VIM instead.
VIM did all the things I needed to do and it did it beautifully. Using the commands and modifiers made me feel I was more efficient doing my work than other students using Emacs or KEdit. I began tweaking VIM to become more efficient -- ignoring the time it cost to tweak VIM of course.

I even started to do my homework and write letters with VIM writing everything in LaTeX. The postscript results were beautiful and I thought I would never need another text processing program again in my life. I looked for a Windows VIM version for my first laptop and installed the Cygwin environment on my Windows computer (Linux didn't run it sufficienty enough). I disregarded Emacs and never thought that I would bother learning it until I had to write my master thesis. Maybe in an act of procrastination I began learning Emacs because somewhere on the Internet I read that the AucTex mode was the ne plus ultra for editing LaTeX documents. So I learned Emacs meticulously - with all its cryptic key combinations, and strange major and minor modes and with Elisp - the Emacs programming language. I tweaked and wasted more time time than anticipated - days, a week - before I was somewhat confident in speaking "Emacs". Oh yeah, of course, I had to pimp the look and appearance of my Windows Emacs installation too, before I could begin writing my thesis. I always told myself that the LaTeX typeset output was worth it and that it was more safe to Edit plain text files than write in Word of OpenOffice. I told myself that the write feeling on Emacs was just different and actually let me write more and better; I still have the feeling that typing in Emacs just feels different from VI - I don't mean the key combinations or shortcuts, but actually the feeling of typing (Anybody feel the same?). Emacs felt more like a text processor -- so I felt it was the right choice for my master thesis.

I eventually finished my thesis with Emacs and stayed with it, even for programming - at least for my personal stuff. At work I had to use Eclipse because everyone else used it and it was just best to use it I was told. I was shocked that my colleagues were so conservative and myopical, considering Eclipse the only way to program Java just like I did with Visual Studio back in the days. Finally I reluctantly submitted to their request to use Eclipse. It certainly was more convenient than writing Ant or Maven by yourself, or converting all the Eclipse projects that they dilettantishly commited as a whole into their CVS. For them Java seemed to be Eclipse - that was my annoyance, not Eclipse itself.

Around that time I also had started to write Web applications with Ruby on Rails. Not being satisfied with neither Emacs nor Eclipse for that task, I went with the Netbeans IDE. With its sleeker and cleaner interface it just seemed to suit the lean and smart Rails paradigm better.

The Netbeans interface was also more Mac-like. I converted to Mac with a Macbook Pro, my second notebook. Everyone seemed to praise Textmate -- the über-editor for the Mac. And the editor of choice for many Rails developers.  It certainly looks nice, but I never bothered to learn it as meticulously as VIM or Emacs because I thought: "What's the point if you can only use it on a Mac?" So I tried out all major Emacs distributions (Aquamacs, Emacs.app, Carbon Emacs). After all Mac OSX is an Unix system under hood.

Yes, Emacs with its cryptic key strokes, modes and Lisp dialect, the almighty text editor that does all and everything for its disciples had even became a religion as noted by its inventor Richard Stallmann. But being an all-in-one device comes at the price of complexity -- some call Emacs an operating system. Recapitulating I don't like the steep learning curve and the huge and messy configuration files, that one accumulates over time. Even on my dual core Macbook Pro booting the beast up felt sluggish at times.

Yesterday I decided I needed a modern editor, something newer than the 35 year(!) old Emacs or VI. After some googling I realized that there simple was nothing as powerful and ubiquitous as those two classic editors. But sine I wanted a change, I went back to VIM. I installed the MacVIM distribution from macports and I'm liking it. It feels fast, sleek and clean and editing files in it is pure fun. You don't need thousands of lines configuration to start typing and navigating away quickly. Using the right color theme (desert at the moment), you even get a Textmate look. VIM is a relief from Emacs. And not only my pinky thanks me for it. Old memories from my first Unix day become lively. Storm and stress, programming for fun. Just the right mood I need for learning some Python these days. Compiling VIM with the +python feature I can even extend the editor with Python and avoid the arcane Vimscript.

My editor Odyssey is certainly not over yet. While I have wasted a lot of time for learning different editors and IDEs so far, I have also learned a lot of new things during that progress. It is true, that sometimes I have spent more time tweaking my editors than doing actual work. But tweaking isn't that what constitutes the excitement about computers anyway? At the moment I am using OpenOffice for word processing, Netbeans for Rails programing, Eclipse for Java and VIM for all other editing tasks. There is no swiss army editor for all my writing or editing tasks. So "religious" statements from fanatic editor disciples about extensive economics when learning a new editor are far more than dubious.

Colophon: This blog entry was written with aforementioned MacVim with linebreak (:help linebreak) set, mapping of the cursor keys to move by visible rather than "physical" lines (:help 25.4) and spell checking enabled (:set spell) so that it almost feels like a real word processor ;)

Calculate total video duration

Here is a little python shell script that uses the "ffmpeg" program to calculate the total duration for specified movies on the command line. This can be quite useful to calculate the required bitrate for encoding a bunch of to fit on a DVD as described in my earlier post. The calculated bitrate is adjusted by a parameter called "extra" to give you an extra buffer for the VBR of MPEG2 stream. So here it is:

#!/usr/bin/env python

"""calculates total movie length of all movies passed as parameters"""

import sys, re, os, subprocess, getopt

def main():
    sum = 0
    target_size = h2f("4.7G")
    audio_rate = h2f("224k")
    extra_rate = h2f("100k")

    try:
        opts, args = getopt.getopt(sys.argv[1:], "s:a:e:", ["size=", "audio-rate=", "extra"])
    except getopt.GetoptError, err:
        print str(err) # will print something like "option -a not recognized"
        usage()
        sys.exit(2)

    for opt, arg in opts:
        if opt in ("-s", "--size"):            
            target_size=h2f(arg)
        elif opt in ("-a", "--audio-rate"):
            audio_rate=h2f(arg)
        elif opt in ("-e", "--extra"):
            extra_rate=h2f(arg)

    for file in args:
        len = get_movie_length(file)
        print file, ":", len,"s"
        sum += len
          
                 
    print "total:", sum,"s"
    print "video bitrate:", calc_bitrate(seconds=sum,target_size=target_size,audio_br=audio_rate,extra=extra_rate), "kbps"
            

def h2f(size):
    d = {'k':1000, 'm':10**6,'g':10**9, 'K':8*1000, 'M':8*10**6, 'G':8*10**9}
    lc = size[-1]
    m = 1
    if lc in d.keys():
        m = d[lc]
    n = float(re.split(r"[" + ''.join(d.keys()) + "]", size)[0])
    return n*m
        

def get_movie_length(file):
    """returns the movie length in seconds"""

    len = 0
    p = subprocess.Popen("ffmpeg -i '" + file + "'", shell=True,
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    for line in p.stderr:
        if "Duration:" in line:
            h,m,s = (float(x) for x in re.findall("Duration: (\d+):(\d+):(\d+\.\d+)", line)[0])
            return h*3600+m*60+s
            break
    return 0


def calc_bitrate(seconds, target_size,audio_br,extra):
    """calculates the target bitrate in kbps for given target size and audio bitrate"""
    return (target_size/seconds - audio_br - extra)/1000


if __name__ == "__main__":
    main()

Python Script to create Python Scripts

This little python script is inspired from its ruby script equivalent found in the book Practical Ruby for System Administration, where the author created a script to create ruby scripts. The point of the script is to automate the tedious steps of creating a file, inserting the SHE-BANG line and making it executable. The original ruby version is more elegant, but I'm not that experienced in Python; so maybe there's room for improvements... suggestions welcome.

#!/usr/bin/env python

import sys,os,tkFileDialog

try:    path = sys.argv[1]
except: path = tkFileDialog.asksaveasfilename()

if not path:
    print "must specify a filename"
    exit(1)

f = open(path, "w")
f.write("#!/usr/bin/env python\n\n")
os.fchmod(f.fileno(), 0755)
f.close

os.system("/usr/bin/open " + path)

How to archive series on a Video DVD with your mac

Say you want to burn your series episodes you got lying on your hard disc to a DVD to enjoy them on your big TV screen or just archive them and free some space on your hard disc. For Mac OSX there are a lot of commercial (Roxio's Toast), and free programs (like iDVD)  out there, that will help you accomplish that task. But, most of them will not let you fit more than around 2 hours on a regular DVD+/-R for quality reasons. For series with 20 or more episodes, on the other hand, I'm willing to sacrifice some quality for space efficiency. The amount of video you can put on a DVD depends on the bitrate used for compressing the video. A lower bitrate comes at the price of lower video quality. But unless you have a high definition TV, compression settings for 7-10 hours still give you more than acceptable quality. In the following I will describe how you can achieve that with only open source quality on your mac. I'm gonna highlight some pitfalls I had on my MacBook Pro with OS X 10.6.2.

I make use of the following open source command line tools.

• ffmpeg: helps you convert your videos to DVD mpg
• dvdauthor: authors a DVD file structure from DVD mpgs
• dvdrw-tools: helps you burn DVD images to a physical DVD; builds on cdrtools
• cdrtools: should contain the `mkisofs` program that creates iso CD/DVD images. But for whatever reason, the mac port didn't include the mkisofs program! After several hours I gave up and took the mkisofs found in the Burn package.

Having above program infrastructure you are ready to create your DVD.

Converting Video

First, you will need to convert your videos to the right mpg format used on DVDs. On a Video DVD MPEG2 and MPEG1 are possible, but need to have certain resolutions and parameters to be standard compliant. ffmpeg to the rescue: the neat program has a dvd target which takes care of all the right parameters. In it's simplest form you could just let it do all the work for you with the default parameters:

ffmpeg -thread 8 -target dvd -i INPUT.AVI OUTPUT.mpg
 
This will take the INPUT.AVI file and convert it to MPEG2 format with AC3 sound. With more threads you can achieve higher FPS for recoding. The default parameters will only fit around 2 hours on a DVD. What you wanna do is tweak bitrate and resolution.

ffmpeg -thread 8 -target ntsc-dvd -aspect 4:3 -s 352x480 - b 1100k -acodec mp2 -ab 224000 -ac 2 -i INPUT.AVI  OUTPUT.mpg

This will give you around 7 hours per 4.7 GB DVD. Let's look at the parameters. First, ntsc-dvd should be used when you deal with ntsc content (29,7 fps). Aspect ratio is forced to 4:3 because the video is in 4:3 format. I use half resolution and a lower bitrate. Instead of AC3  stereo (2 audio channels) mp2 with 224kbs is used here. It is advisable to use lower resolution with lower bitrates. 1MB/s as in my setup is a pretty low bitrate compared to 4-5 MB/s found on commercial DVDs. But when combined with lower resolutions you can still achieve acceptable quality. According to this Wikipedia article here, the following resolutions are possible on a DVD.

At 25 frame/s, interlaced (used in regions with 50 Hz image scanning frequency):

720 × 576 pixels MPEG-2 (Called full D1)

704 × 576 pixels MPEG-2

352 × 576 pixels MPEG-2 (Called Half-D1, same as the China Video Disc standard)

• At 29.97, interlaced (used in regions with 60 Hz image scanning frequency):

720 × 480 pixels MPEG-2 (Called full D1)

704 × 480 pixels MPEG-2

352 × 480 pixels MPEG-2 (Called Half-D1, same as the China Video Disc standard)

The following formats are allowed for MPEG-1 video:

• 352 × 288 pixels MPEG-1 at 25 frame/s, progressive (Same as the VCD Standard)
• 352 × 240 pixels MPEG-1 at 29.97 frame/s, progressive (Same as the VCD Standard)

Keeping a high resolution with a low bitrate will give you a lot of encoding artifacts (blocks). So for very long content it's best to go with the lowest resolution. If you have hardcoded subtitles in the video (like in many asian dramas), it's better to keep a high vertical resolution. That's why I am going with the HALF-D1 resolution in above's example.

To batch encode a bunch of episodes use a for loop on the bash compatible shell:

for i Episode-{01,02,03,04,05,06,07}.avi; do ffmpeg -threads 8 -i $i -target ntsc-dvd -aspect 4:3 -acodec mp2 -ab 224000 -ac 2 -b 1000k -s 352x480 mpg/${i%.avi}.mpg;
done

This will encode the avi files Episode-01, Episode-02, ... Episode-07 and save them into a mpg subfolder. If you have all files converted you are ready for authoring a DVD.

Author DVD

For authoring a DVD, the `dvdauthor` program is used. Since DVD authoring is quite a complex task, an XML file can be generated with all the menus, titles and whistles. But if you all you wanna have is the movie on the DVD, you can just supply command line parameters like this

dvdauthor -o MY_DVD -t -f Episode-01.mpg -f Episode-02.mpg -f Episode-03.mpg -f Episode-04.mpg -f Episode-05.mpg -f Episode-06.mpg -f Episode-07.mpg
dvdauthor -o MY_DVD -T

to author a DVD in the folder MY_DVD with a single title (-t) and episode being accessible as a different chapter on the DVD. Putting each episode in it's own title would be possible, but  on most DVD remotes players switching titles is either not possible or cumbersome, whereas skipping chapters is always possible. The -T command will create the table of contents and complete the DVD authoring. The MY_DVD folder contains an AUDIO_TS and VIDEO_TS folder.

Burn the DVD

The last step is burning your authored DVD. A Video DVD is not a Data DVD. So simply using the mac os function `burn folder to DVD` is not what you want because it will create a data DVD. Instead make use of the dvdrw-tools.

growisofs -dvd-compat -dvd-video -Z /dev/rdisk1 ./MY_DVD

Will burn the contents from the MY_DVD as Video DVD. The DVD drive on my mac is /dev/rdisk1. If you go to disk utility you might see `/dev/disk1` instead, but you will not have permissions to use that drive on the command line. So use `/dev/rdisk`.

Watch your DVD

You should end up with a standard conform Video DVD that is playable on virtually all DVD players. Holding a lot of video content you can save some discs and money. Although the quality is not as good as on commercial DVDs, it is still very watchable on regular TV sets. Even more so for series, where you watch an episode usually only once.

Outlook

It should be no problem to create a small shell or ruby script to automate above steps. I might even write one and publish it here on my blog some time later. Especially automation of the bitrate calculation would be very helpful as I used a lot of trial and error in my first attempt to fit the desired number of episodes on one DVD.

Some useful Firefox plugins

Recently I have been playing with several browsers on my computer. There's the new Safari on my MacBook, Internet Explorer 8 on my virtual machine and finally the all new and lightning-fast Google Chrome browser. I had been a hardcore Firefox user before playing with those new browsers, but since all the browsers become more similar with every version (Tabbed browsing, smart address bar, etc.) I didn't miss my Firefox browser that much.

Now I have come back to Firefox, though. The main reason for that is the huge number of useful plugins (or extensions as Firefox calls them) out there. I figured it might be a good thing to share which plugins I am currently using,  and maybe get some suggestions for other awesome plug-ins out there.

First off are extensions that provide for a better browsing experience:

• Adblock Plus frees web sites from nagging advertisements
• Delicious Bookmarks best bookmark management plugin imho
• feedly puts a sleek newspaper design around your google reader and integrates more social  functionality.
• Quickdrag assigns functions and shortcuts to drag actions.
• Addto Search Bar lets you extend the search bar by a simply right clicking the search box on a search site.
• NoScript protects you by allowing only white listed sites to execute java script code
• CoolPreview provides quick previews for links so that you don't have to clutter your browser with more tabs. It's called Picture-in-picture Browsing.
• SkipScreen skips annoying screens on suspicious download sites like rapidshare.com

For social networking I have found useful:

• TwitterBar let's you twitter right from your address bar. 
• ScribeFire let's you write blogentries like for blogger.com

Now plugins for a better multimedia experience:

• DownloadHelper helps you download and convert videos from all the famous flash video sites.
• DownThemAll is a very useful add-on for downloading galleries from web sites or any other linked mass files - a huge time saver
• fire.fm integrates gives you last.fm radio and functionality for free right beneath your address bar.

And for development:

• Elasticfox gives you a GUI to your amazon EC2 services. very useful and time saving if you work with Amazon cloud services.
• Similarly S3Fox facilitates management of Amazon's Simple Storage Service (Not working with Firefox 3.6 yet, though) 

Those extensions make up my current selection on my Firefox installation. Although Chrome might be a little faster than Firefox, it is the amount of useful extensions and highly customizability that make  Firefox  my browser of choice once again.