Asciideo

2010-06-13

Converting a regular movie to ASCII using just the browser and "HTML5" features like Canvas and Video. Talking about a project that got out of hand. A few months ago I saw some ASCII movie on youtube. It was a regular youtube movie but showing ASCII characters. Mind you, this was way before Youtube pulled the April fools ASCII gag. (And curse you for doing so Youtube, when I'm about to release an ASCII converter in HTML5!) I guess I should be happy that they didn't go the HTML5 way (it was flash only, you couldn't select the text)... Interestingly enough, the whole feature has been removed now (I mean, it was nice). Anyways, when I watched that other video I finally got an idea for a cool demo project to involve HTML5 video stuff; Make an ASCII converting web app...

So first I searched for an ASCII converting script. Was quickly to find Jacob Seidelin's script to do so. It turned out that script had a good idea (although I'm not sure there are many other ways of doing it), but it was relatively slow and unable to continuously accept input. So I rewrote it, heavily optimized it and turned it into a web worker ... Later I removed the webworkers because they proved to be very slow in this context.

Next I experimented a little with getting a fixed width/height for the ASCII output (css). If I knew the size of a letter of output, I wouldn't have to worry about how the output was painted, I could just use white-space: pre and get a single string with letters and returns. No HTML involved! Since I didn't want to get tiny letters (that felt a little like cheating, using pixel sized letters) I ended up with this simple css class:

Code: (CSS)
#out {
padding: 0;
margin: 0;
border: 0;
line-height: 1px;
font-size: 10px;
overflow: hidden;
font-family: monospace;
line-height: 8px;
white-space: pre;
}

This results in letters of 6x8 pixels. I converted the script to make use of this information. So now it first scans the brightness of _all_ the pixels (doesn't skip any, unless you tell it to) and puts them into buckets. One bucket per letter (so 8x6 pixel squares). The second loop averages the bucket which results in the brightness of the letter. Then the script multiplies this normalized value by the number of letters it can choose from and floors the result. That's the letter that's picked and gives us the impression of brightness. These are obviously ordered by brightness.

So after jumping a few more hurdles concerning painting the video to canvas and not being able to immediately read from the canvas, memory limits and other edge case beta issues I had a simple proof of concept for the ASCII converter. This first concept used video, canvas and web workers. I started out with web workers because my initial resolution was too big to simply be processed. I figured web workers would be ideal for this case. In the end it turned out that using them added a large chunk of overhead, which was about 400% compared to doing it without web workers on a small image.

On my machine (quad core Q2200), firefox converts a 320x240 image to a 40x30 (in letters) ASCII image in about 31ms. Chrome (a nightly build) does it in 15ms. Opera (10.5) does it in 10. Using web workers adds about 120ms on firefox and 180ms on chrome. Opera doesn't support them yet. However, when the output size doubles the processing speed jumps to about 500~1000ms per frame in all. So that makes for an interesting benchmark... The web workers aren't completely useless. When processing a larger output size, the video will skip terribly without web workers. This problem is mitigated when running web worker mode, however it clearly shows that firefox is taking up about 4x as much memory per frame, especially when the buffer is small, compared to Chrome. This also showed a clear memory leaking problem for Firefox (but it's an edge case, since, well, when the hell would this happen for a valid reason...).

Anyways, the main idea behind the worker is still the script by Jacob. I've optimized the hell out of it though. It now uses two single loops, NO functions (not even for push, pop and floor) and caches all computable values. Looking at the worker script you see a lot of variables in a closure which are cached inside. This little step optimized the function by 300% to 400%, talking about gains... I've also tested around with some dynamic programming and more caching, but those solutions turned out to be slower due to lookup costs. Note that ~~ is the same as Math.round. Might as well be |0 or >>0. In some browsers even Math.floor is optimized to be just as fast.

Next there were some issues with video. Working with video makes it obvious how much work still needs to be done in this regard. Chrome (nightly/beta) crashes all the frigin time with my demo. Safari won't play ogg, only mp4. Firefox has some minor issues and glitches. After painting video to canvas you cannot get the image data immediately in firefox (throws an error) and there's no way to detect that it is ready. Opera is doing its own wicked thing and seems to refuse to load other videos, or something. Dunno what's up there. And of course, IE can't even be considered here. Note that this demo was created before webM, so it doesn't explicitly support it.

The whole encoding thing got me into a fit anyways. It's impossible to get a decent online H.264 encoding service like tinyogg does for Theora. My (now old) host had problems with streaming, after fixing initial stream configuration settings first, so I moved to Dreamhost (yes, spam ;)). Since they also allowed unlimited data storage and bandwidth, I could add an upload feature and actually make it an interactive demo. Coupled with tinyogg's youtube service, I figured somebody might actually use it too :)

In the end, I had to fix some uploading issues with my host to get uploading a 2mb ASCII movie to work... but luckily that worked out fine too. Finally some minor modifications, adding some css3 for general spiffiness, twitter sharing feature and there we go... my Asciideo HTML5 App demo is ready :)

Note that another converter which uses the same foundation (but not the optmizations) can be found here (thanks to @pesla). Although it was in no way my inspiration, it did push me to try and run the script without web workers ("How the hell can it use the slow version of the script and still beat my script so hard?" ... "oh... web workers _are_ slow"). (By now, that demo has hit other javascript blogs too.)

Possible improvements lie in resolution and colors. Although I've tried adding colors to the letters and that flatout failed. Adding colors to the background worked better, but kind of defies the point. Also, rounding colors wasn't as successfull as I had hoped. Making the letters smaller obviously makes for better (but bigger) asciideos. Making the letters thicker also adds a little to the quality. I might try to take specific letter constructions into account and try to match a group of pixels more accurately to a specific letter, rather than by brightness in general.

So I hope you enjoy it. Start creating your own videos and share them with a link like http://asciideo.qfox.nl/9 ;)