Something had to be done. However, I’m still not totally convinced by the accuracy of spam filters, and by now we know that CAPTCHA is not a silver bullet either. This means no matter which route I take, I’ll still have to keep on moderating. With reCAPTCHA, finally, the spam-fighting became useful at last.

I’ve heard about reCAPTCHA a while ago, and really liked the idea of using computer-illegible text scans of old public archives instead of algorithmically generated images for telling humans and bots apart, however, only this recent article on arstechnica convinced me to give a try.

I’ve quickly and easily set up this wordpress plugin, and registered for the necessary API keys at recaptcha.org. Now any new comment needs some text-recognition work too, but at least it is for some greater good, as any recognized word will contribute to the digitalization effort of old public archives.

Let the comments come, the spammers fade, and the electronic libraries fill up with quality archives at the same time!

Apart from telling the idea, the site has some very elaborate demos to help understanding the concept too.

So, tell me, how long could you resist clicking the button? I survived for several minutes, and can’t wait to see this adopted on my desktop.

While I see that creating quality ebooks this way is a long way ahead, a little piece of python script (gut2lrf.py) from my first attempt would certainly give eTexts a polish.

I started off checking the various existing tools to create BBeB (.lrf) files, and found makelrf3 a simple yet good enough candidate to do the bytecode compilation with. In order to build a version running on linux, I needed to put together a small Makefile, but otherwise it compiled without problems.

Project Gutenberg, with more than 20,000 titles and growing, is an excellent source of free eTexts. However, in order to avoid file-format traps, the project has a strict policy of using plain text files for its text books. What more, there are some conventions keeping us from easily reflow the text within: “Plain text eBooks should have line wraps at 72 characters and skip a line between paragraphs with no indentation.”

My little script – mentioned above – comes to aid here, as it preprocesses Project Gutenberg eTexts to remove unnecessary line breaks. It also fetches the Title and the Author of the book, and then calls makelrf3 to do the text to lrf conversion. Makelrf does a pretty good job splitting up the text to chapters, and quickly generates an lrf file.

The resulting eBook is in BBeB format, which means its file size is smaller than the original plain text document, and at the same time, it is laid out much better on the eBook reader, and also has some meta-info incorporated, which lets you find the book in the book list easier.

These compiled books are much better to read, yet they are still not perfect. They still lack the navigable Table of Contents, rich text formatting, such as stand-out Chapter headings, smaller spacings between paragraphs, words in italic or bold characters, illustrations, footnotes, as well as page headers and footers, so do expect some upgrades to my script down the line. Until that happens, I wish you happy reading on…

With the Sony PRS-505 ebook reader hitting the market I instantly knew I need one. Now that I hold it in my hands, it’s love at first sight. I’d say with chewing on this nicety, my hunger for useful gadgets is cured for a while.

Apart from the sleek design, the most important things securing my choice were the courtesy of Sony to include an SD card slot, and the USB Mass Storage interface, as these two features ensured that the reader can be extended for less (not that the 200MB internal flash can’t hold enough books for many commutes), and will communicate seamlessly with a Linux PC.

This eInk display technology is a salvation for the eyes of many, including myself. An anti-glare, daylight readable, not background-lit, high contrast, 8 grey levels display makes my LCD-strained eyes very happy.

As I was planning to use this device to read not just books but also to keep reference manuals and tutorials at hand, I was curious about the ebook formats it supports. So the first thing I did was playing around with various document formats. The rest of this post is dedicated to this topic.

Text files are a developer’s friend. Fortunately, these are laid out quite fine by the reader. The TXT files appear in the booklist with name of the file as the book title, and the file creation date as the book author. The reader provides 3 zoom levels, with 30, 25, and 20 lines of text per page displayed in portrait mode, or 15 + 2, 12 + 2, and 10 + 1 overlapping lines per (half) page in landscape mode. The font used to render txt documents appears to be Bitstream’s Dutch 801 Roman BT. It seems ISO-8859-1 is assumed being the character encoding of text files.

When an ebook gets opened the first time, the reader works for a couple of seconds to paginate the contents, however, the results get cached, so this only slows things down once per ebook (per zoom level used).

RTF documents add the features of multiple font faces and font decoration to be used. Also, it is the document title and author that gets displayed in the booklist, so these need to be set up properly for easier lookup.

The next widespread format supported is PDF, though it has some issues. The reader’s screen size is too small to display an A4 or a Letter size PDF in a readable way. You may use the landscape function, which shows the top or the bottom half of the page. This, in together with the zoom function results in a readable half-page (without the margins), but the zoom level resets to default when turning page. Furthermore, special fonts/charsets don’t always render properly, and password protected PDFs don’t even show up in the booklist.

On the positive side, internal links in PDFs can be used to navigate within the document. Ah, and I’ve found quite a few reader-optimized ebook titles in PDF format at Feedbooks.com.

Documents in Sony’s proprietary ebook format, BBeB, obviously work the most seamless, providing three zoom levels, where the number of lines displayed depends on the font size settings of the ebook too. However, it’s hard to find anything useful in this format outside the CONNECT eBooks universe. I’m planning to write about tools for creating BBeB documents in a later post.

Concerning pictures (jpeg, png and gif formats), the PRS-505 is littlesomewhat slow on rendering, and with the very limited colorspace of 8 gray levels, the PRS505 is not likely to be used as my primary electronic photo album. However, it is good enough to enjoy my favorite comics, what more, probably an ideal device to share the greatest cartoons of savage chickens with my friends in the offline universe.

Finally, regarding the MP3 playing capabilities, while it is certainly a gimme feature to allow listening and reading at the same time, I don’t yet consider it a big thing, but time will tell whether I’ll use this ebook as a walkman too. For now, I haven’t even tried this feature.All in all, it is a charm to hold and read on. It is definitely worth all the 300 bucks of its introductory price tag.

The original method was checking for existing users by using linear list lookup, which is known to be algorithmically slower than dictionary key lookup. Knowing that, the performance patch was a no-brainer, and, as expected, it eliminated the reported delay. I’ve sent this patch to ingeniweb (the author of GroupUserFolder) in an email on 22 May 2006, but haven’t heard back from them since. Yet, the patch has been accepted, even got refined a bit and was applied on the code trunk on 13 Jun 2006. From the one side, I’m happy to see it at least being fixed, while on the other side, I could have been more happy receiving a feedback on my report.

Nevertheless, I was still wondering how exactly the two lookup methods compare, so I’ve put together a python script to check how much we can actually gain on using haystack_dict.has_key(needle) instead of needle in haystack_list. I’ve compared them with small to larger sample sizes in small to larger sets (and lists) of samples.

I’ve used TOPCAT, an excellent, yet easy to use visualization tool for tabular data, to graphically represent the results.

As you can see, even at the small sets of small samples region, has_key performs much faster, and with the growing set and sample sizes, it keeps performing equally well, while list lookup goes wild in the same time.

Can’t say I was surprised or learned anything by doing this, but thought I’d mention as it might turn out to be useful for someone out there, so let me know if you’ve found this useful.

Although my blog logo is still just scraped, it’s time to write about the making of the font used there. I felt spending a couple of hours creating the font makes things learned worth sharing.

I named this font ArcNBrick mostly because of the basic concept behind its glyph shapes, where everything is built up of – surprise – arcs and bricks.

The tool I’ve chosen to build the font was FontForge, a cross-platform, open source application designed for the very purpose of creating fonts of different types.

Let’s take a quick tour of how I’ve started building a font with FontForge. First of all, the concept behind my font allowed me to work without any pre-drawn, scanned helper images, even though FontForge supports those. Let’s discuss the basics of drawing glyphs, then.

• All paths are built up of curve, tangent, and corner points. Curve points are marked with a circle, tangent points with a triangle, and corner points with a rectangle.
• Control points (marked with x) help you to shape the curves as desired.
• Outer boundary paths should be drawn clockwise, while inner boundaries counter-clockwise

After learning these very few basic rules from the tutorial, I was very soon up to speed in drawing the glyphs for all the letters. In order to accelerate the process and also preserving design consistency, I used cheap transformation tricks to create glyphs from one another. For example the letter b is a perfect candidate to create letters d, p, and q.

So, drawing letter d consisted of the following operations: select all on letter b, copy, paste to letter d, then element: transformations: transform: flip horizontally, and finally element: correct direction. Drawing letter p wasn’t much harder: select all on letter b, copy, paste to letter p, then at element: transformations: transform: first flip vertically then move y by -200, and finally element: correct direction. That was it. Very soon, all letters took shape, so I’ve built the TrueType version to give it a go.

As you can see, the result is nothing useful for everyday use, yet, you might find it appealing for short, emphasized labels used in logos, for example. Should this be the case, feel free to download and use my ArcNBrick.ttf. Make sure to leave a comment here on where and how this font was particularly useful for you, I’d be happy to receive feedback!

I need to confess that I haven’t tackled any of the issues that would make a font professional, such as hinting, accents, proper encodings, to name a few, but by this time you’ve already guessed that I am by no means a typographer. Yet it was fun building a unique font from scratch and seeing it working in a word processor. Some day I might even try building some wingdings fonts too, who knows…

I’ve recently came across an issue with the ordering of the NULL elements. For some reason, PostgreSQL thinks NULL is the highest value. Well, it can be interpreted that way too, but with the following business case it wasn’t so.

The structure (simplified for easier digestion) being queried contains two tables: one for clients, and another for purchases made by clients. Obviously, every purchase is linked to a client, yet not every client has made a purchase.

I needed to create a report that displays every customer, with the total amount of purchases made by them, showing the client with the highest total amount first.

I’ve started off with the following query:
SELECT client.id, SUM(purchase.amount) AS total FROM client LEFT JOIN purchase ON purchase.client_id = client.id GROUP BY client.id ORDER BY total DESC;

Running the query revealed the problem: the aggregate function SUM returned NULL for the clients with no purchase, and NULL is treated as the highest value by Postgres.

This was just about time for looking this issue up. Well, according to the relevant wikipedia entry:

“The SQL standard does not explicitly define a default sort order for Nulls. Instead, on conforming systems, Nulls can be sorted before or after all data values by using the NULLS FIRST or NULLS LAST clauses of the ORDER BY list, respectively. Not all DBMS vendors implement this functionality, however. Vendors who do not implement this functionality may specify different treatments for Null sorting in the DBMS.”

Yes, that NULLS LAST sounded like a promising solution, so I’ve checked which DBMS implement it. A quick search revealed this forum entry that says, by default Postgres, DB2, and Oracle considers NULLs as higher than non-NULLs, while MSSQL and MySQL considers them being lower. DB2 V7 and Oracle 9i supports NULLS FIRST/LAST, while MSSQL, MySQL and PostgreSQL doesn’t.

However, that entry was two and a half years old, while PostgreSQL is known to being actively developed, so I’ve checked its status concerning NULLS LAST again. Its latest documentation does in fact tells NULLS LAST is supported, so I’ve checked when it became integrated. There it was, a patch from last December does in fact take care of the NULL ordering problem. However, it is only included in version 8.3, and an upgrade wasn’t an option in my case.

That meant another route to be taken. After a little more reading, I’ve found my alternative candidate, the COALESCE function, which is implemented in older versions of PostgreSQL too. COALESCE returns the first non-NULL value from its list of parameters, which is exactly what I needed here. My query therefore was altered to this:

SELECT client.id, COALESCE(SUM(purchase.amount),0) AS total FROM client LEFT JOIN purchase ON purchase.client_id = client.id GROUP BY client.id ORDER BY total DESC;

And suddenly, the problem ceased to exist, and every one was happy by looking at the numbers of the biggest customers at last. After all, from the business perspective, they represent the highest value, not NULL.

First of all, the most important rule is: don’t try to fix if it ain’t broken!! A working Toughbook is a charm, but the same machine as a paperweight is useless. Taking it apart is a long and tricky process, putting it together needs just as much care and attention.

All right, so yours is broken, and isn’t so lucky as “Tsiolkovsky” to have a Panasonic Laptop repair shop around, and you think have the skills, patience, and guts to disassemble it, and YOU UNDERSTOOD THAT THIS IS NOT AN OFFICIAL GUIDE, AND I TAKE ABSOLUTELY NO RESPONSIBILITY FOR ANY POSSIBLE DAMAGE OR LOSS EVEN REMOTELY CONNECTED TO THIS POST. The fact that I’m managed to repair and upgrade my CF-W2 this way does not guarantee that you’ll be as lucky as me.

Ok, enough said, let’s see the process…

Make sure you have the proper screwdrivers, some chewing gum, or a magnetic head screwdriver, and some thermal compound at hand, also prepare the new parts to be built in. Allow yourself at least an hour’s worth of time dedicated to the process.

1. Back up all your precious data to minimise damage if anything goes wrong.

2. Clear some flat workspace, and have something to hold the screws in.

3. Before shutting down, open the CD/DVD drive cover (there’s a screw beneath to tackle later).

4. Turn off the computer, remove the battery and unplug the power cord.

5. Take a big breath.

6. Remove the plastic cover on the WiFi antenna on the right side of the notebook. It is secured with two plastic tabs. You may need to strain it a bit.

7. Unwind the screw in the CD/DVD drive

8. Unwind the two screws at the wifi antenna on the right side of the laptop.

9. Unscrew the two bolts holding the VGA out connector.

10. Turn the notebook upside down.

11. Unwind the memory extension’s screw, remove the plastic cover.

12. Unwind the three long screws that hold the keyboard, remove the plastic cover.

13. Flip over the notebook.

14. Dislodge the keyboard. On the top, two plastic tabs keep it in place: one to the left of F1 and one to the right of F12, and a couple more plastic tabs around the spacebar. The keyboard is also glued with some sticky tape on three places, so a little force is needed to dislodge it from the casing. Its cable is also needed to be unplugged from the main board. Take extra care not to harm the ribbon cable during the process.

15. Unwind the 6 black screws underneath the keyboard.

16. unwind the 2 hidden screws: these are very tricky. you may consider using a magnetic head screwdriver, or a bit of chewing gum to prevent the escape of these screws into the casing.

17. Close the screen, then turn the notebook upside down. By closing the screen panel you actually secure the casing together while the bottom cover is removed.

18. Unwind all 8 screws on the bottom of the notebook. Some screws may stuck at first disassemble, be sure to use the right tool, otherwise the head will suffer, and you might be unable to take the notebook apart without considerable damage. There are four short, and four longer screws, mind their positions. (Update: Brett told me that if you mix the short and long screws, you might end up punching a hole in the motherboard during the reassemble, so better check it twice!)

19. Unwind the 2 screws securing the screen panel on both sides.

20. Carefully remove the bottom cover.

Now you can access both the HDD and the Wlan card in the miniPCI slot. Apart from the memory extension and the keyboard, these are the two modularly replaceable parts in the Toughbook W2.

21. The Wlan card has two antenna plugs (a main and an auxiliary). Simply, carefully pull them, then you can flip the module from the bay, and insert the upgrade module instead, and finally don’t forget to attach the antenna plugs.

Upgrading to an ipw2200bg or an ipw2915abg doesn’t raise any hardware compatibility issues, only a driver upgrade is needed then. You can find many of these cards on the second-hand market, usually originating from broken notebooks. I got a 2200bg for around $27 on an auction site.

22. Concerning the HDD: you need to know that most Panasonic Toughbooks use a low voltage (3.3V) HDD, which you can’t readily buy in retail shops, so you need to mod a 5V notebook HDD by clipping pins 41 and 44. (At least this worked in a few reported cases, including my Samsung SpinPoint HM080HC 80GB 2,5″ HDD).
I’ve found this HDD upgrade guide useful.

Ok, now that the parts have been replaced, you can put it back together.

23. Secure the back cover with the 2×2 screws fixing the screen panel, and the 4 long and 4 short screws fixing the cover to the casing (remember, DO NOT MIX THEM UP, as the long screw in a short’s place might damage the motherboard), then turn the notebook around and open the screen.

24. Continue with the two hidden screws: again, use a magnetic head screwdriver, or a bit of chewing gum to secure the screws while twisting. Should they fall, you’ll need to open the bottom panel again to get to them.

25. Next, take care of the the 6 black screws.

26. Now comes an important step: clean the surface of the processor (and the metallic part on the bottom of the keyboard), then apply some thermal compound on it. This is particularly important, as otherwise the CPU can get pretty hot, up to an unhealthy 90 degrees centigrade with processor intensive tasks running.

27. Plug the keyboard back, flip the plastic tabs in place. Again, handle the ribbon cable with extra care.

28. Flip the notebook over, secure the keyboard and the plastic cover with the 3 long screws, then the memory extension cover.

29. Flip once more, screw in the 2+1 screws, and put the WiFi antenna cover back in place.

30. Finally, screw the VGA connector’s bolts back to their respective place.

Now put the battery back in place, and hold your breath while starting up the notebook. If it works, well, congratulations, you’re done. In every other case, I’m sorry to say, but now it is your turn to find out what have gone wrong.

Should you be succeed or fail, I’d like to read about it, so please do post a comment here. Thanks!

Sure, it is useful to know where you are, and in what general direction your destination is, what more, how on Earth can you get there on four wheels. All this requires a GPS receiver (to read your current location), a mapping software (to calculate your moving direction from subsequent reads), and an accurate map (to give some context and meaning to your location readouts). The first two are more or less out there. Still, maps can be considered anything but reliable or accurate.

With news on blind faith put in navigation systems, people have already got wet, others found themselves down the stream, on the track, or fighting the dunes, the map inaccuracy issue finally got into the headlines.
People, of course, are eager to come up with various solutions.
* An almost-victim said she won’t ever use a navigation system again.
* others said people should update the maps in their navigation systems every once in a while
* some Welsh officers has even created a new, but ambiguous traffic sign to warn lorry drivers of the road being narrower than expected.
* while others came up with fairly good ideas, such as superimposing the map on a front-camera’s stream, so that even faithful users can compare the map with the reality on the same screen.

However, keeping maps up-to-date is not as simple as one may think. Downloading the latest updates doesn’t do the trick. Maps become inaccurate at (or even before) the very moment they are published. As it is their purpose to sketch the spatial layout of real-world features, the only way they can remain up to date is if they constantly evolve along with their “mapped” real-life features. And constantly means very short term here. Think of traffic, for example: maps that fails to mention actual road congestion, current roadworks or accidents, has only a limited use, routing their users through nasty traffic jams.

In order to better understand the situation, first let’s take a look at the various approaches of creating and updating maps:

The commercial route is what proprietary navigation software vendors take. Usually you get these applications bundled with a GPS receiver, and preloaded with some not-too-up-to-date maps. The map data used by them is something commercially available, which most likely originates from the digitized version of maps of the “pen and pencil” era, with updates from geocoded aerial photography, and most likely not thoroughly validated by actual GPS reading on the ground. User feedbacks are usually welcome, and the major inaccuracies gradually get updated , but that’s a slow process, and the more recent map you need the more deep you need to dig into your wallet for it.

On the other end, there are the open approaches, thinking of open source projects with an enthusiastic community of hobbyists and professionals dedicating their free time to make something useful or at least interesting.

These projects take different approaches to acquire map data.
Some of them make use of openly available data (where there is such a thing), for example, Roadmap uses the TIGER GIS data, where of course, data accuracy is limited again.

Others, such as gpsdrive use dumbed down version of “available” data in form of snapshots of rendered maps from various web map services, such as google maps or yahoo maps to mention the two biggest. This approach can’t do routing, though, as the map is treated as a simple raster image only, without any routing information or so.

Yet another approach is based on a mapping community, and let contributors draw the maps, like cgpsmapper’s MapCenter, or the geocaching communities offsprings (e.g. turistautak for Hungary), or the very active openstreetmap project, or the much less active FREEMAP project to mention a few. The resulting maps are definitely GPS-validated, as the data source itself is the GPS tracks gathered, but it takes enormous amount of total community time to create a quite complete, routable map for even a relatively small region, nonetheless, quite impressive results already exist.

Still another concept is algorithmic map generation from GPS tracks, using statistical methods, thus reducing the amount of expertise needed on the map creation side. Mapgeneration takes this algorithmic map-drawing route. The problem (again) is gathering enough usable tracks to work on.

Yet, these all are just base maps needing constant updates, having no information on current traffic data. For this latter, other, more sophisticated techniques are needed. Let’s take a look at some of these approaches:

Starting with the statistical approach, when previously collected and analyzed traffic patterns are attached to each road. With this incorporated traffic flow data, the routing algorithm can do better time estimates for the possible routes. Even though this approach gives a better route, it still didn’t solve the problem of keeping its data up-to-date.

Being up-to-date necessitates an interface to refresh data. The online service approach solves this by having a central hub that constantly collects traffic data from various sources (traffic surveillance and control systems, or sampling a subset of cars also known as Floating Car Data), and sends updates to subscribers through some wireless medium, typically over a cellphone data network. Such systems can provide information on traffic events minutes or even hours earlier than radio news, saving precious time for their users.

However, not all types of traffic data needs a central database. The decentralized version of Floating Car Data uses traffic information collected from neighbouring cars in an ad-hoc network fashion. In this scenario, cars broadcast their position, direction, velocity and other traffic information, as well as relay data from cars ahead, and filter data broadcasted by other cars for relevance. The strength of this approach lays in its adaptability to spontaneous short term situations, e.g. assisting drivers to cooperate for driving safety. In the same time it may also provide enough data for making longer term decisions, such as choosing a better route.

These advanced systems are still mostly lay on the drawing boards, and are still too expensive to become widespread. However, with the recent trend of car manufacturers (e.g. Toyota, GM, and others) equipping all new cars with GPS receivers, the location and velocity data will be available from more and more vehicles on the move. At the same time, ad-hoc wireless networks also getting more of a commodity, it seems finally things are getting in place for distributed, always up-to-date, cheap and reliable navigational systems to appear in the near future.

Better be soon, as driving safety is also more and more of an issue as people rush towards their goals with an ever hastening pace, and they need to be guided, not further confused by navigation systems.