While I see that creating quality ebooks this way is a long way ahead, a little piece of python script (gut2lrf.py) from my first attempt would certainly give eTexts a polish.

I started off checking the various existing tools to create BBeB (.lrf) files, and found makelrf3 a simple yet good enough candidate to do the bytecode compilation with. In order to build a version running on linux, I needed to put together a small Makefile, but otherwise it compiled without problems.

Project Gutenberg, with more than 20,000 titles and growing, is an excellent source of free eTexts. However, in order to avoid file-format traps, the project has a strict policy of using plain text files for its text books. What more, there are some conventions keeping us from easily reflow the text within: “Plain text eBooks should have line wraps at 72 characters and skip a line between paragraphs with no indentation.”

My little script – mentioned above – comes to aid here, as it preprocesses Project Gutenberg eTexts to remove unnecessary line breaks. It also fetches the Title and the Author of the book, and then calls makelrf3 to do the text to lrf conversion. Makelrf does a pretty good job splitting up the text to chapters, and quickly generates an lrf file.

The resulting eBook is in BBeB format, which means its file size is smaller than the original plain text document, and at the same time, it is laid out much better on the eBook reader, and also has some meta-info incorporated, which lets you find the book in the book list easier.

These compiled books are much better to read, yet they are still not perfect. They still lack the navigable Table of Contents, rich text formatting, such as stand-out Chapter headings, smaller spacings between paragraphs, words in italic or bold characters, illustrations, footnotes, as well as page headers and footers, so do expect some upgrades to my script down the line. Until that happens, I wish you happy reading on…

The original method was checking for existing users by using linear list lookup, which is known to be algorithmically slower than dictionary key lookup. Knowing that, the performance patch was a no-brainer, and, as expected, it eliminated the reported delay. I’ve sent this patch to ingeniweb (the author of GroupUserFolder) in an email on 22 May 2006, but haven’t heard back from them since. Yet, the patch has been accepted, even got refined a bit and was applied on the code trunk on 13 Jun 2006. From the one side, I’m happy to see it at least being fixed, while on the other side, I could have been more happy receiving a feedback on my report.

Nevertheless, I was still wondering how exactly the two lookup methods compare, so I’ve put together a python script to check how much we can actually gain on using haystack_dict.has_key(needle) instead of needle in haystack_list. I’ve compared them with small to larger sample sizes in small to larger sets (and lists) of samples.

I’ve used TOPCAT, an excellent, yet easy to use visualization tool for tabular data, to graphically represent the results.

As you can see, even at the small sets of small samples region, has_key performs much faster, and with the growing set and sample sizes, it keeps performing equally well, while list lookup goes wild in the same time.

Can’t say I was surprised or learned anything by doing this, but thought I’d mention as it might turn out to be useful for someone out there, so let me know if you’ve found this useful.

Phidgets are a painless way to extend a USB-equipped computer’s interactions with the physical world, providing a wide range of sensors, displays, and controllers as simple as plugging them in. With an acceptable pricetag, phidgets open up a world of new possibilities for software developers with only limited skills in electronics, ultimately resulting in some fun projects.

Phidgets Inc. provides software drivers and libraries for Windows, Windows CE, Mac OSX, and Linux, including driver source code (with a restrictive license, though). There are APIs for COM, Java, .Net, C, ActionScript, and bindings to a variety of languages to speed up development. Browsing the web and the forum for python support I’ve also came across the python-phidgets and the phydget projects that provide python bindings.

I decided to give phidgets a try, so I’ve purchased a 3-axis phidget accelerometer, a phidget interface kit with text-LCD, and a few analog sensors to measure temperature, magnetic fields, and light, and also a touch sensor. The sensors are going to be used in a home-automation project, while the accelerometer will somehow become an input device (thinking of something like the accelerometer mouse).

With the limited time I have spent with my phidgets so far, I’ve added accelerometer support to phydget (should you be interested, here’s the source code for my accelerometer-enabled version), and created a minimalistic visualizer for the accelerometer readings using the excellent vpython library. More to come as soon as I could make some progress in either of my planned phidgets projects…

It took quite a time to realize that all the affected numbers are always small. Then I wrote unit tests to determine the range in which the numbers fail to keep their signs. That turned out to be the range of (-1.0 .. 0.0). That was the point when I was beginning to suspect that there’s something fishy in the underlying maths… so I descended to the realm of Plone and Archetypes code to narrow down the problematic part. It wasn’t too long before I found that the sign gets lost in the following piece of code from Archetypes/Fields.py:

value = value.split('.')
__traceback_info__ = (self, value)
if len(value) < 2:
    value = (int(value[0]), 0)
else:
    fra = value[1][:self.precision]
fra += '0' * (self.precision - len(fra))
#handle leading comma e.g. .36
if value[0]=='':
    value[0]='0'
    value = (int(value[0]), int(fra))
return value

Arrgh. Too bad. I wish I had never seen this code. Archetypes has reinvented fixed point storage and handling in a very – hmm – “clever” way. For those the problem still isn’t obvious, let’s see how, say, “-0.5″ gets handled: as a string it gets split into whole and fraction at the decimal separator, and only then it gets converted into integer. Too bad there’s no difference between -0 and 0, isn’t it? So “-0.5″ becomes (”-0″,”5″), and finally ending up as (0,5), which equals to “0.5″.

The fix, from here, is obvious, though it raises some concerns. First I thought the sign need to be stored separately in the value tuple, however, that would mean tuples from there on will consist of 3 items, so the code becomes incompatible with already existing FixedPointField values in the world, which is a big no-no. This leaves us with the sole chance of applying sign to the fraction when in range (-1..0). Let’s see how much of the code need to be changed for that:

#handle leading comma e.g. .36
if value[0]=='' or value[0]=='-':
    value[0]+='0'
value = (int(value[0]), int(fra)*([1,-1][value[0]=='-0'])

Since this stores “-0.5″ as (0,-5), we also need to change the get() method:

def get(self, instance, **kwargs):
    template = '%%d.%%0%dd' % self.precision
    value = ObjectField.get(self, instance, **kwargs)
    __traceback_info__ = (template, value)
    if value is None: return self.getDefault(instance)
    if isinstance(value, basestring): value = self._to_tuple(instance, value)
    return template % value

To something like this:

def get(self, instance, **kwargs):
     template = '%%s%%d.%%0%dd' % self.precision
     value = ObjectField.get(self, instance, **kwargs)
     __traceback_info__ = (template, value)
    if value is None: return self.getDefault(instance)
    if isinstance(value, basestring):
        value = self._to_tuple(instance, value)
    return template % (['','-'][value[1]<0],value[0],abs(value[1]))

And we’re done. Nevertheless, code quality remains questionable. (how do you change precision on the fly, for example. And I dare not mention performance issues of all those unnecessary string to integer and vice versa conversions.)

Unfortunately, submitting this bug report and patch isn’t so obvious as the fix. Archetypes trac only accepts new issues from registered members, but doesn’t provide a link to register. The Plone support page mentions the Plone issue tracker, which does link to a page where you can register, but registration for some reason failed in my case, because either the e-mail or the user name was considered wrong, even after several different attempts. And finally, nobody really cared on the Plone support mailing list either. Sorry guys, you lost my vote there and then. Recent svn still has the bug, and it is unlikely to be fixed without being reported.

The lessons to be learned here are:

1. for me not to take any “complete” frameworks granted as bug-free, even in the basics.
2. for developers in general, that no matter how clever you’re trying to reinvent the wheel, it is almost certain that there will be something overlooked, potentially causing nasty corner case bugs, so unless there is a rock solid reason behind reworking basic methods from scratch, it does not worth the potential havoc it surely going to wreak.
3. for the Plone project to set up at least one obvious, and working channel to submit bug reports and patches.

UPDATE 05 Sep 2008: the above fix has finally been integrated into plone codebase, changelog here. Thanks, Maurits! ]]> http://www.forwhatitworths.com/posts/2007/07/the-fixedpointfield-bug/feed/ 2