Monday, November 28, 2005

Stupid Nevow Tricks

Actually, this stupid Nevow trick is pretty damn clever.

So, you want to write a db-backed app with a web frontend. You plan to have, let's say, 60k users registered and you want them to be able to send one another a message via a webform by entering first and last real name as the recipient.

This could be a real pain. In the old days you'd need a separate form just to look up the user in the database first, then you can start the message sending bit. Not any more.

Check out this demo. This uses Divmod's Axiom as the database backend, and Nevow's new LivePage API called "athena" to drive the frontend. I'm very impressed with the power and performance of the Axiom database so far, and it holds up beautifully in this example. LivePage works wonderfully with it for a very compelling combination.

What does it do?

The demo database contains a single table with 10k names in it. (Locally I experimented with a 60k name database before trimming the fat for the checkin; performance did not suffer one iota.) The names are randomly generated from US Census data. When you start the demo and point your web browser at it (see the README.txt) you'll see a field. Type a name into this field; if part of what you typed has a database hit, you'll see the possible completions:

.. until you get a unique match, at which point it fills in your text field for you:

You can, of course, also click a name in the menu.

What's neat?

A lot of the interesting part is in JavaScript, so make sure you look at typeahead.html. In particular, selectRange handles selecting the untyped portion of the text field, and complete handles filling in the field and the options in the select element.


The Nevow stuff is pretty straightforward. You create a LivePage instance as your page. Then you create elements capable of talking to the server as LiveFragments, and stuff them into the page. Here's the whole LiveFragment for the input field. This renders the input box and has a single method to handle a callRemote from JavaScript.
class TypeAheadFieldFragment(athena.LiveFragment):
docFactory = loaders.stan(T.input(type="text", id="typehere", **athena.liveFragmentID))
allowedMethods = { 'loadCompletion' : True }

def loadCompletion(self, typed):
assert type(typed) is unicode
if typed == u'':
return None
q =, u'%')
matches = theStore.count(Person, q)
if matches <= 10:
return [ for p in theStore.query(Person, q)]
return None

The matching JavaScript is nice and simple:
var node = $('typehere');
var typed = node.value;
var d = Nevow.Athena.refByDOM(node).callRemote('loadCompletion', typed);

(Again, be sure to read the rest of the JavaScript on the page.)


Axiom is damn bloody simple to use. The demo doesn't really show off much in the way of Axiom features, but it does show off how simple your ORM code can be. A couple of things stood out for me.

One was that Axiom's latency is very very low indeed. At the place where I do the query I treat it as non-blocking, and for me on my one laptop it had better be. In fact, if this demo works at all it has to be non-blocking; a person typing would quickly get ahead of the database otherwise. The speed of Axiom at returning a list of 10 names from a list of 10k names is impressively subsecond. Now, I'll be the first to acknowledge that what works nice and fast for me on my laptop might not be so fast when you've got 10k live users banging away on their keyboards, but it establishes that this low latency is at least possible to achieve. I'll leave the macho scaling stuff to others.

The other was that Axiom's batch performance is pretty darn good. I left in the batchInsert function in The function I used to to import the original 60k names took about 7 seconds to run using Store's transact method. Be sure to use .transact() if you're going to be manipulating a bunch of rows at once though; before someone (I think JP?) pointed out that Axiom had transactions, I was trying to do the 60k inserts by creating 60k Items at one per second. It's worth highlighting that the transact method is so simple to use that as soon as I knew about it I was using it with no explanation whatsoever. Simplicity is beauty.


Another surprise winner here was MochiKit. I've heard so many good things about it, but until now only used it indirectly through LivePage. Even if I didn't already know who wrote it, it would have been obvious that this JS library was written by an artful Python hacker. I made use of MochiKit.DOM for constructing the items in the select widget, but I was particularly impressed with MochiKit's "bookmarklet debugging". Getting complex scripts to work in JavaScript is nothing short of horrifying for me, but having the log messaging and viewing facilities in such a nice format cut in half the time it would have taken me to get the JS right.

Thursday, September 22, 2005


I thought I'd give a shout-out here to pydispatcher. It was written by a gent named Pat O'Brien, whom I've met and had lunch with, and it's coming in very handy for Vellum. I've refactored Vellum to use it, and it's remarkably simple and robust. Where it comes in handy is building code where lots of things need to know when program state changes, and need to know where those events came from to decide whether to use them.

The essence is:
   def receiver(sender, arg1, arg2):
if sender=='ninja':
print arg1, arg2

def otherReceiver(arg1, arg2):
if arg1=='oNOerror': sys.exit(1)

dispatcher.connect(receiver, "foo")
dispatcher.connect(otherReceiver, "foo")
dispatcher.send('foo', sender='ninja', arg1='yaysuccess', arg2='whatever')
dispatcher.send('foo', sender='phb', arg1='oNOerror', arg2=123)

#==> yaysuccess whatever
#==> eek, error

Signals can be any (hashable?) python object, as can senders.

Even if you think this is a bad way to design an app (signals going all over the place, confusing the call stack), you might find it useful if you don't know how to design your app. Do it with the event dispatch mechanism first, then look for all the places where signals are sent, and figure out how to get them upstream in the call stack from the code handling them.

Thursday, June 23, 2005

Prohibited from using the wagon

Oh, BANDwagon.

Take the MIT Weblog Survey

Tuesday, June 14, 2005

Vellum is a Gnome Canvas Application

I have successfully made the switch from PyGame to Gnome Canvas in Vellum. While it has been by no means seamless, most of my problems were due to misunderstanding how it works, and general unfamiliarity with graphic programming in general and making drawing tools in particular.

Vellum maps now have scrollbars, which I never managed to implement in PyGame because it doesn't have a scrolling concept. It's not really meant for use with a toolkit, so this shouldn't be surprising. With Gnome Canvas, the scrollbars were the source of most of my problems. I was using GTK's Viewport widget, and dropping my Canvas in that to get scrollbars. In short, when it's in a Viewport, Canvas isn't double-buffered and it doesn't get the advantage of its infinite-width scrolling areas. In fact, resizing large scroll areas inside a Viewport is VERY slow. It was only when I realized that Canvas has native support for moving scrollbars around that I threw out the Viewport widget and got the performance that Canvas promises. Now the app is very fast, drawing is double-buffered and graphic operations take place near instantly.

After adding the working scrollbars I gave Vellum two new tools.
  1. There's a Pan tool, selectable by clicking on the hand icon in the toolbar. Pan allows you to drag the map around freely without having to reach for the scrollbars. This gives you finer control over what portion of the map you're looking at. I like Pan tools.
  2. There's a Magnify tool. This works like the magnify tool in Gimp. Click the Zoom button, then click somewhere in the canvas and drag. You get a rectangle which shows you what the inscribed area will be. When you release, the Canvas zooms in and centers on the area you inscribed with your rectangle. This tool will be one of the ones I use the most, as it allows a user to focus on particular aspects of a map or combat.
I will be adding keyboard shortcuts for all these tools in due time. For now I'm going to be taking pride in the fact that I've added a new application area to my repertoire of things I know how to program: Graphic editing applications.

This also represents another minor milestone for Vellum. The GUI actually does something useful. Granted, it's basically a crippled image viewer at this point, but look at it in context. There's now a way for a GM to share a map with his clients: create a .map file and put it in the server. The clients can connect and get that map, which happens automatically after they type in the right address. And everyone can scroll around or zoom to parts of the map that are interesting, which gives them a shared frame of reference for IRC-based games. It's certainly not earth-shattering, but it's something to dogfood.

I promised in my last post to provide more details on how I got Gnome Canvas to compile on Windows. There's really no special magic. If you want to see how it's done:

svn co svn://

There's a shell script in there to build libgnomecanvas and another one to build gnomecanvas-bindings. Ignore the libgnomecanvas one, as the binaries are checked into that directory as well (no special magic is required to build the C library runtime, it turns out). README.txt will fill in the rest. You just run the shell script, which invokes the code generator (I know, I didn't write it), compiles the two C files and links them.

Edit: Original post said I switched from "PyGTK", not from "PyGame" which is what I meant.

Monday, June 06, 2005

Gnome Canvas on Win32

Compiled libgnomecanvas and its bindings successfully (no errors or warnings!), ran a demo with it. Have yet to test text rendering or anything moderately complex with it, so cross your fingers. So far, though, lookin' gooood. It's compiled "native" (without Cygwin) so it'll be distributable without Cygwin.

Props to muntyan @ for guidance.

Details to come.

Saturday, June 04, 2005

Why Hacks are Bad

So, Pygame in a GTK window is a dead end. It requires the SDL window hack (see vellum/gui/ for implementation details). This works, with a lot of kludgity, in Python 2.3. I just discovered that it's completely broken again in Python 2.4 if you're on Windows. This, combined with the fundamental problem that you can only have one Pygame window in your program (not "at a time"; ever) makes Pygame a total dead end for Vellum. Maybe I can try to make gnomecanvas bindings compile again. (Sigh of a thousand hurricanes.)

Saturday, May 14, 2005

Vellum Early Screenshot

Vellum is really coming along now. The initial work integrating GTK with Pygame at a basic level is done. I can download and display icons. More importantly, I have the framework of the object model for representing these things in the program, and a clear understanding of the requirements.

In addition, on the IRC bot side, VellumTalk steadily gets smarter and better. It now understands how to subscribe an irc nick to its private messages channel, so the GM can see the results of all players' secret die rolls (with context). I'm currently in the process of rewriting the whole parsing mess in IRC using pyparsing, and that's going swimmingly.

This is the fun part, when development steadily accelerates and things change visibly very quickly.

Wednesday, April 27, 2005


Some interesting challenges in the newest project. Check out and the project page. Nothing much there yet, but you'll be able to find our source code. So far what we've got is a sandbox with some fine work on a very interesting character model by Aaron Lehmann, and a trunk with some rudimentary GUI code and a surprisingly useful IRC bot I spent a few hours on.

The goal of the project is to have a non-intrusive RPG client. (See previous post if you think RPG means Rocket-Propelled Grenade.) These are the target features:
  • An HTTP server to deliver data files such as map graphics and character icons
  • A graphical client capable of displaying a map with obscurement, icons, scaling, targetting arrows, and other visual effects
  • A PB server for realtime communication with the GUI clients
  • An IRC bot, integrated with HTTP and PB, to assist combat interactions, die rolls, and character management.
  • Export filters for various character generation programs, in particular PCGen.
  • Every component cross-platform. Runnable on Windows, Linux, Mac, hey maybe a PDA too.
What works:
  • There's an HTTP server. It delivers files.
  • There's a PB server. Right now it only delivers lists of filenames that are served by the HTTP server, along with their checksums and metadata.
  • There's a GUI client. It's written in GTK. It starts up, has a widget to specify the PB server to connect to, and when it connects it receives the list of files and requests those as well, and verifies their checksums.
  • There's an export filter for PCGen which writes (most of) the character data as YAML. YAML is almost perfectly-suited for this data format.
  • There's a rich character model. It isn't connected to anything yet, but it's capable of applying a variety of effects to various combat roles. For example, it knows about ability scores and equipment, and how to calculate armor class based on these things. (This is more complex than it sounds. Take a look at the code.)
  • There's an IRC bot. It has already proven useful, as we played a full 4.5-hour game session using only the VellumTalk bot and a web server for maps. It remembers "aliases" right now
I don't know how far away from a release we are, but I'm pretty happy about the progress so far.