Goonmill repo from scratch

In case you hadn't guessed by my last Mercurial post, I screwed up my Goonmill repository. It's now back, with all the history intact but the one file I shouldn't have committed now purged.

If you have a checkout, please get another checkout, you won't be able to update your existing one.

We apologize for the inconvenience.

Announcing a Playtools Mailing List

I have created a list for Playtools users and developers.

• List info page
  

(is also visible from the project's home page.)

I also added some branding to the Launchpad pages for both projects.

Expunging a problem file from Mercurial repo

Mercurial is almost the perfect version control system: fast, lean, distributed, easily extensible and reliable. It works by copying an entire repository, compressed, every time a clone needs to be made. This is much more efficient than it may sound, and the time it takes to transfer a Mercurial repo is quite comparable to the time it takes to checkout a Subversion repository.

The only time the system breaks down is if you lose your head and commit a large compressed database to the repository. Since I'm still learning how everything in Mercurial works, I did this accidentally, and made several commits before I realized my repo was ballooning out of control due to this one file with only 7 commits.

Here's how you recover.

1. Make sure you fix this before anyone clones your repository from upstream. The following procedure renders your repository unusable to anyone working on their own clone of it; they will have to clone a copy of the results and start fresh.
2. Get a list of all the commits that changed your file.
   $ hg log -M -r0:tip --template "{rev} {files}\n" goonmill/srd35.db.gz
11 goonmill/srd35.db.gz
52 goonmill/srd35.db.gz
59 goonmill/srd35.db.gz
99 goonmill/alter3.sql goonmill/srd35.db.gz
115 goonmill/srd35.db.gz
160 goonmill/srd35.db.gz srd35.odb
189 goonmill/srd35.db.gz
3. Export the entire repository as patches; use hg export to extract each one to a separate file.
   $ mkdir ../Goonmill-revs; hg export -g -o ../Goonmill-revs/%r:%n-of-%N $(hg log -M --template "{rev} ")
   (Hundreds of patch files created)
4. Delete the ones that you found in step 2. The reason I had you print {files} in that step was so that you could check to see if any other files would be affected by the new hole in the history. For example, my commit 99 above changes another file. Instead of deleting that revision, I will edit the patch so only the patch to the sql script is committed.
5. "hg init" a new repository and use "hg import Goonmill-revs/*" to recreate.

You now have a new repository with the same revision history as before, but with your one problem file not contained in the repo history. Next time, commit that database uncompressed, because Mercurial does efficient binary diffs that don't work very well on compressed files but work great on uncompressed structured binary files like a database.

Playtools is Born

Per jml's suggestions:

• The bug tracker and a project page live on Launchpad, and the source link is prominently featured there too.
  
• There's a dedicated home page on the website for it. Anyone want to contribute a logo? :-)
• The new name is "Playtools".
  

I've installed mailman, but I need to configure Apache correctly for it at an hour when I'm not falling asleep in my seat.

Oh by the way, I gave Goonmill the Launchpad treatment as well.

D&D Datahead Community, III

(Jump back to Part I.)

In a database of typed columns in tables, you must have a single schema, hosted at one location. Someone must be gatekeeper of what goes into it, and how it gets extended. To be able to use the data, applications written for it must trust the gatekeeper. If something goes into it, it becomes part of all applications written against it, and an app developer can't reject that data without making special cases in the application (blacklisting). Worse, if something useful doesn't go into it, your application can never make use of that thing. Your application loses the freedom of choice; neither the application itself nor its users can decide whether to trust some data and not other data.

An application built on published documents of triples, however, can be written so that both the application and the users of the application can choose which data to trust, by choosing from the list of URLs that the application loads as its data.

This goes right to the heart of the Paper and Pencil RPG world. This is a world of publish-or-perish; companies must produce lots of supplements to the rules of existing games, or fail to make money. Players, meanwhile, do not have the wherewithal to buy all supplements everywhere, so they pick and choose which rules to buy, and which ones ultimately to use in play. Not all publishers are equally popular, not all suplements are equally popular. Some gamers play extremely old versions of the rules; some even adapt supplements of the new rules to the old rules (the P&P version of a backport). And, any GM worth anything has his own set of House Rules, which he writes down only for the benefit of the rules lawyer in his group. In short, the RPG world is information anarchy.

In order for an open source software community to form in this environment, some of this chaos must be reigned in so that GMs can make use of the software in the crazy scenarios that actually exist at every game table. And I do mean every one; I have yet to meet a D&D player who took the stock PHB/DMG/MM and played strictly by those rules and only by those rules.

Supplements at the Application Level

The solution lies in a format that lets anyone publish their own rules. N3 is the basis, not for tables, but for documents, and documents can be published by anyone. But, unlike the pages of the SRD, these documents are unordered. This lack of order actually becomes an asset in this context! Think about XML. In order to modify an XML document, you must parse it, insert your concept, and then re-emit it. This makes you a publisher of the entire set of facts the document is describing. Therefore, anyone who wants to build on the original document but include your facts, must derive from your document, and so on.

In the unordered N3 format, however, all new facts just go on the pile. They can appear anywhere, even on another webserver. They do not have to be accompanied by a schema. In short, they are an explicit genuflection to the information anarchy of P&P game publishing. If you want to write an application about my dragon, and include the fact that dragons have a caster level of 15, you don't have to republish my entire dragon, just publish this document:

@prefix cdd: <http://thesoftworld.com/2007/monsters#>
cdd:youngAdultGoldDragon :casterLevel 15;

There, now anyone who wants to use a gold dragon and include its caster level in the facts about it, needs only whitelist my document and your new one at the same time. The facts might be new lists of monsters. They might be new rules to apply to already-published monsters--even in a blanket fashion: you can publish a statement such as "anything which is a monster has 15 Time Points", and now they do. They might even be cancellation of some other whitelisted rules if, for example, you like some parts of one publication but not other parts. An app can load any of these documents the developer wants to load, even cache them so they don't have to be requested over the Internet, but include only the facts that the application developer wants to include.

Or better yet, the ones the User wants to include. Because now that we've established these documents should be published somewhere, an application can simply collect all the documents it knows about and present them to the user as a list; the user can now pick and choose which rules his creatures obey and which ones his creature ignores. The user can even type in a new URL, and the application will fetch that document and include its facts, even though it didn't know about it before.

Communities can vote on these publications, so they get "most popular" and "highest rated" rankings and can compete with each other. Any member of the community can publish material, and instantly see how his new monster or new rules plays with any other member's stuff.

Next Steps

The technology, and in particular the standard, are only half-finished here, but they have a real application that needs them, which is Goonmill. Goonmill is far more than half-finished; in fact, I've already partaken of its delicious dogfood, and used the statblocks in my own campaign world. So I'm really not far from being able to publish real standards. But already there are some documents online that yearn to be pounded on by brutal application developers until they do something sensible. I am hosting them in what should be their permanent home, so developers will be able to link directly to these concepts and make use of them in their own applications.

1. For Goonmill to continue, I now get to undertake converting a SQL database made from the SRD, into a series of N3 documents representing the same concepts as triples. For monsters, the vast majority of the work has been done in Goonmill itself. Converting monsters is simply a matter of rendering all 650-ish of them to N3 instead of HTML, a labor which will eventually become monster.n3 at the above URL. Two of those files are in use by the application already, family.n3 and specialAbility.n3.

2. I'll then use the same process to convert all spells, skills, items and feats found in the SRD. When done, this will be a set of the core OGC documents, converted into a form that applications can easily load and understand, even access over the Internet. I cannot change the license they are published under, which is the OGL, but the OGL is a license that grants freedoms, not unlike the GPL as applied to game content.

3. An archive form of the document standard will have to be brainstormed and spec'd out. This format should allow us to zip/bundle N3 with a more prose-focused format like XHTML, so we don't have to crowbar a human-readable description of a red dragon into N3.

4. Tools will have to be written so game developers, who may not want to know what the inside of Vim looks like, can convert their content to the N3 format. But it need not be the publishers themselves who do this; any content which is OGC or CC-sharealike licensed can be easily converted by any fan of the material. It doesn't even have to be an application developer, just someone with a website who can publish mime-type "text/rdf+n3". It also must be noted that one reason I chose N3 as my format was so that a person can create these documents without a fancy XML editor, and by learning only a modicum of syntax. (I was, and am still, a fan of the "YAML" syntax, and N3 is simpler to learn than that.)

Conclusions

When these initiatives get under way in earnest--and I am not waiting for anyone's help, this is a banner I will take up for my own sake if nobody else is ready to jump in yet--others can and will write applications against it, including the data and even the N3 markup itself entirely within their applications, or loading them from their canonical site directly. We will start to see the beginnings of actual cooperation among game software developers, and maybe an increase in the number of game software developers.

As important, though, is that game developers take the opportunity to publish their game materials using a new standard that they've never had before. If you are thinking about writing or publishing game content, talk to me about this format, because I want to give you all the technical help I can give. I hope to publish some of my own game material this way to kick things off, which is really more like a #5 for next steps, since people will want to see how this thing really works when they consider whether to get on board.

D&D Datahead Community, II

In my last post I discussed the goals of the projects I'm working on. Pushing on, this post is about the means I am using to that end.

A long-winded definition: game software means software written to support paper-and-pencil games, board games, basically any kind of game that requires a bunch of friends to get together, learn the rules, and muddle through a session on their own. In this discussion, game software isn't about computer games, even CRPGs, although certain CRPGS (NWN comes to mind) lie on the border between that space and this one, especially where their construction toolkits are concerned. In particular, game software will mean software that helps Game Masters--GMs--create, recruit players for, publish modules for, and run their games.

The idea at the center of this is that developers should be able to write software for their favorite games. Enabling them to do so shouldn't be hard. The game software community currently suffers from a lack of cooperation. The open source "revolution" just hasn't found this niche yet. Software publishers all seem to start from scratch, nobody writes frameworks or talks about best practices or forms committees. Nobody standardizes anything. Games software suffers and its developers spin their wheels.

Goonmill will be my first attempt to make a standard for something. Like a lot of standards, it builds on other standards. In this case, it makes use of N3, shorthand for Notation3. N3 provides a wonderfully extensible data framework for this initiative. So extensible that it seems to be completely about extensibility.

Suppose I publish a document like this:

:goldDragonYoungAdult
p:cr 14;
p:alignment c:lawfulGood;
p:size c:huge;

p:initiative +4;

p:hitPoints "20d12+100";
p:damageReduction [ c:magic 5 ];
p:armorClass 27;
p:immunity c:fire, c:paralysis, c:sleep;
p:save [ c:fort +17 ], [ c:ref +12 ], [ c:will +16 ];

p:speed [ c:ground "60 ft." ],
 [ c:flight "200 ft. (poor)" ],
 [ c:swim "60 ft." ]
;
p:bab +20;
p:grapple +38;
p:space "15 ft.";
p:reach "10 ft. (15 ft. with bite)";
p:melee [ a c:AttackGroup;
 c:claw (+29  "2d6+5");
 c:claw (+29  "2d6+5");
 c:bite (+29  "2d18+10");
 c:wing (+28  "1d8+5");
 c:wing (+28  "1d8+5");
 c:tail (+28  "2d6+15");
];
p:abilityScore [ c:str 31 ],
 [ c:dex 10 ],
 [ c:con 21 ],
 [ c:int 18 ],
 [ c:wis 19 ],
 [ c:cha 18 ]
;
.

I won't get into the subtleties of syntax here, just present the bare bones; if you're waiting to hear about the D&D stuff, bear with me one little bit more. The above is pretty readable and pretty easy to write with a text editor, as well, which gives it two marks up on XML. The document above breaks down into a number of subject-verb-object statements. On most of the lines above, you see only the verb-object part followed by a semicolon; this is because the subject in every case is :goldDragonYoungAdult. Semicolons separate two S-V-O statements with the same Subject, so each of the following lines needs to give only a verb-object. In a similar fashion, commas separate statements with the same Subject and Verb, so only the Object is stated for the comma-separated groups. () presents an ordered list, and [] presents an anonymous subject, so e.g. [ :foo :bar ] means "there exists some thing, whose :foo is :bar".

However, the important thing here is that N3, like all triples formats, describes a set of unordered relations. Unlike, say, a relational database which stores tables of strongly-typed columns with relations, N3 essentially stores only relations.

Why Does N3 Matter?

This has gotten pretty long--more tomorrow in Part III.

The Game Master Show

This isn't the "next post" I just promised. I just wanted to point visibly at The Game Master Show and cheer. I heard a tip, decided to listen to the podcast just now. I'm 2/3 of the way through "Fred Hicks Spends My Money" and I'm already hooked on this podcast. It's for gamers. Check it out.

D&D Datahead Community

One of my agenda items for Goonmill is to build a better community for software nerds who like to roleplay, and are interested in software about roleplaying. Vellum is definitely aimed at this idea as well.

With the software I'm creating, I have a few specific goals in mind. Most important to me are:

1. Develop tools for the community to use. Part of community-building is connecting people, and the tools should help us do that. In particular, they should help gamers build and then play games with each other.
2. Bring together programmers who want to work on games. This is entirely selfish. I want to hear from the professional programmers who game, and help them implement their ideas for more gaming tools. The tools should form a cloud of gaming-centric working parts, interlocking and building on each other. This, in turn, strengthens my own offerings.
   
3. Stimulate interest in role-playing games. Bring together gamers. I'm passionate about them, and I know a lot of other people are also passionate about them. Games are fun, and creative games are more fun. There are a lot of people out there who played as kids and don't think they can find anyone who plays as an adult. I'd like them to know that's not true.
4. Make it possible for adult gamers with lives and jobs to still roll out a game quickly. This goal is very personally applicable. The tools should support, not dominate, our existing lifestyles. I don't have 4 hours to spend for each hour gameplay. A tool that saves me an a hour of prep will get me into 33% more games per week.
   
5. Profit? No, really! I hope to found a company dedicated to developing tools for the gamer, and games for the gamer. People like me have disposable income, and no worthwhile hobby products to spend it on.
   
6. Build resources for would-be game publishers. Putting out professional-quality playing materials takes time, but it's time some of us want to spend because we enjoy the act of creation and we want to present what we've created in a nice frame. Or we want to sell it online. Or we want to give it as gifts to our fellow gamers. The tools should help us do that, in as many formats as possible: print, HTML, VGT.
   

At first, the people I see using my tools are the GMs of the world. We care about the creation. We have our own ideas. We have our own rules. We have players who want their ideas to make it into the game.

In my next post, I'll talk about why the underlying data framework matters in the effort to serve this community, and what Goonmill is doing with it.

Continue to part II.

Goonmill

For the past few months I have been putting my extra few minutes into an application called "Goonmill". This is software for generating MM4-format statblocks for monsters. I wrote it as a tool I can use when scripting adventures to save the drudgery of looking up monsters, statting them out, and presenting them in a format convenient for use at the game table.

Take a look at the online demo. Features include:

• Über-simple workflow. Search, click, optionally tweak, repeat.
  
• A (nearly invisible) box near the top of the statblock is clickable and lets you enter a count. When you do so, Goonmill rolls hit points for that many monsters, and displays them on the statblock.
• Similar for naming the monster (next to the hit point box).
• Similar for setting a custom alignment on the creature.
  
• A ton of work has gone into generating correct statblocks. There are simpleparse parsers for a large number of the stats on the page, dedicated to reading the text from the original database (designed for old-format statblocks) and putting it into the much-more-picky new-format statblock.
  
• Makes use of Nevow Athena for everything, which means it feels a bit more like a desktop app than a web app.
  

It's not perfect.

• The search is crap and needs to be replaced--it's currently just a substring search on the beginnings of words. There isn't even sorting of links by relevance.
• Lots of little CSS/UI issues.
• The (nearly invisible) boxes near the top need some rethinking so they aren't nearly invisible, and so they don't cause the line to overflow when you edit them.
• Need a print version of the monster history, especially with the ability to print one monster per page for GMing.
• It's not very pretty. Someone with an eye for web aesthetics should polish it a bit.
  
• I'm not sure what to do with the possessions and spells, the only yellow (TODO) items left in the statblock. I would like to create a whole workflow around those items, with UI for choosing manually and a button for generating sane values there.
• Same goes for NPC characteristics. I'd like to be able to just add class levels to a creature.
  
• Same goes for making animal companions and familiars.
• Same goes for using monster templates.
  
• Data work: Output everything to a triplestore (N3 or similar) as an alternative to output to a statblock. This would allow me to get rid of all the parsers and the sqlite part of the backend. Better yet, this would allow anyone to publish their own classes, creatures, items, and spells, on their own websites, and use Goonmill to generate those creatures. Initial (manual) work has been done; a lot of the data already exists in N3 format, and there's a preliminary O3M (object-triplestore mapper) for making Python objects from this data format. There's even a triplestore web sandbox built into Goonmill, for testing SPARQL queries against it.

This open source project needs developers!

If you are a fan of role-playing games and a software developer, I need your help. Any of the items in the above list are fair game right now. Source access requires Mercurial.

hg clone http://goonmill-source.thesoftworld.com/

Leave a comment if you want write access.