Monday, May 11, 2015

Things going on

We are currently in a maintenance and cleanup phase of Evennia, where bugs are found and reported and things are getting more and more stable as people learn and use the new features we merged a few months back.

Overall though I must say the relatively big changes we did to the infrastructure (making Evennia into a full library and making a complete overhaul of the typeclass system behind the scenes) went over with surprising smoothness. There were a flurry of things to fix immediately after the devel-branch merger but no more than expected. For the big changes it really worked very well I think, with no big disaster stories. We have a few bugs lingering in the issue tracker that need to be addressed but nothing show-stopping.

I have been a bit busy with various projects off-MUD so to speak. I was contracted for making the cover and illustration for a book (this is not hobby art for once, but a professional commission which I was quite excited to be asked to do). I also author and draw a fantasy comic as part of another project.

I've not been slacking off on on the MUD side though: I have written and submitted an article for the revived Imaginary Realities e-zine (next issue should be out end of May/early June?) and another article (on Evennia) for the new Optional Realities MUD community website. I also contributed a game-design blurb for the latter's Dreaming Big contest, where you compete (for cash prizes, actually!) by submitting a game sale's pitch under 600 words.

The above mentioned Optional Realities website is so far picking up pace with quite good discussion in its forums (the similarity of name with Imaginary Realities is unfortunate, apparently they were not aware of the former when choosing it).  While targeted at discussions of RPI-style games (a sort of sub-genre of roleplay-enforced MUDs with perma-death), it already hosts several invited articles on game design and general game development that can be interesting for any MU* dev.

People should know by now that I like to support MUD community efforts when possible, and Evennia is thus  listed as an official "affiliate" to Optional Realities (which admittedly means little more than us linking to each other but still). The team behind OR is however also using Evennia for their own "Project Redshift" Sci-fi mud, so we hope to get plenty of good feedback as their project matures.

Image: Truss from Gems NFX

Monday, March 9, 2015

Documenting Python without Sphinx

Last week Evennia merged its development branch with all the features mentioned in the last post. Post-merger we have since gone through and fixed remaining bugs and shortened the list at a good clip.

One thing I have been considering is how to make Evennia's API auto-documenting - we are after all a MUD creation library and whereas our code has always been well-documented the docs were always only accessible from the source files themselves.

Now, when you hear "Python" and "documentation" in the same sentence, the first thought usually involves Sphinx or Sphinx autodoc in some form. Sphinx produces very nice looking documentation indeed. My problem is however as follows:
  • I don't want our API documentation to be written in a different format from the rest of our documentation, which is in Github's wiki using Markdown.  Our users should be able to help document Evennia without remembering which formatting language is to be used.
  • I don't like reStructuredText syntax. This is a personal thing. I get that it is powerful but it is also really, really ugly to read in its raw form in the source code. I feel the sources must be easy to read on their own.
  • Sphinx plugins like napoleon understands this ugliness and allows you to document your functions and classes in a saner form, such as the "Google style". One still needs reST for in-place formatting though.
  • Organizing sphinx document trees is fiddly and having had a few runs with sphinx autodoc it's just a mess trying to get it to section the Evennia sources in a way that makes sense. It could probably be done if I worked a lot more with it, but it's a generic page generator and I feel that I will eventually have to get down to make those toctrees myself before I'm happy.
  • I want to host the API docs as normal Wiki files on Github (this might be possible with reST too I suppose).

Long story short, rather than messing with getting Sphinx to do what I want, I ended up writing my own api-to-github-Markdown parser for the Evennia sources: api2md. Using Python's inspect module and aiming for a subset of the Google formatted docstrings, this was maybe a day's work in total - the rest was/is fine-tuning for prettiness.
Now whenever the source is updated, I follow the following procedure to fully update the API docs:

  1. I pull the latest version of Evennia's wiki git repository from github alongside the latest version of the main Evennia repository.
  2. I run api2md on the changed Evennia sources. This crawls the main repo for top-level package imports (which is a small list currently hard-coded in the crawler - this is to know which modules should create "submodule" pages rather than try to list class contents etc). Under each package I specify it then recursively gets all modules. For each module in that package, it creates a new Markdown formatted wiki page which it drops in a folder in the wiki repository. The files are named after the model's path in the library, meaning you get files like and can easily cross-link to subsections (aka classes and functions) on a page using page anchors.
  3. I add eventual new files and commit the changes, then push the result to the Github wiki online. Done!
(I could probably automate this with a git hook. Maybe as a future project.)

The api2md program currently has some Evennia-custom elements in it (notably in which packages it imports) but it's otherwise a very generic parser of Python code into Markdown. It could maybe be broken out into its own package at some point if there's interest.   

The interesting thing is that since I already have code for converting our wiki to reST and ReadTheDocs, I should be able to get the best of both worlds and convert our API wiki pages the same way later. The result will probably not be quite as custom-stunning as a Sphinx generated autodoc (markdown is a lot simpler in what formatting options it can offer) but that is a lesser concern.
So far very few of Evennia's docstrings are actually updated for the Google style syntax (or any type of formatting, really) so the result is often not too useful. We hope that many people will help us with the doc strings in the future - it's a great and easy way to get to know Evennia while helping out.

But where the sources are updated, the auto-generated wiki page looks pretty neat.

(Image from Wikimedia commons)

Monday, January 19, 2015

Building Django proxies and MUD libraries

2015 is here and there is a lot of activity going on in Evennia's repository, mailing list and IRC channel right now, with plenty of people asking questions and starting to use the system to build online games.

We get newcomers of all kinds, from experienced coders wanting to migrate from other code bases to newbies who are well versed in mudding but who aim to use Evennia for learning Python. At the moment the types of games planned or under development seems rather evenly distributed between RPI-style MUDs and MUSH games (maybe with a little dominance of MUSH) but there are also a couple of hack-and-slash concepts thrown into the mix. We also get some really wild concepts pitched to us now and then. What final games actually comes of it, who can tell, but people are certainly getting their MU*-creative urges scratched in greater numbers, which is a good sign.

Since Christmas our "devel" branch is visible online and is teeming with activity. So I thought I'd post an summary about it in this blog. The more detailed technical details for active developers can be found on Evennia's mailing list here (note that full docs are not yet written for devel-branch).

Django proxies for Typeclasses

I have written about Evennia's Typeclass system before on this blog. It is basically a way to "decorate" Django database models with a second set of classes to allow Evennia developers to create any type of game entity without having to modify the database schema. It does so by connecting one django model instance to one typeclass instance and overloading __setattr__ and __getattribute__ to transparently communicate between the two.

For the devel branch I have refactored our typeclass system to make use of Django's proxy models instead. Proxy models have existed for quite a while in Django, but they simply slipped under my radar until a user pointed them out to me late last year. A proxy model is basically a way to "replace the Python representation of a database table with a proxy class". Sounds like a Typeclass, doesn't it?
Now, proxy models doesn't work quite like typeclasses out of the box - for one thing if you query for them in the database you will get back the original model and not the proxy one. They also do not allow multiple inheritance. Finally I don't want Evennia users to have to set up django Meta info every time they use a proxy. So most work went into overloading the proxy multiclass inheritance check (there is a django issue about how to fix this). Along the way I also redefined the default managers and __init__ methods to always load the proxy actually searched for and not the model. I finally created metaclasses to handle all the boilerplate. We choose to keep the name Typeclass also for this extended proxy. This is partly for legacy reasons, but typeclasses do have their own identity: they are not vanilla Django-proxies nor completely normal Python classes (although they are very close to the latter from the perspective of the end user).
Since typeclasses now are directly inheriting from the base class (due to meta-classing this looks like normal Python inheritance), it makes things a lot easier to visualize, explain and use. Performance-wise this system is en par with the old, or maybe a little faster, but it will also be a lot more straight forward to cache than the old. I have done preliminary testing with threading and it looks promising (but more on that in a future post). 

Evennia as a Python library package

Evennia has until now been solely distributed as a version controlled source tree (first under SVN, then Mercurial and now via GIT and Github). In its current inception you clone the tree and find inside it a game/ directory where you create your game. A problem we have when helping newbies is that we can't easily put pre-filled templates in there - if people used them there might be merge conflicts when we update the templates upstream. So the way people configure Evennia is to make copies of template modules and then change the settings to point to that copy rather than the default module. This works well but it means a higher threshold of setup for new users and a lot of describing text. Also, while learning GIT is a useful skill, it's another hurdle to get past for those who just want to change something minor to see if Evennia is for them.

In the devel branch, Evennia is now a library. The game/ folder is no longer distributed as part of the repository but is created dynamically by using the new binary evennia launcher program, which is also responsible for creating (or migrating) the database as well as operating the server:

evennia --init mygame
cd mygame
evennia migrate
evennia start

Since this new folder is not under our source tree, we can set up and copy pre-made template modules to it that people can just immediately start filling in without worrying about merge conflicts. We can also dynamically create a setting file that fits the environment as well as set up a correct tree for overloading web functionality and so on. It also makes it a lot easier for people wanting to create multiple games and to put their work under separate version control.

Rather than traversing the repository structure as before you henceforth will just do import evennia in your code to have access to the entirety of the API. And finally this means it will (eventually) be possible to install Evennia from pypi with something like pip install evennia. This will greatly ease the first steps for those not keen on learning GIT.

For existing users

Both the typeclasses-as-proxies and the evennia library changes are now live in the devel branch. Some brave users have already started taking it through its paces (and is helping to flesh it out) but it will take some time before it merges into master.

The interesting thing is that despite all this sounding like a huge change to Evennia, the coding API doesn't change very much, the database schema almost not at all. With the exception of some properties specific to the old connection between the typeclass and model, code translate over pretty much without change from the developer's standpoint.

The main translation work for existing developers lies in copying over their code from the old game/ directory to the new dynamically created game folder. They need to do a search-and-replace so that they import from evennia rather than from src or ev. There may possibly be some other minor things. But so far testers have not found it too cumbersome or time consuming to do. And all agree that the new structure is worth it.

So, onward into 2015!

Image: "Bibliothek St. Florian" by Original uploader was Stephan Brunker at de.wikipedia Later versions were uploaded by Luestling at de.wikipedia. - Originally from de.wikipedia; description page is/was here.. Licensed under CC BY-SA 3.0 via Wikimedia Commons -

Thursday, October 2, 2014

Slowly moving through town

After getting questions about it I recently added the Slow Exit contribution to the main repository as an example. 

Delayed movement is something often seen in various text games, it simply means that the time to move from room to room is artificially extended.

Evennia's default model uses traditional MU* rooms. These are simple nodes with exits linking them together. Such Rooms have no internal size and no inherent spatial relationship to each other. Moving from any Room to any other is happening as fast as the system can process the movement.

Introducing a delay on exit traversal can have a surprisingly big effect on a game:
  • It dramatically changes the "feel" of the game. It often makes the game feel less "twitch" and slows things down in a very real way. It lets Players consider movement as a "cost".
  • It simulates movement speed. A "quick" (or maybe well-rested) character might perceive an actual difference in traversal. The traversal speed can vary depending on if the Character is "running" or "walking".
  • It can emulate travel distance. An Exit leading to "the top of the mountain" may take longer to traverse than going "inside the tent".
  • It makes movement a "cost" to take into consideration in the game. Moving back and forth over and over across a distance of multiple rooms becomes a much more daunting prospect with a time delay than if you could just zip along as quickly as you could press the button. This also has effects on map and quest design.
Introducing delayed movement in Evennia is simple. But to explain the idea, let's first briefly explain how Evennia implements Exits.

A brief sideline: About Exits

An Exit in Evennia is a persistent Object sitting in a room. The Exit class is just like any Object except for two things - it stores a "destination" property and it houses a CommandSet on itself. This particular CommandSet holds a single command with the same name as the Exit object.

Commands and CommandSets are things I've covered in earlier blog posts. Suffice to say is that any number of command sets can be merged together dynamically to at any moment represent the commands available to the Character at any given time or situation.

What happens when an Exit bject is in the same room as a Character is that the Exit's command set is dynamically merged with that of the Character. This means a new command - which always has the same name as the Exit - becomes available. The result is that if the Exit object is called "south", the Character can use the command "south". By default all the command does is to call a hook method on the Exit object. This hook hooks simply moves the calling Character to the "destination" stored by the Exit. Done!

The nice thing with this is that the whole system is implemented without any special cases or custom hard-wired code. It also means that the entire Exit system can be changed and modified without ever touching Evennia's core.

Delaying Exits

To delay the traversal, the principle is simple - after the Exit command has triggered, wait for a little while before continuing.

Technically we define a new class of Exit, let's call it SlowExit, inheriting from the default Exit. We locate the spot where the Exit normally sends traversing objects on their way (this is a method called move_to()).

Since Evennia is based on Twisted, we use Twisted's intrinsic CallLater() function to delay the move for as many seconds we desire (in the contrib I use a thin wrapper around CallLater called delay()). The result is that the command is called, you get a little text saying that you have started moving ... and a few seconds later you actually move.

Once one understands how Exits work it's really quite straight forward - see the code on github for more details (it's got plenty of comments).

In the contrib are also some example utility commands for setting one's movement speed and to abort movement if you change your mind before the timeout has passed.

This simple start can easily be expanded as befits each individual game. One can imagine introducing anything from stamina costs to make travel time be dynamically calculated based on terrain or other factors.

Monday, August 4, 2014

Dance my puppets

In many traditional multiplayer text engines for MUD/MUSH/MU*, the player connects to the game with an account name that also becomes their character's in-game name. When they log into the game they immediately "become" that character. If they want to play with another character, they need to create a new account.

A single-login system is easy to implement but many code bases try to expand with some sort of "account system" where a single login "account" will allow you to manage one or more game characters. Matthew “Chaos” Sheahan  beautifully argues for the benefits of an account system in the April issue of Imaginary Realities; you can read his article here.

Evennia and account systems

First a brief show of how Evennia handles this. We use the following separation:

Session(s) <-> Player <-> Objects/Characters(s)

The Session object represents individual client connections to Evennia. The Player is our "account" object. It holds the password hash and your login name but has no in-game existence. Finally we have Objects, the most common being a subclass of Object we call Character. Objects exist in the game. They are "puppeted" by Sessions via the Player account.

From this separation an account system follows naturally. Evennia also offers fully flexible puppeting out of the box: Changing characters (or for staff to puppet an NPC) is simply a matter of "disconnecting" from one Character and connecting to another (presuming you have permission to do so).

The Multisession modes of Evennia

This is the main gist of this entry since we just added another of these (mode 3). Evennia now offers four different multisession modes for the game designer to choose between. They affect how you gamers may control their characters and can be changed with just a server reload. 


Mode 0

This is emulates the "traditional" mud codebase style. In mode 0 a Session controls one Character and one character only. Only one Session per account is allowed - that is, if a user try to connect to their Player account with a different client the old connection will be disconnected. In the default command set a new Character is created with the same name as the Player account and the two are automatically connected whenever they log in. To the user this makes Player and Character seem to be virtually the same thing.


Mode 1

In this mode, multiple Sessions are allowed per Player account. You still only have one Character per account but you can control that Character from any number of simultaneously connected clients. This is a requirement from MUSHes and some slower-moving games where there are communities of gamers who want to conveniently track the progress of the game continuously on multiple clients and computers. 


Mode 2

In multisession mode 2, multiple Characters are allowed per Player account. No Characters are created by default in this mode, rather the default command set will drop you to a simplified OOC management screen where you can create new characters, list the ones you already have and puppet them. This mode offers true multiplaying, where you can connect via several clients simultaneously, each Session controlling a different Character.


Mode 3

This mode allows gamers not only to play multiple Characters on the same Player account (as in mode 2) but to also connect multiple Sessions to each Character. This is a multi-character version of Mode 1, where players can control the same Character via Player logins from several different clients on different machines in any combination.

It's interesting that some of these modes may seem silly or superfluous to people used to a certain type of MU* yet are killer features for other communities. It goes to show how different the needs are for users of different game styles.

Monday, June 30, 2014

Webby stuff

Latest Evennia come with a range of improvements, mainly related to its integration with the web.

New and improved ways to expand the website/webclient

Thanks to the work of contributor Kelketek, Evennia's Django-based web system (website and webclient) has been restructured to be much easier to expand. Previously you had to basically copy the entire web/ folder into your game and modify things in-place. This was not ideal since it made it inherently harder to update when things changed upstream. Now Evennia makes use of Django's collectstatic functionality to allow people to plugin and overload only the media files and templates that they need. Kelketek wrote a new and shiny web tutorial explaining just how things work. 

Websocket-based webclient with OOB

Evennia's webclient was an ajax-based one using a long polling ("comet") paradigm to work. These days all modern browsers support websockets though, a protocol that allows asynchronous server-client communication without the cludgery of long polling. So Evennia's new webclient will now use websockets if the browser supports it and fall back to the old comet client if it does not.

The new client also has full support for OOB (Out-of-band) communication. The client uses JSON for straight forward OOB messaging with the server. As part of this, I had an excuse to go back to clean up and make the OOB backbone of Evennia more complete. The server-side oob commands are borrowed from MSDP but the server side is of course independent of communication protocol (so webclient and telnet extensions can call the same server-side callbacks). I've not yet finalized the documentation for how to use the OOB yet, that will come soon-ish.

Sunday, June 15, 2014

Bringing back python memory

Lately I've done work on the memory management of Evennia. Analyzing the memory footprint of a python program is a rather educational thing in general.

Python keeps tracks of all objects (from variables to classes and everything in between) via a memory reference. When other objects reference that object it tracks this too.
Once nothing references an object, it does not need to be in memory any more - in a more low-level languages this might lead to a memory leak. Python's garbage collector handles this for us though - it goes through all abandoned objects and frees the memory for usage by other things. The garbage collector will however not do its thing as long as some other object (which will not be garbage-collected) still holds a reference to the object. This is what you want - you don't want existing objects to stop working because an object they rely on is suddenly not there.

Normally in Django, whenever you retrieve an database model instance, that exists only in memory then and there. If you later retrieve the same object from the database, the model instance you have to work with is most likely a new one. This is okay for most usage, but Evennia's typeclass system (described in an earlier blog entry) as well our wish to store temporary properties on models (existing until next server restart) does not work if the model instance we get is always different. It would also help if we didn't have to load models from the database more than necessary.

For this reason, Evennia uses something called the idmapper. This is a cache mechanism (heavily modified for Evennia) that allows objects to be loaded from the database only once and then be reused when later accessed. The speedup achieved from this is important, but as said it also makes critical systems work properly.

The tradeoff of speed and utility is memory usage. Since the idmapper never drops those references it means that objects will never be garbage collected. The result was that the memory usage of Evennia could rise rapidly with an increasing number of objects. Whereas some objects (like those with temporary attributes) should indeed not be garbage collected, in a working game there is likely to be objects without such volatile data. An example might be objects that are not used some of the time - simply because players or the game don't need them for the moment. For such objects it may be okay to re-load them on demand rather than keep them in memory indefinitely.

When looking into this I found that simply force-flushing the idmapper did not clean up all objects from memory. The reason for this has to do with how Evennia references objects via a range of other means. The reference count never went to zero and so the garbage collector never got around to it. 

With the excellent objgraph library it is actually pretty easy to track just what is referencing what, and to figure out what to remove. Using this I went through a rather prolonged spree of cleanups where I gradually (and carefully) cleaned up Evennia's object referencing to a point where the only external reference to most objects were the idmapper cache reference. So removing that (like when deliberately flushing the cache) will now make the object possible to garbage-collect.

This is how the reference map used to look for one type of Evennia object (ObjectDB) before the cleanup. Note the several references into the ObjectDB and the cyclic references for all handlers (the cyclic reference is in itself not a problem for reference-counting but they are slow and unnecessary; I now made all handlers use lazy-loading with weak referencing instead).

This is how the reference map looks for the same object now. The __instance__ cache is the idmapper reference. There are also no more cyclic references for handlers (the display don't even pick up on them for this depth of recursion). Just removing that single link will now garbage-collect ObjectDB and its typeclass (ignore the g reference, that is just the variable holding the object in ipython).
We also see that the dbobj.typeclass <-> typeclass.dbobj references keep each other alive and when one goes the other one goes too - just as expected.

An curious aspect of Python memory handling is that (C-)Python does not actually release the memory back to operating system when flushing the idmapper cache. Rather Python makes it internally available so that it does not need to request any more. The result is that if you look at Evennia with the top command, its memory requirement (for example while continuously creating new objects) will not actually drop on a idmapper flush, it will just stop rising.  This is discussed at length in this blog, it was good to learn it was not something I did at least.

Apart from the memory stuff, there is work ongoing with fixing the latest batch of user issue reports. Another dev is working on cleaning up the web-related code, it should make it a lot cleaner to overload web functionality with custom code. One of those days I'll also try to sit down and finally convert our web client from long-polling to use web sockets now that Evennia suppports web sockets natively. Time, time ...