Out-of-band mergings

As of today the development repository of Evennia, which has been brewing for a few months now, merged into the main repository. This update grew from one experimental feature to a relatively big update in the end. Together with the "many-character-per-player" feature released earlier, this update covers all the stuff I talked about in my Behind the Scenes blog post.

•  Evennia's webserver was moved from Portal to Server. This moves all database-modifying operations into the same process and neatly avoids race conditions when modifying a game world from various interfaces. 
• The OOB (Out Of Band) handler was implemented. This goes together with a protocol for telnet sub-negotiations according to the MSDP specification. The handler allows on-demand reporting whenever database fields update. It also offers regular polling of properties if needed. A user can customize which oob commands are available to the client and write whatever handlers are needed for their particular game. In the future we'll also add support for GMCP, but the lack of a central, official specification is off-putting (if there is a central document besides accounts of how individual games chose to implement GMCP, please let me know). For our own included web client, we'll likely just use JSON straight off.
• Our channel system is now typeclassed. If you are not familiar with Evennia this won't mean much to you - In short it means developers will be able to customize their channel system much easier than in the past since a channel can be treated pretty much like any Python class (thanks go to user Kelketek who actually did the implementation).
• We added the concept of Tagging, as a more generalized version of our old Alias system. Tagging is just what it sounds like - it allows you to tag all your objects in order to group them and easily (and efficiently) find them later. Tagging offers a powerful way to create what other code bases refer to as "zones". There are many other possible uses though, such as having effects appear only in certainly tagged rooms, indicate which Characters have joined a particular guild and so on. 
• Behind the scenes there were a lot of cleanups, along with minor API changes mentioned on the mailing list. A slew of older Issues were also fixed with this merge.

Now, in the immediate future we'll be working on updating the documentation. Evennia prides itself with having a large body of documentation and this update comes with small changes to various existing systems. There are also a few bigger new features (such as OOB). No doubt the near future will also see other issues popping up as more people try things out.

 

Imaginary Realities update

Apparently the reboot of Imaginary Realities (to which I contribute two articles) has been pushed forward a week or two. Reason being, apparently, to finalize the actual presentation of the content. I already signed off on the last editorial fixes way before deadline, so I guess it's just to wait and see what comes of it!

A list of Evennia topics

Some Evennia updates.

Development
Lots of work has been happening in the dev-clone of Evennia over the last few months.
As alluded to in the last blog, the main work has been to move Evennia's webserver component into the Server-half of Evennia for various reasons, the most obvious one to make sure that all database writes happen in the same process, avoiding race conditions. But this move lead to a rework of the cache system, which in turn lead to me having to finalize the plans for how Out-of-Band protocols should be implemented server-side. And once that was finalized, OOB was pretty much implemented anyway. As part of making sure OOB trackers were updated correctly at all times meant reworking some of the ways data is stored ... So one thing led to another making this a bigger update than originally planned.

I plan to make a more detailed post to the mailing list soon with more technical details of the (relatively minor) API changes existing users should expect. The merging of the clone into the main repo is still a little way off, but adventurous users have already started testing things.

Google Code
I like Google Code. It's easy to manage and maintain, it has a good wiki and Issue system, not to mention that it allows the use of Mercurial. But in the beginning of September, suddenly all links to our user's clone repositories were gone from the front of the project page. Not only that, creating new clones just didn't work anymore.
Now any site can have bugs, and we made an issue for it (other projects were similarly affected). But nothing happened for the longest time - at least two months given that we didn't report it right away. Just recently the functionality came back but there is no confirmation or comments from Google (our issue is not even closed).
That such a fundamental feature can go unheeded for so long is disturbing to me, driving home the fact that Google is certainly not putting much priority in their code hosting.

Community
Some furious activity in the IRC chat lately, with new people dropping in to chat and ask about Evennia. For example, an enthusiastic new user learned not only about Evennia but also Python for the first time. It was a lot of fun to see him go from having no programming experience except mush softcode to doing advanced Evennia system implementations in the course of a week and offering good feedback on new features in two. Good show! The freedom you get upgrading from something like softcode to Evennia's use of a full modern programming language was seemingly quite eye-opening.

Other discussions have concerned the policies around using clones/branches for development as well as the benefits of some other hosting solution. Nothing has been decided on this. There is however now also an official GitHub mirror of Evennia's main repo to be found here.

Imaginary Realities
The deadline for entering articles for the Imaginary Realities web zine reboot has passed. It's a good initiative to bring this back - the original (archived) webzine remains a useful mud-creation resource to this day. I entered two articles, one about Evennia and another about general mud-roleplaying. It will be fun to see how it comes out, apparently the first issue will appear Nov 13. 

One to Many

As of yesterday, I completed and merged the first of the three upcoming Evennia features I mentioned in my Churning Behind the Scenes blog post: the "Multiple Characters per Player" feature.

Evennia makes a strict division between Player (this is an object storing login-info and represents the person connecting to the game) and their Character (their representation in-game; Characters are just Objects with some nice defaults). When you log into the game with a client, a Session tracks that particular connection.

Previously the Player class would normally only handle one Session at a time. This made for an easy implementation and this behavior is quite familiar to users of many other mud code bases. There was an option to allow more than one Session, but each were then treated equally: all Sessions would see the same returns and the same in-game entities were controlled by all (and giving the quit command from one would kick all out).

What changed now is that the Player class will manage each Session separately, without interfering with other Sessions connected to the same Player. Each Session can be connected, through the Player, to an individual Character. So multiple Characters could in principle be controlled simultaneously by the same real-world player using different open mud clients. This gives a lot of flexibility for games supporting multi-play but also as a nice way to transparently puppet temporary extras in heavy roleplaying games.

It is still possible to force Evennia to accept only one Session per Player just like before, but this is now an option, not a limitation. And even in hardcore one-character-at-a-time roleplaying games it is nice for builders and admins to be able to have separate staff or npc characters without needing a separate account for each.

This feature took a lot more work than I anticipated - it consitutes a lot of under-the-hood changes. But it also gave me ample opportunity to fix and clean up older systems and fix bugs. The outcome is more consistency and standardization in several places. There are plenty of other noteworthy changes that were made along the way in the dev branch along with some API changes users should be aware of.
So if you are an Evennia game developer you should peek at the more detailed mailing list  announcement on what has changed. The wiki is not updated yet, that will come soon.

Now onward to the next feature!

Churning behind the scenes

At the moment there are several Evennia projects churning along behind the scenes, none of which I've yet gotten to the point of pushing into a finished state.  Apart from bug fixes and other minor things happening, these are the main updates in the pipeline at the moment.

Multiple Characters per Player/Session

Evennia has for a long time enforced a clean separation between the Player and the Character. It's a much appreciated feature among our users. The Player is "you", the human playing the game. It knows your password, eventual user profile etc. The Character is your avatar in-game. This setup makes it easy for a Player to have many characters, and to "puppet" characters - all you need to do is "disconnect" the Player object from the Character object, then connect to another Character object (assuming you are allowed to puppet that object, obviously). So far so good. 

What Evennia currently doesn't support is being logged in with different client sessions to the same Player/account while puppeting multiple characters at the same time. Currently multiple client sessions may log into the same Player account, but they will then all just act as separate views of the same action (all will see the same output, you can send commands from each but they will end up with the same Character). 

Allowing each session to control a separate Character suggests changing the way the session is tracked by the player and Character. This turns out to be more work than I originally envisioned when seeing the feature request in the issue tracker. But if my plan works out it will indeed become quite easy to use Evennia to both allow multi-play or not as you please, without having to remember separate passwords for each Character/account.

Webserver change to Server level

Evennia consists of two main processes, the Portal and the Server. The details of those were covered in an earlier blog post here. Evennia comes with a Twisted-based webserver which is currently operating on the Portal level. This has the advantage of not being affected by Server-reboots. The drawback is however that being in a different process from the main Server, accessing the database and notably its server-side caches becomes a problem - changing the database from the Portal side does not automatically update the caches on the Server side, telling them that the database has changed. Also writing to the database from two processes may introduce race conditions. 

For our simple default setup (like a website just listing some database statistics) this is not a terrible problem, but as more users start to use Evennia, there is a growing interest in more advanced uses of the webserver. Several developers want to use the webserver to build game-related rich website experiences for their games - online character generation, tie-in forums and things like that. Out-of-sync caches then becomes a real concern. 

One way around this could be to implement a framework (such as memcached) for homogenizing caches across all Evennia processes. After lots of IRC discussions I'm going with what seems to be the more elegant and clean solution though - moving the webserver into the Server process altogether. The Portal side will thus only hold a web proxy and the webclient protocol. This way all database access will happen from the same process simplifying things a lot. It will make it much easier for users to use django to create rich web experiences without having to worry about pesky behind the scenes things like caches and the like. 

Out-of-band communication 

This has been "brewing" for quite some time, I've been strangely unmotivated to finalize it. Out of band communication means the MUD client can send and receive data to/from the server directly, without the player having to necessesarily enter an active command or see any immediate effect. This could be things like updating a health bar in a client-side GUI, redirect text to a specific client window but also potentially more advanced stuff. I created the Evennia-side oob-handler over Christmas; it allows for client sessions to "sign up" for "listening" to attribute updates, do scheduled checks and so on. It's already in the codebase but is not activated nor tested yet.

On the protocol side (for serializing data to the client) I have a MSDP implementation ready for telnet subnegotiation, it should be simple to add also GMCP once everything is tested. A JSON-based side channel for the webclient is already in place since a long time if I remember correctly, it just need to be connected to the server-side oob-handler once that's finished. 

Evennia changes to BSD license

As of today, Evennia changes to use the very permissive BSD license.

Now, our previous "Artistic License" was also very friendly. One main feature was that it made sure that changes people made to the core Evennia library (i.e. not the game-specific files) were also made available for possible inclusion upstream. A good notion perhaps, but the licensing text was also quite long and it was clear some newcomers parsed it as more restrictive than it actually was.

... And let's be honest, it's not like I would have come hunting down anyone not complying fully with the Artistic license's terms. Changing to the much simpler and more well-known BSD license better clarifies the actual licensing situation.

After all, far too many older MUD-code bases are weighted by a legacy of licensing issues. Anything we can do to avoid this is better in the long run. Indeed we hope this change in licensing will remove eventual licensing doubts for new adopters and have more people join and contribute to the project.

Community interest

It's fun to see a growing level of activity in the Evennia community. The last few months have seen an increase in the number of people showing up in our IRC channel and mailing list. With this has come a slew of interesting ideas, projects and MUD-related discussion (as well as a few inevitable excursions into interesting but very non-mud-related territory - sorry to those reading our chat logs).

One sign of more people starting to actually use Evennia "for real" is how the number of bugs/feature requests have been increasing. These are sometimes issues related to new things being implemented but also features that have been there for some time and which people are now wanting to use in new and creative ways - or systems which noone has yet really used "seriously" before. This is very encouraging, especially since a lot of good alternative solutions, variations and edge cases can be ironed out this way. So keep submitting those Issues, people!


The budding Evennia community consists of people with a wide variety of interests, skillset and ambition.

There are quite a few people who sees Evennia as a great stepping stone for learning Python, or for getting experience with creating a bigger programming project in general. Some are skilled programmers in other languages but we also have a few with only limited prior coding experience. From the experience in chat, it's really quite striking how fast members pick up the ropes. I'd like to think our documentation is at least partially helping here, but of course it helps that Python is inherently very easy a language to learn and use in the first place.
Not all are participating with the goal of building a specific game. The general flow of patches and clone repository merges have also picked up. We have some users which are primarily interested in a coding challenge, to help with fixing bugs and features, or which uses Evennia as a starting point for exploring various web- and technical solutions that may or may not be a part of Evennia in the future.


The proposed Evennia game projects are just as varied as its users - and none are yet open to the public. As is common with these things, it's of course hard to determine who actually has the time and discipline to bring their plans to fruition. But I should really start to keep some sort of record of who works on what, I'm terrible with remembering this stuff ... so below is just some sort of summary of my impressions, not a comprehensive listing.

As can be expected, most proposed Evennia projects concern relatively standard MUD-style games. A few people are into building traditional hack-and-slash variety games, but most want to expand on the concept considerably. There was even one user exploring using Evennia for a RobotWars kind of experience (where you "program" robot programs in a custom language and battle them). Another project (Avaloria, also blogging on the MUD-dev rss feed) aims for a sort of base-building/strategy mechanic combined with more traditional MUD elements. There are at least two zombie-survival concepts floating around and a few large-scale procedural-content-driven science-fiction text games. One user has apparently a working Smaug->Evennia importer.

It seems that most Evennia users want to offer some sort of roleplaying environment, or at least a "roleplay-friendly" one. Currently we have at least two MUCK admins who aim to convert their existing, running games to Evennia. Whereas the initial idea was to implement parsers for MUCK's MUF language, it seems the conclusion has now shifted to it being faster and easier to just rewrite the MUF-coded functionality in Python (and maybe use something like Evlang for player scripting instead). Several people have announced their interest in creating "RPI"-style games (Armageddon seems to be a big inspiration here), but there was also a MOO admin and even a writer of Interactive Fiction who dropped into the mailing list to see if Evennia could be used for their style of game.

How many of these projects actually reach a point of maturity remains to be seen. But that people are wanting to use the system and is really putting it through its paces is encouraging and very helpful for general Evennia development.

Combining Twisted and Django

Franco Nero as the twisted gunslinger Django

Newcomers to Evennia sometimes misunderstand it as being a "Django mud codebase somehow using Twisted". The correct description is rather that Evennia is a "Twisted-based mud server using Django". Allow me to elaborate.

A mud/mux/moo/mu* is per definition a multi-user online game system. All these users need to co-exist on the server. If one player does something, other players shouldn't have to (noticeably) wait for that something to end before they can do anything. Furthermore it's important for the database schema to be easy to handle and upgrade. Finally, in a modern game, internet presence and web browser access is becoming a must. We combine two frameworks to achieve this.

Two frameworks combined

Twisted is a asynchronous Python framework. "Asynchronous" in this context means, very simplified, that Twisted chops up code execution into as small bits as the code lets it. It then flips through these snippets rapidly, executing each in turn. The result is the illusion of everything happening at the same time. The asynchronous operation is the basis for the framework, but it also helps that twisted makes it easy to support (and create) a massive range of different network protocols.

Django implements a very nice abstract Python API for accessing a variety of SQL-like databases. It makes it very convenient to maintain the database schema (not to mention that django-South gives us easy database migrations). The fact that Django is really a web framework also makes it easy to offer various web features. There is for example an "admin site" that comes with Django. It allows to modify the database graphically (in Evennia's case the admin site is not quite as polished as we would like yet, but it's coming).

Here are some highlights of our architecture:

• Portal - This is a stand-alone Twisted process talking to the outside world. It implements a range of communication protocols, such as telnet (traditional in MUD-world), ssh, ssl, a comet webclient and others. It is an auto-connecting client to Server (below).
• Server - This is the main MUD server. This twisted server handles everything related to the MUD world. It accesses and updates the database through Django models. It makes the world tick. Since all Players connect to the Server through the Portal's AMP connection, it means Server can be restarted without any players getting kicked off the game (they will re-sync from Portal as soon as Server is back up again).
• Webserver - Evennia optionally starts its own Twisted webserver. This serves the game's website (using the same database as the game for showing game statistics, for example). The website is of course a full Django project, with all the possibilities that entails. The Django admin site allows for modifying the database via a graphical interface. 
• Webclient - There is a stand-alone MUD web client existing in a page on the default website. This uses Twisted to implement a long-polling ("comet") connection to a javascript client. As far as Evennia's concerned, this is just another outgoing protocol served by the Portal.
• Other protocols - Since it's easy to add new connectivity, Evennia also offers a bunch of other connectivity options, such as relaying in-game channels to IRC and IMC2 as well as RSS feeds and some other goodies. 

On the joining of the two

An important thing to note about Twisted's asynchronous model is that there is no magic at work here: Each little snippet of code Twisted loops over is blocking. It's just hopefully not blocking long enough for you to notice. So if you were to put sleep(10) in one of those snippets, then congratulations, you just froze the entire server for ten seconds.

Profiling becomes very important here. Evennia's main launcher takes command arguments to run either of its processes under Python's cProfile module. It also offers the ability to connect any number of dummy Players doing all sorts of automated random actions on the server. Such profile data is invaluable to know what is a bottleneck and what is not.

I never found Twisted asynchronous paradigms much harder to understand than other code. But there are sure ways to write stupid blocking code that will come back and bite you. For example, much of  Evennia's workload is spent in the Server, most notably in its command handler. This is not so strange; the command handler takes care of parsing and executing all input coming from Players, often modifying the game world in various ways (see my previous post for more info about the command handler).
The command handler used to be a monolithic, single method. This meant that Twisted had to let it run its full course before letting anyone else do their turn. Using Twisted's inlineCallbacks instead allowed for yielding at many, many places in this method, giving Twisted ample possibilities to split execution. The effect on multi-user performance was quite impressive. Far from all code can  be rewritten like this though.

Another important bottleneck on asynchronous operations is database operations. Django, as opposed to Twisted, is not an asynchronous framework. Accessing the database is a blocking operation and can be potentially expensive. It was never extremely bad in testing, to be honest. But for large database operations (e.g. many Players) database access was a noticeable effect.

I have read of some people using Twisted's deferToThread to do database writes. The idea sounds reasonable - just offload the operation to another thread and go on your merry way. It did not help us at all though - rather it made things slower. I don't know if this is some sort of overhead (or error) in my test implementation - or an effect of Python just not being ideal with using threading for concurrency (due to the GIL). Either way, certain databases like SQlite3 doesn't support multiple threads very well anyway, and we prefer to keep giving plenty of options with that. So no deferToThread for database writes. I also did a little testing with parallel processes but found that even slower, at least once the number of writes started to pile up (we will offer easy process-pool offloading for other reasons though).

As many have found out before us, caching is king here. There is not so much to do about writes, but at least in our case the database is more often read than written to. Caching data and accessing the cache instead of accessing a field is doing much for performance, sometimes a lot. Database access is always going to cost, but it does not dominate the profile. We are now at a point where one of the most expensive single operations a Player (even a Builder) performs during an entire gaming session is the hashing of their password during login. I'd say that's good enough for our use case anyway.

Django + MUD?

It's interesting that whereas Twisted is a pretty natural fit for a Python MUD (I have learned that Twisted was in fact first intended for mudding, long ago), many tend to be intrigued and/or surprised about our use of Django. In the end these are only behind-the-scenes details though. The actual game designer using Evennia don't really see any of this. They don't really need to know neither Django nor Twisted to code their own dream MUD. It's possible the combination fits less for some projects than for others. But at least in our case it has just helped us to offer more features faster and with less headaches.