Evennia changes to BSD license

As of today, Evennia changes to use the very permissive BSD license.

Now, our previous "Artistic License" was also very friendly. One main feature was that it made sure that changes people made to the core Evennia library (i.e. not the game-specific files) were also made available for possible inclusion upstream. A good notion perhaps, but the licensing text was also quite long and it was clear some newcomers parsed it as more restrictive than it actually was.

... And let's be honest, it's not like I would have come hunting down anyone not complying fully with the Artistic license's terms. Changing to the much simpler and more well-known BSD license better clarifies the actual licensing situation.

After all, far too many older MUD-code bases are weighted by a legacy of licensing issues. Anything we can do to avoid this is better in the long run. Indeed we hope this change in licensing will remove eventual licensing doubts for new adopters and have more people join and contribute to the project.

Community interest

It's fun to see a growing level of activity in the Evennia community. The last few months have seen an increase in the number of people showing up in our IRC channel and mailing list. With this has come a slew of interesting ideas, projects and MUD-related discussion (as well as a few inevitable excursions into interesting but very non-mud-related territory - sorry to those reading our chat logs).

One sign of more people starting to actually use Evennia "for real" is how the number of bugs/feature requests have been increasing. These are sometimes issues related to new things being implemented but also features that have been there for some time and which people are now wanting to use in new and creative ways - or systems which noone has yet really used "seriously" before. This is very encouraging, especially since a lot of good alternative solutions, variations and edge cases can be ironed out this way. So keep submitting those Issues, people!


The budding Evennia community consists of people with a wide variety of interests, skillset and ambition.

There are quite a few people who sees Evennia as a great stepping stone for learning Python, or for getting experience with creating a bigger programming project in general. Some are skilled programmers in other languages but we also have a few with only limited prior coding experience. From the experience in chat, it's really quite striking how fast members pick up the ropes. I'd like to think our documentation is at least partially helping here, but of course it helps that Python is inherently very easy a language to learn and use in the first place.
Not all are participating with the goal of building a specific game. The general flow of patches and clone repository merges have also picked up. We have some users which are primarily interested in a coding challenge, to help with fixing bugs and features, or which uses Evennia as a starting point for exploring various web- and technical solutions that may or may not be a part of Evennia in the future.


The proposed Evennia game projects are just as varied as its users - and none are yet open to the public. As is common with these things, it's of course hard to determine who actually has the time and discipline to bring their plans to fruition. But I should really start to keep some sort of record of who works on what, I'm terrible with remembering this stuff ... so below is just some sort of summary of my impressions, not a comprehensive listing.

As can be expected, most proposed Evennia projects concern relatively standard MUD-style games. A few people are into building traditional hack-and-slash variety games, but most want to expand on the concept considerably. There was even one user exploring using Evennia for a RobotWars kind of experience (where you "program" robot programs in a custom language and battle them). Another project (Avaloria, also blogging on the MUD-dev rss feed) aims for a sort of base-building/strategy mechanic combined with more traditional MUD elements. There are at least two zombie-survival concepts floating around and a few large-scale procedural-content-driven science-fiction text games. One user has apparently a working Smaug->Evennia importer.

It seems that most Evennia users want to offer some sort of roleplaying environment, or at least a "roleplay-friendly" one. Currently we have at least two MUCK admins who aim to convert their existing, running games to Evennia. Whereas the initial idea was to implement parsers for MUCK's MUF language, it seems the conclusion has now shifted to it being faster and easier to just rewrite the MUF-coded functionality in Python (and maybe use something like Evlang for player scripting instead). Several people have announced their interest in creating "RPI"-style games (Armageddon seems to be a big inspiration here), but there was also a MOO admin and even a writer of Interactive Fiction who dropped into the mailing list to see if Evennia could be used for their style of game.

How many of these projects actually reach a point of maturity remains to be seen. But that people are wanting to use the system and is really putting it through its paces is encouraging and very helpful for general Evennia development.

Combining Twisted and Django

Franco Nero as the twisted gunslinger Django

Newcomers to Evennia sometimes misunderstand it as being a "Django mud codebase somehow using Twisted". The correct description is rather that Evennia is a "Twisted-based mud server using Django". Allow me to elaborate.

A mud/mux/moo/mu* is per definition a multi-user online game system. All these users need to co-exist on the server. If one player does something, other players shouldn't have to (noticeably) wait for that something to end before they can do anything. Furthermore it's important for the database schema to be easy to handle and upgrade. Finally, in a modern game, internet presence and web browser access is becoming a must. We combine two frameworks to achieve this.

Two frameworks combined

Twisted is a asynchronous Python framework. "Asynchronous" in this context means, very simplified, that Twisted chops up code execution into as small bits as the code lets it. It then flips through these snippets rapidly, executing each in turn. The result is the illusion of everything happening at the same time. The asynchronous operation is the basis for the framework, but it also helps that twisted makes it easy to support (and create) a massive range of different network protocols.

Django implements a very nice abstract Python API for accessing a variety of SQL-like databases. It makes it very convenient to maintain the database schema (not to mention that django-South gives us easy database migrations). The fact that Django is really a web framework also makes it easy to offer various web features. There is for example an "admin site" that comes with Django. It allows to modify the database graphically (in Evennia's case the admin site is not quite as polished as we would like yet, but it's coming).

Here are some highlights of our architecture:

• Portal - This is a stand-alone Twisted process talking to the outside world. It implements a range of communication protocols, such as telnet (traditional in MUD-world), ssh, ssl, a comet webclient and others. It is an auto-connecting client to Server (below).
• Server - This is the main MUD server. This twisted server handles everything related to the MUD world. It accesses and updates the database through Django models. It makes the world tick. Since all Players connect to the Server through the Portal's AMP connection, it means Server can be restarted without any players getting kicked off the game (they will re-sync from Portal as soon as Server is back up again).
• Webserver - Evennia optionally starts its own Twisted webserver. This serves the game's website (using the same database as the game for showing game statistics, for example). The website is of course a full Django project, with all the possibilities that entails. The Django admin site allows for modifying the database via a graphical interface. 
• Webclient - There is a stand-alone MUD web client existing in a page on the default website. This uses Twisted to implement a long-polling ("comet") connection to a javascript client. As far as Evennia's concerned, this is just another outgoing protocol served by the Portal.
• Other protocols - Since it's easy to add new connectivity, Evennia also offers a bunch of other connectivity options, such as relaying in-game channels to IRC and IMC2 as well as RSS feeds and some other goodies. 

On the joining of the two

An important thing to note about Twisted's asynchronous model is that there is no magic at work here: Each little snippet of code Twisted loops over is blocking. It's just hopefully not blocking long enough for you to notice. So if you were to put sleep(10) in one of those snippets, then congratulations, you just froze the entire server for ten seconds.

Profiling becomes very important here. Evennia's main launcher takes command arguments to run either of its processes under Python's cProfile module. It also offers the ability to connect any number of dummy Players doing all sorts of automated random actions on the server. Such profile data is invaluable to know what is a bottleneck and what is not.

I never found Twisted asynchronous paradigms much harder to understand than other code. But there are sure ways to write stupid blocking code that will come back and bite you. For example, much of  Evennia's workload is spent in the Server, most notably in its command handler. This is not so strange; the command handler takes care of parsing and executing all input coming from Players, often modifying the game world in various ways (see my previous post for more info about the command handler).
The command handler used to be a monolithic, single method. This meant that Twisted had to let it run its full course before letting anyone else do their turn. Using Twisted's inlineCallbacks instead allowed for yielding at many, many places in this method, giving Twisted ample possibilities to split execution. The effect on multi-user performance was quite impressive. Far from all code can  be rewritten like this though.

Another important bottleneck on asynchronous operations is database operations. Django, as opposed to Twisted, is not an asynchronous framework. Accessing the database is a blocking operation and can be potentially expensive. It was never extremely bad in testing, to be honest. But for large database operations (e.g. many Players) database access was a noticeable effect.

I have read of some people using Twisted's deferToThread to do database writes. The idea sounds reasonable - just offload the operation to another thread and go on your merry way. It did not help us at all though - rather it made things slower. I don't know if this is some sort of overhead (or error) in my test implementation - or an effect of Python just not being ideal with using threading for concurrency (due to the GIL). Either way, certain databases like SQlite3 doesn't support multiple threads very well anyway, and we prefer to keep giving plenty of options with that. So no deferToThread for database writes. I also did a little testing with parallel processes but found that even slower, at least once the number of writes started to pile up (we will offer easy process-pool offloading for other reasons though).

As many have found out before us, caching is king here. There is not so much to do about writes, but at least in our case the database is more often read than written to. Caching data and accessing the cache instead of accessing a field is doing much for performance, sometimes a lot. Database access is always going to cost, but it does not dominate the profile. We are now at a point where one of the most expensive single operations a Player (even a Builder) performs during an entire gaming session is the hashing of their password during login. I'd say that's good enough for our use case anyway.

Django + MUD?

It's interesting that whereas Twisted is a pretty natural fit for a Python MUD (I have learned that Twisted was in fact first intended for mudding, long ago), many tend to be intrigued and/or surprised about our use of Django. In the end these are only behind-the-scenes details though. The actual game designer using Evennia don't really see any of this. They don't really need to know neither Django nor Twisted to code their own dream MUD. It's possible the combination fits less for some projects than for others. But at least in our case it has just helped us to offer more features faster and with less headaches. 

Taking command

Commands are the bread and butter of any game. Commands are the instructions coming in from the player telling the game (or their avatar in the game) to do stuff. This post will outline the reasoning leading up to Evennia's somewhat (I think) non-standard way of handling commands.

In the case of MUDs and other text games commands usually come in the form of entered text. But clicking on a graphical button or using a joystick is also at some level issuing a command - one way or another the Player instructs the game in a way it understands. In this post I will stick to text commands though. So open door with red key is a potential command.

Evennia, being a MUD design system, needs to offer a stable and extensive way to handle new and old commands.  More than that, we need to allow developers pretty big freedom with developing their own command syntax if they so please (our default is not for everyone). A small hard-coded command set is not an option.

Identifying the command

First step is identifying the command coming in. When looking at open door with red key it's probably open that is the unique command. The other words are "options" to the command, stuff the open command supposedly knows what to do with. If you know already at this stage exactly how the command syntax looks, you could hard-code the parsing already here. In Evennia's case that's not possible though - we aim to let people define their command syntax as freely as possible. Our identifier actually requires no more than that the uniquely identifying command word (or words) appear first on the input line. It is hard to picture a command syntax where this isn't true ... but if so people may freely plug in their own identifyer routine.

So the identifyer digs out the open command and sends it its options ... but what kind of code object is open?

 

 The way to define the command

A common variant I've seen in various Python codebases is to implement commands as functions. A function maps intuitively to a command - it can take arguments and it does stuff in return. It is probably more than enough for some types of games.

Evennia chooses to let the command be defined as a class instead. There are a few reasons. Most predominantly, classes can inherit and require less boiler plate (there are a few more reasons that has to do with storing the results of a command between calls, but that's not as commonly useful). Each Evennia command class has two primary methods:

• parse() - this is responsible for parsing and splitting up the options part of the command into easy-to use chunks. In the case of open door with red key, it could be as simple as splitting the options into a list of strings. But this may potentially be more complex. A mux-like command, for exampe, takes /switches to control its functionality. They also have a recurring syntax using the '=' character to set properties. These components could maybe be parsed into a list switches and two parameters lhs and rhs holding the left- and right hand side of the equation sign. 
• func() - this takes the chunks of pre-parsed input and actually does stuff with it. 

One of of the good things with executing class instances is that neither of these methods need to have any arguments or returns. They just store the data on their object (self.switches) and the next method can just access them as it pleases. Same is true when the command system instantiates the command. It will set a few useful properties on the command for the programmer to make use of in their code (self.caller always references the one executing the command, for example). This shortcut may sound like a minor thing, but for developers using Evennia to create countless custom commands for their game, it's really very nice to not have to have all the input/output boilerplate to remember. 

... And of course, class objects support inheritance. In Evennia's default command set the parse() function is  only implemented once, all handling all possible permutations of the syntax. Other commands just inherit from it and only needs to implement func(). Some advanced build commands just use a parent with an overloaded and slightly expanded parse().

 Commands in States

So we have individual commands. Just as important is how we now group and access them. The most common way to do this (also used in an older version of Evennia) is to use a simple global list. Whenever a player enters a command, the identifier looks the command up in the list. Every player has access to this list (admin commands check permissions before running). It seems this is what is used in a large amount of code bases and thus obviously works well for many types of games. Where it starts to crack is when it comes to game states.

• A first example is an in-game menu. Selecting a menu item means an instruction from the player - i.e. a command. A menu could have numbered options but it might also have named options that vary from menu node to menu node. Each of these are a command name that must be identified by the parser. Should you make all those possible commands globally available to your players at all times? Or do you hide them somehow until the player actually is in a menu? Or do you bypass the command system entirely and write new code only for handling menus...?
• Second example: Picture this scenario: You are walking down a dark hallway, torch in hand. Suddenly your light goes out and you are thrown into darkness. You cannot see anything now, not even to look in your own backpack. How would you handle this in code? Trivially you can put if statements in your look and inventory commands. They check for the "dark" flag. Fair enough. Next you knock your head and goes 'dizzy'. Suddenly your "navigation" skill is gone and your movement commands may randomly be turned around. Dizziness combined with darkness means your inventory command now returns a strange confused mess. Next you get into a fight ... the number of if statements starts piling up.  
• Last example: In the hypothetical FishingMUD,. you have lots of detailed skills for fishing. But different types of fishing rods makes different types of throws (commands) available. Also, they all work differently if you are on a shore as compared to being on a boat. Again, lots of if statements. It's all possible to do, but the problem is maintenance; your command body keep growing to handle edge cases. Especially in a MUD, where new features tend to be added gradually over the course of years, this gives lots of possibilities for regressions.

All of these are examples of situation-dependent (or object-dependent) commands. Let's jointly call them state-dependent commands. You could picture handling the in-game menu by somehow dynamically changing the global list of commands available. But then the global bit becomes problematic - not all players are in the same menu at the same time. So you'll then have to start to track who has which list of commands available to them. And what happens when a state ends? How do you get back to the previous state - a state which may itself be different from the "default" state (like clearing your dizzy state while still being in darkness)? This means you have to track the previous few states and ...

A few iterations of such thinking lead to what Evennia now uses: a non-global command set system. A command set (cmdset) is a structure that looks pretty much like a mathematical set. It can contain any number of (unique) command objects, and a particular command can occur in any number of command sets.

• A cmdset stored on an object makes all commands in that cmdset available to the object. So all player characters in the game has a "default cmdset" stored on them with all the common commands like look, get and so on.
• Optionally, an object can make its cmdset available to other objects in the same location instead. This allows for commands only applicable with a given object or location, such as wind up grandfather clock. Or the various commands of different types of fishing rods. 
• Cmdsets can be non-destructively combined and merged like mathematical sets, using operations like "Union", "Intersect" and a few other cmdset-special operations. Each cmdset can have priorities and exceptions to the various operations applied to them. Removing a set from the mix will dynamically rebuild the remaining sets into a new mixed set.

The last point is the most interesting aspect of cmdsets. The ability to merge cmdsets allows you to develop your game states in isolation. You then just merge them in dynamically whenever the game state changes. So to implement the dark example above, you would define two types of "look" (the dark version probably being a child of the normal version). Normally you use your "default cmdset" containing the normal look. But once you end up in a dark room the system (or more likely the room) "merges" the dark cmdset with the default one on the player, replacing same-named commands with new ones. The dark cmdset contains the commands that are different (or new) to the dark condition - such as the look command and the changed inventory command.  Becoming dazed just means yet another merger - merging the dazed set on top of the other two. Since all merges are non-destructive, you can later remove either of the sets to rebuild a new "combined" set only involving the remaining ones in any combination. 

Similarly, the menu becomes very simple to create in isolation (in Evennia it's actually an optional contrib). All it needs to do is define the required menu-commands in its own cmdset. Whenever someone triggers the menu, that cmdset is loaded onto the player. All relevant commands are then made available. Once the menu is exited, the menu-cmdset is simply removed and the player automatically returns to whichever state he or she was in before.

Final words

The combination of commands-as-classes and command sets has proved to very flexible. It's not as easy to conceptualize as is the simple functions in a list, but so far it seems people are not having too much trouble. I also think it makes it pretty easy to both create and, importantly, expand a game with interesting new forms of gameplay without drastically rewriting old systems.

Extending time and details

For the fun of it I added an "Extended Room" contrib to Evennia the other night.

("Contribs" are optional code snippets and systems that are not part of the actual codebase. They are intended for you to use or dissect as you like in your game development efforts).

The ExtendedRoom is a room typeclass meant to showcase some more advanced features than the default one. Its functionality is by all means nothing revolutionary in MUD-world, but it was fun and very simple to do using only Evennia's basic building blocks - the whole thing took me some two hours to code, document and clean up for a contrib push. The "ExtendedRoom" contribution has the following features:

• Season-based descriptions. The new Room typeclass will change its overall description based on the time of year (the contrib supports the four seasons, you can hack this as you please). It's interesting from an implementation point of view since it doesn't require any Script or ticker at all - it just checks on-demand, whenever it is being looked at, only updating if the season has actually changed. There is also a general description used as a fallback in case of a missing seasonal one.
• Time-of-day-based descriptions. Within each Season-based description you can embed time-of-day based ones with special tags. The contrib supports four time slots out of the box (morning, afternoon, evening, night). In the description, you just embed time-dependent text within tags, like <morning>Morning sunlight is shining through the windows</morning>. Only time-relevant tags will be shown. This is a simple regular expression substitution, should be easy to expand on if one wants more fine-grained time slots.
• Details. I took the inspiration of these from a MOO tutorial I read a long time ago. "Details" are "virtual" look-targets in the room. It allows you to add visual interest without having to add a bunch of actual objects for that purpose. Details are simply stored in a dictionary on the room. Details don't change with Season in this implementation, but they are parsed for time-of-day based tags!
• Custom commands. The room is supported by extending two of the custom commands. The Details require a slightly modified version of the look command. There is also a new @desc for setting/listing details and seasonal descriptions. The new time command, finally, simply shows the current game time and season in the room.

Installing and testing the snippet is simple - just add the new commands to the default cmdset (they will dynamically replace the same-named default ones), dig a few rooms of the new typeclass and play around! Especially the details do make building interesting rooms a lot more fun (I got hung up playing with them way too long last night).

Coding from the inside

Some time ago, a message on the Evennia mailing list asked about "softcode" support in Evennia. Softcode, a defacto standard in the MUX/MUCK/MUSH/MOO world, is conceptually a "safe" in-game scripting language that allows Players to extend the functionality of things without having access to the server source.

Now, Evennia is meant to be extended by normal Python modules. For coding game systems and advanced stuff, there is really no reason (in my opinion) for a small development team to not use a modern version control system and proper text editors rather than entering things on a command line without formatting.

But there is a potential fun aspect of having an online scripting language - and that is player content creation. Crafters wanting to add some pizazz to their objects, builders getting an extra venue of creativity with their rooms - that kind of thing. I didn't plan to add softcode support to Evennia, but it "sounded like an interesting problem" and one thing led to another.

Python is of course an excellent scripting language from the start. Problem is that it's notoriously tricky to make it run safely with untrusted code - like that inserted by careless or even potentially malignant Players. Scanning the Internet on this topic is a pretty intimidating experience - everywhere you hear that it shouldn't be done, and that the various suggested solutions of a "sandbox" are all inherently unsafe. Python's awesome introspection utilities is its own enemy in this particular case.

For Evennia we are however not looking for a full sandbox. We want a Python-like way for Players to influence a few determined systems. Moreover, we expect short, simple scripts that can do without most of Python's functionality (since our policy is that if it's too complex or large, it belongs in an external Python module). We could supply black-box "safe" functions to hide away complex functionality while still letting people change things we expect them to want to script. This willingness to accept heavy restrictions to the language should work to our advantage, I hope.

Evennia actually already has a safe "mini-language" in the form its "lock system", and thus it was a natural way for me to start looking. A "lock string" has a very restricted syntax - it's basically function calls optionally separated by boolean operators, like this:

lockfunc1(*args) and lockfunc(*args, **kwargs) and not lockfunc2()

The result of this evaluation will always be a boolean True/False (if the lock is passed or not). Only certain functions are available to use (controlled by the coder). The interesting thing is that this string can be supplied by the Player, but it is not evaluated - rather it's manually parsed, from left to right. The function names and arguments are identified (as for the rest, only and/or/not are allowed). The functions are then called explicitly (in Python code, not evaluated as a string) and combined to get a result. This should be perfectly safe as long as your functions are well-defined.


For the potential softcode language, I first took this hands-on approach - manually parsing the string into its components. I got a pretty decent demo going, but the possibilities are much larger than in the simple lockstring case. Just parsing would be one thing, but then to also make sure that each part is okay to use too is another matter ... It would probably be doable, but then I got to supplying some sort of flow-control. The code starts to become littered with special cases which is never a good sign.

So eventually I drifted off from the "lock-like" approach and looked into Python's ast module. This allows you to view Python code as an "abstract syntax tree" (AST).  This solves the parsing issues but once you start dealing with the AST tree you are sort of looking at the problem from the other end - rather than parsing and building the script from scratch it more becomes a matter of removing what is already there (an AST tree can be compiled directly back into Python code after all). It nevertheless seemed like the best way forward.

Testing a few different recipes from the web, I eventually settled on an approach which (with some modifications compared to the original) uses a whitelist (and a blacklist for some other things) to only allow a given set of ast nodes and items in the execution environment. It walks the AST tree before execution and kills dangerous Python structures in one large swath. I expanded on this a fair bit, cutting away a lot of Python functionality for our particular use case. Stuff like attribute acces and assignments, while loops and many other Pythonesque things went out the window.

Around this highly stunted execution system I then built the Evennia in-game scripting system. This includes in-game commands as well as scriptable objects with suitable slots defining certain functionality the Player might want to change. Each Evennia developer can also supply any set of "safe" blackbox functions to offer more functionality to their Player-coders.

A drawback is the lack of a functional timeout watchdog in case of a script running too long. I'm using Twisted's deferToThread to make sure the code blocks as little as possible, but alas, whereas I can check timeouts just fine,  the problem lies in reliably killing the errant sub-thread. Internet experts suggest this to be tricky to do safely at the best of times (with threads running arbitrary code at least), and not wanting to kill the Twisted server is not helping things. I pondered using Twisted's subprocess support, but haven't gotten further into that at this point. Fact is that most of the obvious DOS attack vectors (such as the while loop and huge powers etc) are completely disabled, so it should hopefully not be trivial to DOS the system (famous last words).

I've tentatively dubbed the softcode system "Evlang" to differentiate it from our normal database-related "Scripts".
So is Evlang "safe" to use by untrusted evil Players? Well, suffice to say I'm putting it up with a huge EXPERIMENTAL flag, with plenty of warnings and mentions of "on your own risk". Running Evennia in a chroot jail and with minimum permissions is probably to recommend for the security paranoid. Hopefully Evennia coders will try all sorts of nasty stuff with it in the future and report their finding in our Issue tracker!

But implementation details aside, I must admit it's cool to be able to add custom code like this - the creative possibilities really do open up. And Python - even a stunted version of it - is really very nice to work with, also from inside the game.

Dummies doing (even more) dummy things

This is a follow-up to the Dummies doing dummy things post. I originally posted info about this update on the mailing list some time back, but it has been pointed out to me that it might be a nice thing to put on the dev blog too since it's well, related to development!

I have been at it with further profiling in Evennia. Notably even more aggressive on-demand caching of objects as well as on-object attributes. I found from profiling that there was an issue with how object access checks were done - they caused the lock handler to hit the database every lock check as it retrieved the needed attributes.

Whereas this was not much of a hit per call, access checks are done all the time, for commands, objects, scripts, well everything that might need restricted access.
After caching also attributes, there is no need to hit the database as often. Some commands, such as listing all command help entries do see this effect (although you still probably wouldn't notice it unless you checked before and after like I did). More importantly, under the hood I'm happy to see that the profile for normal Evennia usage is no longer dominated by Django db calls but by the functional python code in each command - that is, in code that the end user have full control over anyway. I'd say this is a good state of affairs for a mud creation system.

 
In the previous "Dummies ..." post I ran tests with rather extreme conditions - I had dummy clients logging to basically act like heavy builders.  They dug rooms, created and defined objects randomly every five seconds (as well as walking around, reading help files, examining objects and other spurious things). In that post I found that my puny laptop could handle about 75 to 100 such builders at a time without me seeing a slowdown when playing. My old but more powerful desktop could handle some 200 or so.

Now, I didn't re-run these build-heavy tests with the new caches in place. I imagine the numbers will improve a bit, but it's just a guess. By all means, if you expect regularly having more than 100 builders on your game continuously creating 250 new rooms/objects per minute, do get back to me ...

... Instead I ran similar tests with more "normal" client usage. That is, I connected dummy clients that do what most players would do - they walk around, look at stuff, read help files and so on. I connected clients in batches of 100 at a time, letting them create accounts and logging in fully before connecting the next set of 100.

All in all I added 1000 dummy clients this way before I saw a noticeable lag on my small laptop. I didn't find it necessary to try the desktop at this point. Whereas this of course was with a vanilla Evennia install, I'd say it should be reasonable room for most realistic mud concepts to grow in.

With the rather extensive caching going on, it is interesting to know what the memory consumption is.
 
This graph shows memory info I noted down after adding each block of 100 players. The numbers fluctuated up and down a bit between readings (especially what the OS reported as total usage), which is why the lines are not perfectly straight.

In the end the database holds 1000 players (which also means there are 1000 Character objects), about as many rooms and about twice as many attributes.  The "idmapper cache" is the mapper that makes sure all Django model instances retain their references between accesses (as opposed to normal Django were you can never be sure of this). "Attribute cache" is a cache storing the attribute objects themselves on the Objects, to avoid an extra database lookup. All in all we see that keeping the entire database in memory takes about 450MB.

Evennia's caching is on-demand (so e.g. a room would not be loaded/cached until someone actually accessed it somehow). One could in principle run a script to clean all cached regularly if one was short on RAM - time will tell if this is something any user needs to worry about on modern hardware.

Shortcuts to goodness

Evennia, being a MUD-design system, needs to take some special considerations with its source code - its sole purpose is after all to be read, understood and extended.
Python is of course very readable by default and we have worked hard to give extensive comments and documentation. But for a new user looking into the code for the first time, it's still a lot of stuff to take in. Evennia consists of a set of Django-style "applications" interacting and in some cases inheriting from each other so as to avoid code duplication. For a new user to get an overview could therefore mean diving into more layers of code than one would like.

I have now gone through the process of making Evennia's API (Application Programming Interface) "flatter". This has meant exposing some of the most commonly used methods and classes at a higher level and fully documenting exactly what they inherit av every layer one looks at. But I have also added a new module ev.py to the root directory. It  implements "shortcuts" to all the most commonly used parts of the system, forming a very flat API. This means that what used to be

      from src.objects.objects import Object

can now be done as 

      from ev import Object

Not only should it be easier to find things (and less boilerplate code to write) but I like that one can also easier explore Evennia interactively this way.  Using a Python interpreter (I recommend ipython) you can just import ev and easily inspect all the important object classes, tab to their properties, helper functions and read their extensive doc strings.

Creating this API, i.e. going through and identifying all the useful entry points a developer will need, was also interesting in that it shows how small the API really is. Most of the ev interface is really various search functions and convenient options to inspect the database in various ways. The MUD-specific parts of the API is really lean, as befits a barebones MUD server/creation system.

Dummies doing dummy things

It can often be interesting to test hypothetical situations. So the other day I did a little stress test to see how Evennia handles many players. This is a follow-up to the open bottlenecks post.

Evennia, being based on Twisted, is an asynchronous mud server. This means that the program is not multi-threaded but instead it tries to divide up the code it runs into smaller parts. It then quickly switches between executing these parts in turn, giving the impression of doing many things at the same time. There is nothing magical about this - each little piece of code blocks the queue as it runs. So if a particular part takes too long, everyone else will have to wait for their turn. Likewise, if Twisted cannot flip through the queue as fast or faster than new things get added to it, it will start to fall behind.

The good thing is that all code in the queue will run, although if the event loop cannot keep up there will be a noticeable delay before it does. In a mud, this results in "lagging" (in addition to whatever lag is introduced by your network connection). Running Evennia with a handful of users shows no effect of this, but what happens when we start to add more and more players?

My "dummy runner" is a stand alone Twisted program that opens any number of Telnet connections to the Evennia server. Each such "dummy" client is meant to represent a player connecting. In order to mimic actual usage, a dummy client can perform a range of actions:

• They can create a new account and log in
• They can look around and read random help files
• They can create objects, name and describe them (for testing, the server is set to grant build privileges to all new players)
• They can examine objects, rooms and themselves 
• They can dig new rooms, naming all exits and giving aliases
• They can move between rooms

The clients tick every 5 seconds, at which time there is a 10% chance each will perform an action from the list above (login first, then one of the others at random).  This is meant to spread out the usage a bit, like it would be with real people. Some of these actions actually consist of multiple commands sent to the server (create + describe + set etc), possibly simulating people using shortcuts in their client to send several commands at the same time.

Results

Note that I didn't do a proper objective benchmark. Rather, I logged in and ran commands to see, very subjectively, how the game "felt" with a number of different players. The lag times are rough estimates by putting time() printouts in the server.

First I tried with my development laptop, a Thinkpad X61s. It's about 5 years old by now and certainly not the fastest thing out there, mainly it's small and thin and has great battery life. I used a MySQL database.

• 1-50 dummy players didn't show any real difference from playing alone. It's not so strange; 50 players meant on average 5 of them would do an action every 5 seconds. My laptop could handle that just fine.
• 50-75 dummy players introduced a brief lag (less than a second) when they logged in. 5-7 players logging in at exactly the same time will after all mean a lot of commands sent to the server (they not only log in, they create a new character at the same time). Throughout these tests, clients logging in showed the greatest effect on lag. I think this is mostly an artifact of how the clients operate by sending all relevant login commands at the same time. Once all were logged in, only brief lag occurred at random times as coincidence had all clients do something expensive at the same time.
• 75-100 dummy players introduced longer lags during login, as on average 7-10 clients were now logging in at exactly the same time. Most commands were unaffected, but occasionally there were some noticeable "hiccups" of about a second depending on what the clients were doing.
• 100-150 dummy players - here things started to go downhill rapidly. Dummy client login caused seriously lagging and at 150 logged in players, there were 15 exactly simultaneous actions coming in. This was clearly too much for my laptop; it lead to a very varying lag of 1-10 seconds also for simple commands.

Next I tried my desktop machine. This is also not bleeding edge; it's a 4-year old machine with 3GHz processor and 4GB RAM. Since I don't have MySQL on this machine, I used SQLite3, which is interesting in its own right since it's Evennia's default database.

• 1-75 dummy players didn't affect the feel of the game one bit.
• 75-100 showed some occasional effects starting to appear during dummy client logins. 
• 100-150 dummy players didn't introduce more than a few hiccups of less than a second when approximately 15 clients decided to do something expensive at the same time.
• 150-200 introduces 2-3 seconds lag during client mass-logins, but once all clients had connected, things worked nicely with only brief random hiccups.
• 200-250 showed the lag getting more uneven, varying from no lag at all to 2-3 seconds for normal times and up to 5 seconds when clients were logging in.
• 250-300 dummy players showed login lag getting worse. The general lag varied from 0-5 seconds depending on what the other clients were up to.
• 300-350 dummy players, about double of what the laptop could handle,  the CPU was not able to keep up anymore. The system remained stable and all commands got executed - eventually. But the lag times went up very rapidly.

Conclusions

So, based on this I would say 50-75 was a comfortable maximum number of (dummy) players to have on my laptop whereas the desktop could handle around 150 without much effort, maybe up to 250 on a good day.

So what does these numbers mean? Well, the dummy clients are rather extreme, and 100+ builders walking around building stuff every 5 seconds is not something one will see in a game. Also so many logging on at the same time is not very likely (although it could happen after a crash or similar). If anything the usage pattern of human players will be more random and spread out, which helps the server to catch up on its event queue.

On the other hand these tests were run with a vanilla Evennia install - a full game might introduce more expensive commands and subsystems. Human players may also introduce random spikes of usage. So take this for what it is - a most subjective, un-scientific and back-of-the-envelope measure.

All in all though, I would say that few MUDs these days see 30 concurrent online users, even less 100 ...

Commands and you

Commands define how a Player interacts with a given game. In a text-based game it's not amiss to say that the available commands are paramount to the user experience. In principle commands could represent mouse clicks and other modernistic GUI sugar - but for this blog I'll stick with the traditional entered text.

Like most things in Evennia, Commands are Python classes. If you read the documentation about them you'll find that the command classes are clumped together and tacked onto all objects in-game. Commands hanging onto a Character object will be available all the time, whereas commands tacked onto a grandfather's clock will only be available to you when you stand in front of said clock.

The interesting thing with Commands being classes is that each Character gets a separate instance of each command. So when you do look 100 times in a row, it's always the same Look command instance that has its methods called by the engine. Apart from being an efficient way to handle things, this has a major neat side-effect:

You can store things on the Command object and whatever you store can be retrieved next time you execute that command.

I find this very cool mainly because I didn't really plan with this in mind when designing the command system - it was a happy side effect. A use I have thought of is to implement cooldowns. Say you have a powerful bash command. It makes a lot of damage, but you need time to recover between bashes. So when you do the bash  command the Bash command object simply stores the current time on itself:

self.last_bash = time.time()

Next time the Player tries to use bash, all the command object needs to is to check if self.last_bash is set, and compare the time stored herein with the current time. No twisted tasks needed, no overhead. Very neat and tidy.

Another nice functionality (just added today in fact) is that Evennia can be set to store a copy of the last command object executed. What can one do with this? For starters, it allows for commands to check what a previous command was. This can be useful in itself, but since the next command actually have access to (a copy of) the previous command object itself, it will allow a lot more than that.

Consider a look command that remembers whatever object it is looking at. Since the Look command is a Python object, it simply stores the looked-at object on itself before returning the normal info to the Player. Next, let's assume we use a get command. If no argument is given to this get (no given object to pick up), the get normally returns an error. But it can now instead peek at the previous command (look) and see what that command operated on. This allows for nice potential constructs like

>> look [at] box 

>> get [it]

Evennia does not use this functionality in its default command set, but it offers some very powerful possibilities for MUD creators to design clever parsing schemes.

Such a small thing ...

Lately I went back to clean up and optimize the workings of Evennia's Attributes. I had a nice idea for making the code easier to read and also faster by caching more aggressively. The end result was of course that I managed to break things. In the end it took me two weeks to get my new scheme to a state where it did what it already did before (although faster).

Doing so, some of the trickier aspects of implementing easily accessible Attributes came back into view, and I thought I'd cover them here. Python intricacies and black magic to follow. You have been warned. 

Attributes are, in Evennia lingo, arbitrary things a user may want to permanently store on an object, script or player. It could be numbers or strings like health, mana or short descriptions, but also more advanced stuff like lists, dictionaries or custom Python objects.

Now, Evennia allows this syntax for defining an attribute on e.g. the object myobj:

myobj.db.test = [1,2,3,4]

This very Pythonic-looking thing allows a user to transparently save that list (or whatever) to an attribute named, in this example, test. This will save to the database.
 What happens is that db, which is a special object defined on all Evennia objects, takes all attributes on itself and saves them by overloading its __setattr__ default method (you can actually skip writing db most of the time, and just use this like you would any Python attribute, but that's another story).

Vice-versa,

value = myobj.db.test

This makes use of the db object's custom __get_attribute__ method behind the scenes. The test attribute is transparently retrieved from the database (or cache) for you.

Now, the (that is, my) headache comes when you try to do this:

myobj.db.test[3] = 5

Such a small, normal thing to do! Looks simple, right? It is actually trickier than it looks to allow for this basic functionality.
The problem is that Python do everything by reference. The list is a separate object and has no idea it is connected to db. db's __get_attribute__ is called, and happily hands over the list test. And then db is out of the picture!. My nifty save-to-database feature (which sits in db) knows nothing about how the 3rd index of the list test now has a 5 instead of a 4.

Now, of course, you could always do this:

temp = myobj.db.test

temp[3] = 5

myobj.db.test = temp

This will work fine. It is however also clumsy and hardly intuitive. The only solution I have been able to come up with is to have db return something which is almost a list but not quite. It's in fact returning an object I not-so-imaginatively named a PackedList. This object works just like a list, except all modifying methods on it makes sure to save the result to the database. So for example, what is called when you do mylist[3] = 4 is a method on the list named __setitem__. I overload this, lets it do its usual thing, then call the save.

myobj.db.test[3] = 5

now works fine, since test is in fact a PackedList and knows that changes to it should be saved to the database. I do the same for dictionaries and for nested combinations of lists and dictionaries. So all is nice and dandy, right?  Things work just like Python now?

No, unfortunately not. Consider this:

myobj.db.test = [1, 3, 4, [5, 6, 7]]

A list with a list inside it. This is perfectly legal, and you can access all parts of this just fine:

val = myobj.db.test[3][2] # returns 7!

But how about assigning data to that internal nested list?

myobj.db.test[3][2] = 8

We would now expect test to be [1, 3, 4, [5, 6, 8]]. It is not. It is infact only [5, 6, 8]. The inner list has replaced the entire attribute! 

What actually happens here? db returns a nested structure of two PackedLists. All nice and dandy.  But Python thinks they are two separate objects! The main list holds a reference to the internal list, but as far as I know there is no way for the nested list to get the back-reference to the list holding it! As far as the nested list knows, it is all alone in the world, and therefore there is no way to trigger a save in the "parent" list.
 The result is that we update the nested list just fine - and that triggers the save operation to neatly overwrite the main list in the cache and the database.
 
This latter problem is not something I've been able to solve. The only way around it seems to use a temporary variable, assign properly, then save it back, as suggested earlier. I'm thinking this is a fundamental limitation in the way cPython is implemented, but maybe I'm missing some clever hack here (so anyone reading who has a better solution)?

Either way, the db functionality makes for easy coding when saving things to the database, so despite it not working quite like normal Python, I think it's pretty useful.

Evennia's open bottlenecks

Since Evennia hit beta I have been mostly looking behind the scenes to see what can be cleaned up and not in the core server. One way to do that is to check where the bottlenecks are. Since a few commits, Evennia's runner has additions for launching the Python cProfiler at startup. This can be done for both the Portal and the Server separately.

I have created a test runner (soon to hit trunk) that launches dummy clients. They log on and then goes about their way executing commands, creating objects and so on. The result of looking at the profiling data (using e.g. runsnake) has been very interesting.

Firstly, a comparably small part of execution time is actually spent in Evennia modules - most is spent using Python built-ins and in the Twisted/Django libraries. This is promising in a way - at least there are no expensive loops in the Evennia code itself that sucks up cycles. Of course it also means we depend heavily on django/twisted (no surprise there) and especially when it comes to database access, I know there could be more caching going on so as to cut down on db calls. 

Of the Evennia modules, a lot of time is spent in getting properties off objects - this is hard to avoid, it a side effect of Evennia's typeclass system - and this is cached very aggressively already. What also stands out as taking a considerable time is the command system - the merging of command sets in particular does a lot of comparing operations. This happens every time a command is called, so there are things to be done there. The same goes for the help system; it needs to collect all the currently active command sets for the calling player. Maybe this could be cached somehow.

More work on this is needed, but as usual optimization is a double-edged sword. Evennia is written to have a very readable code. Optimization is usually the opposite of readable, so one needs to thread carefully. 

About this dev blog

This is to be my Evennia dev blog, but it will also cover various other musings concerning programming in general and mud designing in particular. Whereas the Evennia mailing list remains the main venue for discussion, I will probably use this blog for announcing features too.

Some background:
Evennia is a Python MUD/MUX/MU* server. More correct is probably to call it a "MUD-building system". The developer codes their entire game using normal Python modules. All development is done with the full power of Python - gritty stuff like database operations and network communication are hidden away behind custom classes that you can treat and modify mostly like any Python primitives.

Since the server is based on the Twisted and Django technologies we can offer many modern features out of the box. Evennia is for example its own web server and comes with both its own website and an "comet"-style browser mud client. But before this turns into even more of a sales pitch, I'll just just direct you to the evennia website if you want to know more. :)

I, Griatch, took over the development of Evennia from the original author, Greg Taylor, in 2010.