The luxury of a creative community

For this blog post I want to focus on the series of very nice pull requests coming in from a growing cadre of contributors over the last few months.

Contributed goodness

People have put in a lot of good work to boost Evennia, both by improving existing things and by adding new features. Thanks a lot everyone (below is just a small selection)!

•  Contrib: Turn-based combat system - this is a full, if intentionally bare-bones implementation of a combat system, meant as a template to put in your particular game system into.
• Contrib: Clothing sytem - a roleplaying mechanic where a character can 'wear' items and have this show in their descriptions. Worn items can also be layered to hide that underneath. Had plenty of opportunities for extensions to a given game.
• Contrib: An 'event' system is in the works, for allowing privileged builders to add dynamic code to objects that fires when particular events happen. The PR is not yet merged but promises the oft pondered feature of in-game coding without using softcode (and notably also without the security of softcode!). 

• A lot of PRs, especially from one user, dealt with cleanup and adherence to PEP8 as well as fixing the 'alerts' given by LGTM on our code (LGTM is by the way a pretty nifty service, they parse the code from the github repo without actually running it and try to find problems. Abiding by their advice results is cleaner code and it also found some actual edge-case bugs here and there not covered by unit tests. The joint effort has brought us down from some 600+ alerts to somewhere around 90 - the remaining ones are alerts which I don't agree with or which are not important enough to spend effort on). 
• The help mechanics of Evennia were improved by splitting up the default help command into smaller parts, making it easier to inject some changes to your help system without completely replacing the default one. 
• Evennia's Xterm256 implementation was not correctly including the additional greyscale colors, those were added with new tags |=a ... |=z.
• Evennia has the ability to relay data to external services through 'bots'. An example of this is the IRC bot, which is a sort of 'player' that sits in an in-game channel and connects that to a counterpart-bot sitting in a remote IRC channel. It allows for direct game-IRC communication, something enjoyed by people in the Evennia demo for many years now. The way the bot was defined used to be pretty hard-coded though. A crafty contributor changed that though, but incorporating the bot mechanism into Evennia's normal message flow. This allows for adding new types of bots or extending existing ones without having to modify Evennia's core. There is already an alternative IRC bot out there that represents everyone in the IRC room as a room full of people in the MUD. 
• Evennia's Attributes is a database table connected to other objects via a ForeignKey relation. This relation is cached on the object. A user however found that for certain implementations, such as using Attributes for large coordinate systems, non-matches (that is failed Attribute lookups on the object) can also be cached and leads to dramatic speed increases for those particular use cases. A PR followed. You live and learn.
• Another contributor helped improve the EvEditor (Evennia's VIM-like in-game text editor) by giving it a code-mode for editing Python code in-game with auto-indents and code execution. Jump into the code mode with the command @py/edit.
• Time scheduling is another feature that has been discussed now and then and has now been added through a PR. This means that rather than specifying 'Do this in 400 seconds' you can say 'do this at 12AM, in-game time'. The core system works with the real-world time units. If you want 10 hours to a day or two weeks to a month the same contributor also made an optional calendar contrib for that!
• A new 'whisper' command was added to the Default cmdset. It's an in-game command for whispering to someone in the same room without other people hearing it. This is a nice thing to have considering Evennia is out-of-the-box pretty much offering the features of a 'talker' type of game.
• Lots of bug fixes big and small!
• Some at_* hooks were added, such as at_give(giver, getter). This allows for finer control of the give process without handling all the logics at the command level. There are others hooks in the works but those will not be added until in Evennia 0.7. 

About that Evennia 0.7 ...

So while PRs are popping up left and right in master I've been working in the devel branch towards what will be the Evennia 0.7 release. The branch is not ready for public consumption and testing yet But tentatively it's about halfway there as I am slowly progressing through the tickets. Most of the upcoming features were covered in the previous blog post so I'll leave it at that.

I just want to end by saying that it's a very luxurious (and awesome) feeling for me to see master-branch Evennia expand with lots of new stuff "without me" so to speak. The power of Open Source indeed!
  

Image from http://maxpixel.freegreatpicture.com, released as public domain.

News items from the new year

The last few months have been mostly occupied with fixing bugs and straightening out usage quirks as more and more people take Evennia through its paces.

Webclient progress

One of our contributors, mewser/titeuf87 has put in work on implementing part of our roadmap for the webclient. In the first merged batch, the client now has an option window for adjusting and saving settings. This is an important first step towards expanding the client's functionality. Other  features is showing help in an (optional) popup window and to report window activity by popup and/or sound.

The goal for the future is to allow the user or developer to split the client window into panes to which they can then direct various output from the server as they please It's early days still but some of the example designs being discussed can be found in the wiki webclient brainstorm (see the title image of this blog for one of the mockups).
 
New server stuff

Last year saw the death of our old demo server on horizondark.com, luckily the new one at silvren.com has worked out fine with no hickups. As part of setting that up, we also got together a more proper list of recommended hosts for Evennia games. Evennia requires more memory than your average C code base so this is important information to have. It seems most of our users run Evennia on various cloud hosting services rather than from a traditional remote server login.

Arx going strong 

The currently largest Evennia game, the mush Arx - After the Reckoning has helped a lot in stress testing. Their lead coder Tehom has also been active both in reporting issues and fixing them - kudos! There are however some lingering issues which appears rarely enough that they have not been possible to reproduce yet; we're working on those. Overall though I must say that considering how active Arx is, I would have expected to have seen even more "childhood diseases" than we have. 

Launch scripts and discussions

It is always interesting with feedback, and some time back another discussion thread erupted over on musoapbox, here. The musoapbox regulars have strong opinions about many things and this time some were critical of Evennia's install process. They felt it was too elaborate with too many steps, especially if you are approaching the system with no knowledge about Python. Apparently the average MUSH server has a much shorter path to go (even though that does require C compiling). Whereas I don't necessarily agree with all notions in that thread, it's valuable feedback - I've long acknowledged that it's hard to know just what is hard or not for a beginner.

Whereas we are planning to eventually move Evennia to pypi (so you can do pip install evennia), the instructions around getting virtualenv setup is not likely to change. So there is now unix shell scripts supplied with the system for installing on debian-derived systems (Debian, Ubuntu, Mint etc). I also made scripts for automating the setup and launch of Evennia and to use it with linux' initd within the scope of a virtualenv.
So far these scripts are not tested by enough people to warrant them being generally recommended, but if you are on a supported OS and is interested to try they are found from the top of the Evennia repo, in bin/unix/. More info can be found on their documentation page.

Docker

Speaking of installing, Evennia now has an official Docker image, courtesy of the work of contributor and Ainneve dev feend78. The image is automatically kept up-to.date with the latest Evennia repo and allows Evennia to be easily deployed in a production environment (most cloud services supports this). See Docker wiki page for more info.


Lots of new PRs

There was a whole slew of contributions waiting for me when returning from Chistmas break, and this has not slowed since. Github makes it easy to contribute and I think we are really starting to see this effect (Google Code back in the day was not as simple in this regard). The best thing with many of these PRs is that they address common things that people need to do but which could be made simpler or more flexible. It's hard to plan for all possibilities, so many people using the system is the best way to find such solutions.

Apart from the map-creation contribs from last year we also have a new Wildnerness system by mewser/titeuf87. This implements wilderness according to an old idea I had on the mailing list - instead of making a room per location, players get a single room.  The room tracks its coordinate in the wildnerness and updates its description and exits dynamically every time you move. This way you could in principle have an infinite wilderness without it taking any space. It's great to see the idea turned into a practical implementation and that it seems to work so well. Will be fun to see what people can do with it in the future!

Birthday retrospective

So, recently Evennia celebrated its ten-year anniversary. That is, it was on Nov 20, 2006, Greg Taylor made the first repo commit to what would eventually become the Evennia of today. Greg has said that Evennia started out as a "weird experiment" of building a MUD/MUX using Django. The strange name he got from a cheesy NPC in the Guild Wars MMORPG and Greg's first post to the mailing list also echoes the experimental intention of the codebase. The merger with Twisted came pretty early too, replacing the early asyncore hack he used and immediately seeing a massive speedup. Evennia got attention from the MUD community - clearly a Python-based MUD system sounded attractive.

When I first joined the project I had been looking at doing something MUD-like in Python for a good while. I had looked over the various existing Python code bases at the time and found them all to be either abandoned or very limited. I had a few week's stunt working with pymoo before asking myself why I was going through the trouble of parsing a custom script language ... in Python ... Why not use Python throughout? This is when I came upon Evennia. I started making contributions and around 2010 I took over the development as real life commitments forced Greg to step down.

Over the years we have gone through a series of changes. We have gone from using SVN to Mercurial and then to using GIT. We have transited from GoogleCode to GitHub - the main problem of which was converting the wiki documentation (Evennia has extensive documentation).

For a long time we used Python's reload() function to add code to the running game. It worked ... sometimes, depending on what you changed. Eventually it turned out to be so unpredictable that we now use two processes, one to connect clients to and the other running the game, meaning we can completely restart one process without disconnecting anyone.

Back in the day you were also expected to create your own game in a folder game/ inside the Evennia repo itself. It made it really hard for us to update that folder without creating merge conflicts all over. Now Evennia is a proper library and the code you write is properly separated from ours.

So in summary, many things have happened over the years, much of it documented in this blog. With 3500 commits, 28 000 lines of code (+46% comments) and some 25 people contributing in the last year, Openhub lists us as

"A mature, well-established codebase with a stable commit history, a large development team and very well documented source code". 

It's just words compiled by an algorithm, but they still feel kinda good!


While Evennia was always meant to be used for any type of multiplayer text game, this general use have been expanded and cleaned up a lot over the years.

This has been reflected in the width of people wanting to use it for different genres: Over time the MUSH people influenced us into adding the option to play the same character from many different clients at the same time (apparently, playing on the bus and then continuing on another device later is common for such games). Others have wanted to use Evennia for interactive fiction, for hack&slash, deep roleplay, strategy, education or just for learning Python.

Since Evennia is a framework/library and tries to not impose any particular game systems, it means there is much work to do when building a game using Evennia. The result is that there are dozens of games "in production" using Evennia (and more we probably don't know about), but few public releases yet.

The first active "game" may have been an Evennia game/chat supporting the Russian version of 4chan... The community driven Evennia demo-game Ainneve is also progressing, recently adding combat for testing. This is aimed at offering an example of more game-specific code people can build from (the way Diku does). There are similar projects meant for helping people create RPI (RolePlay Intensive) and MUSH-style games. That said, the Evennia-game Arx, After the Reckoning is progressing through beta at a good clip and is showing all signs of becoming the first full-fledged released Evennia game. 


So cheers, Evennia for turning 10. That's enough of the introspection and history. I'll get back to more technical aspects in the next post.

Season of fixes

The last few months has been dominated by bug-fixing and testing in Evennia-land. A lot more new users appears to be starting to use Evennia, especially from the MUSH world where the Evennia-based Arx, After the Reckoning is, while still in alpha, currently leading the charge.

With a new influx of users comes the application of all sorts of use- and edge-cases that stretch and exercise the framework in all the places where it matters. There is no better test of code than new users trying to use it unsupervised! Evennia is holding up well overall but there are always things that can be improved. 

• I reworked the way on-object Attributes was cached (from a stupid but simple way to a lot more sophisticated but harder way) and achieved three times faster performance in certain special cases people had complained about. Other issues also came to view while diving into this, which could be fixed.
• I reworked the venerable batch command and batchcode processors (these allow to create a game world from a script file) and made their inputs make more sense to people. This was one of the older parts of Evennia and apart from the module needing a big refactoring to be easier to read, some parts were pretty fragile and prone to break. Especially when passing it file names tended to be confusing since it understood only certain relative paths to the files to read in and not even I could remember if one should include the file ending or not. This was cleaned up a lot. 
• Lots of changes and updates were made to the RPSystem contrib, which optionally adds more advanced roleplaying mechanics to Evennia. The use of this in Evennia's demo game (Ainneve, being separately developed) helps a lot for ironing out any remaining wrinkles.
• Lots and lots of other fixes and smaller feature updates were done (About 150 commits and 50 Issues closed since end of summer).

A fun thing with a growing user base is that we are also starting to see a lot more Pull requests and contributions. Thanks a lot, keep 'em coming!

• Map system contrib (merged), for creating a world based on ASCII map. Talking about maps, users contributed not just one but two new tutorials for implementing both static and dynamic maps with Evennia. 
• Webclient notifications (pending), for making the webclient show us in a clearer way when it gets updated in a different tab. A more advanced implementation awaits the webclient being expanded with a proper client-side option window; there is currently a feature request for this if anyone's interested in taking it on.   
• AI system contrib (pending). This is a full AI backend for adding complex behaviors to game agents. It uses Behavioral trees and is designed to be modified both in code and from inside the game.
• Action system contrib (pending). This contrib assigns the actions of Characters a time cost and delays the results of commands the given time. It also allows players to go into turn-based mode to enforce a strict action order. 
• Lots of now closed PRs were contributed by the Arx lead developer to fix various bugs and edge-cases as they came up in live use.

The fixing and tightening of the nuts and bolts will likely continue the remainder of the year. I'm currently working on a refactoring of the way command sets are merged together (see the end of my blog post on Evennia in pictures for a brief summary of the command system). But with so much new blood in the community, who can tell where things will turn from here!

The art of sharing nicks and descriptions

In the month or so since the merger of Evennia's development branch and all its web-client updates, we have been in bug-fixing mode as more people use and stress the code.

There have been some new features as well though - I thought it could be interesting to those of you not slavishly following the mailing list.

 

Shared web login

When you are logged into the website you will now also auto-login to your account in the web client - no need to re-enter the login information! The inverse is also true. You still need to connect to game at least once to create the account, but after that you will stay connected while the browser session lasts.

Behind the scenes the shared login uses cookies linked to server-side Django sessions which is a robust and safe way to manage access tokens. Obviously browser sessions are irrelevant to telnet- or ssh connections.

Extended Nicks 

Evennia's nick(name) system is a way to create a personal alias for things in game - both to on-the-fly replacing text you input and for referring to in-game objects. In the old implementation this replacement was simply matched from the beginning of the input - if the string matched, it was replaced with the nick.

In this new implementation, the matching part can be much more elaborate. For example you can catch arguments and put those arguments into the replacement nick in another order.

Let's say we often use the @dig command this limited way:

>  @dig roomname;alias = exit;alias, backexit;alias
 
Let's say we find this syntax unintuitive. The new nick system allows to change this by catching the arguments in your nick and put it into the "real" command. Here is an example of a syntax putting the aliases in parentheses and separating all components with commas:

> nick newroom $1($2), $3($4), $5($6) = @dig $1;$2 = $3;$4, $5;$6

From here on you can now create your rooms with entries like this: 

> newroom The great castle(castle), to the castle(castle), back to the path(back)

Multidescer contrib

I have added a new "multidescer" to the contrib folder. A multidescer is (I think) a MUSH term for a mechanism managing more than one description. You can then combine any of these various descriptions into your "active" description.

An example of usage:

desc hat = a blue hat.
desc basic = This is a tall man with narrow features.
desc clothing = He is wearing black, flowing robes.

 These commands store the description on the Character and references them as unique keywords. Next we can combine these strings together in any order to build the actual current description: 

> desc/set basic + |/ + clothing + On his head he has + hat
> look self
This is a tall man with narrow features. 
He is wearing black, flowing robes. On his head he has a blue hat.

This allows for both very flexible and easy-to-update descriptions but also a way to handle freeform equipment and clothing. And you can of course use the nick system to pre-format the output

> nick setdesc $1 $2 $3 $4 = $1 + |/ + clothing + On his head he has a $4

This way you can clothe yourself in different outfits easily using the same output format:

> setdesc basic clothing hat 
 
 The multidescer is a single, self-contained command that is easy to import and add to your game as needed.



... There's also plenty of bug fixes, documentation work and other minor things or course.

Anyway, summer is now upon us here in the northern hemisphere so things will calm down for a bit, at least from my end. Have a good 'un!
.
Griatch

Image by ryancr (released under Creative Commons)

Evennia 0.6 !

As of today, I merged the development branch to make version 0.6 of the MU* development system and server Evennia.

Evennia 0.6 comes with a lot of updates, mainly in the way Evennia talks to the outside world. All communication is now standardized, so there are no particular treatment of things like text - text is just one of any standardized commands being passed between the server the client (whether over telnet, ssh, websockets or ajax/comet).

For example the user can now easily plug in "inputfuncs" to handle any data coming from the client. If you want your client to offer some particular functionality, you just need to plop in a python function to handle it, server-side. We also now offer a lot of utility functions for things like monitoring change (tell the client whenever your health status changes so it can update a health bar or flash the screen).

The HTML5 webclient has itself updated considerably. Most is happening behind the scenes though. Notably the webclient's javascript component is split into two:

•  evennia.js, acts as a library for handling all communication with the server part of Evennia. It offers events for a gui library to plug into and send/receive. It will also gracefully degrade from a websocket connection to AJAX/COMET long-polling if the player uses an older browser. 
• evennia_gui.js is the default front-end and implements a traditional and stable "telnet-like" interface. The html part uses uses Django templating to make it easy to customize and expand. Since this simply makes use of the resources offered by evennia.js, one could pretty easily slip in some other gui library here, or set up calls to get all sorts of interesting information from the server (which talks back using inputfuncs). 

There are a truckload of more updates and features that are covered on the mailing list.
.
Griatch

Technical stuff happening

Hi folks, a bit more technical entry this time. These usually go onto the Evennia mailing list but I thought it would be interesting to put it in the dev-blog for once.

So, I'm now halfway through the TODO list issue of the wclient development branch as alluded to in the last post. The wclient branch aims to rework and beef up the web client infrastructure of Evennia.

The first steps, which has been done a while was converting the SSH/SSL and IRC input/output protocols to use the new protocol infrastructure (telnet and websockets was done since before). That's just under-the-hood stuff though. Today I finished the changes to the Monitor/TickerHandlers, which may be of more general interest.

With the changes to the the way OOB (Out-Of-Band) messages are passing through Evennia (see this mailing list post for more details), the OOBHandler is no more. As discussed there, the handling of incoming data is becoming a lot freer and will be easily expandable to everyone wanting to make for a custom client experience. The idea is thus for Evennia to offer resources for various input commands to make use of, rather than prescribing such functionality in a monolothic way in the OOBHandler. There were three main functionalities the OOBHandler offered, and which will now be offered by separate components:

1. Direct function callback. The instruction from the client should be able to trigger a named server-side function. This is the core of the inputfunc system described previously.
2. Field/Attribute monitoring. The client should be able to request monitoring of an object's database fields or Attributes. For example, the client may request to be notified whenever the Character's "health" Attribute changes in some way. This is now handled by the new monitorhandler. See below.
3. Non-persistent function repeats. One should be able to set up a repeating ticker that survives a server reload but does not survive a cold shutdown - this mimics the life cycle of server Sessions. Scripts could do this already but I wanted to be able to use the TickerHandler for quick assignment. Problem was that the Tickerhandler in master branch is not only always-persistent, it also only calls database object methods. So I have now expanded the tickerhandler to also accept arbitrary module functions, without any connection to a database object.

The MonitorHandler 

evennia.MONITOR_HANDLER is the new singleton managing monitoring of on-object field/attribute changes. It is used like this:

MONITOR_HANDLER.add(obj, field_or_attrname, callback, **kwargs)


Here obj is a database entity, like a Character or another Object. The field_or_attrname is a string giving the name of a db_* database field (like "db_key", "db_location" etc). Any name not starting with db_ is assumed to be the name of an on-object Attribute (like "health"). Henceforth, whenever this field or attribute changes in any way (that is, whenever it is re-saved to the database), the callback will be called with the optional kwargs, as well as a way to easily get to the changed value. As all handlers you can also list and remove monitors using the standard MONITOR_HANDLER.remove(), .all() etc.


The TickerHandler

evennia.TICKER_HANDLER should be familiar to Evennia users from before - it's been around for a good while. It allows for creating arbitrary "tickers" that is being "subscribed" to - one ticker will call all subscribers rather than each object or function having its own timer.

Before, the syntax for adding a new ticker required you specify a typeclassed entity and the name of the method on it to call every N seconds. This will now change. This is the new callsign for creating a new ticker:

TICKER_HANDLER.add(interval, callback, idstring="", persistent=True, *args, **kwargs)

Here, interval, like before, defines how often to call callback(*args, **kwargs).

The big change here is that callback should be given as a valid, already imported callable, which can be either an on-entity method (like obj.func) or a global function in any module (like world.test.func) - the TickerHandler will analyze it and internally store it properly.

idstring works as before, to separate tickers with the same intervals. Finally persistent=False means the ticker will behave the same way a Script with persistent=False does: it will survive a server reload but will not survive a server shutdown. This latter functionality is particularly useful for client-side commands since the client Session will also not survive a shutdown.

... So this is a rather big API change to the TickerHandler, which will mean some conflicts for those of you relying heavily on tickers. Easiest will definitely be to simply stop the old and start new ones. It's not clear yet if we'll offer some automated way to convert old tickers to new ones. Chime in if this is something important to you.

Happening Next

The next steps involves making use of these new utilities to implement the basic OOB commands recommended by the MSDP and GMCP protocols along with some recommended functionality. We'll see how long that takes, but progress is being made. And if you are a web guy, do consider helping out.
.
Griatch

Climbing up Branches

Today I pushed the latest Evennia development branch "wclient". This has a bunch of updates to how Evennia's webclient infrastructure works, by making all exchanged data be treated equal (instead of treating text separately from other types of client instructions).

It also reworks the javascript client into a library that should be a lot easier to expand on and customize. The actual client GUI is still pretty rudimentary though, so I hope a user with more web development experience can take upon themselves to look it over for best practices.

A much more detailed description of what is currently going on (including how to check out the latest for yourself) is found in this mailing list post. Enjoy!

A summary of a year

As 2015 is slowly drawing to an end, I looked back through Evennia's repository to see just what was going on this year. And it turns out it was a lot! I honestly didn't remember some things happened as recently as they did.

Note: For those reading this not familiar with Evennia, it's a Python library for creating MUDs (text-based multiplayer games).
 

Making Evennia into a library

In February of 2015 we merged what was likely the biggest change happening for a good while in Evennia - the complete refactoring of the Evennia repository into a library. It used to be that when you cloned the Evennia repo, it would come with a pre-made game/ folder where you were supposed to put your custom files. Mixing the stuff you downloaded from us with your own files (which you might want to keep under version control of your own) was not a very clean solution.

In the big "library update", we instead created a stand-alone evennia program which you use to create your own "game dir" with pre-created templates. Not only does this allow you to treat the cloned evennia repo as a proper library, you can also use the same evennia install for multiple games and makes it a lot clearer just what comes from us and what is your custom code.

Below is a fun Gource representation (courtesy of Youtuber Landon Wilkins) of how the Evennia repository structure has changed over the years. You can see the latest library-change a little after the 3minute, 20 second mark.

Typeclasses into proxies

At the same time as the library change, I also completely overhauled the way typeclasses are represented in Evennia: Typeclasses are Python classes that links back to a database model under the hood. They used to use a custom overloading of varioust get/set methods, now they instead utilize django proxy models extended to support multiple inheritance. T

his radically increased performance and made the code considerably cleaner, as well as completely hid the core django model from the end user (no longer a need to track if you were dealing with a model or its typeclass - you only ever run into typeclasses). This, together with the library-model led to some changes in people's source codes.

Even so, this change led to a lot less problems and edge-cases than I had anticipated: it seems more people had issues with upgrading django than with adopting their codes to the new typeclass changes ...

Evennia Autodocs

Following the big library merger I sat down to write a more comprehensive autodoc utility. We had been distributing a Doxygen config file with the repo for a long time, but I wanted something that integrated with our github wiki, using markdown in the source (because frankly, while Sphinx produces very pretty output, ReST markup looks really ugly in source code, in my opinion).

The result was the api2md program, which is now a part of our wiki repository. It allows our source code to be decorated in "Google style", very readable output:

def funcname(a, b, c, d=False):
    """
    This is a brief introduction to the 
    function/class/method

    Args:
        a (str): This is a string argument that 
          we can talk about over multiple lines.
        b (int or str): Another argument
        c (list): A list argument
        d (bool, optional): An optional 
           keyword argument

    Returns:
        e (str): The result of the function

    Notes:
        This is an example function. 
        If `d=True`, something amazing will happen.

    """


This will be parsed and converted to a Markdown entry and put into the Github wiki, one page per module. The result is the automatically generated Evennia API autodocs, reachable as any other wiki page.

The convertion/prettification of all core functions of Evennia to actually use  the Google-style docstrings took almost all year, finishing late in autumn. But now almost all of Evennia uses this style. Coincidentally this also secures us at a 45% comment/code ratio. This places us in the top 10% of well-documented open-source projects according to openhub (gotta love statistics).

Imaginary realities / Optional Realities

Spring of 2015 saw some more articles for the Imaginary Realities e-zine as well as some for the newly-opened Optional Realities web forum (unrelated, despite the similar name). The latter was made by a team working on a commercial Evennia-based sci-fi game, but the forums were open for other games (of any engine) and general discussion on mud and mud design.

Optional Realities published an impressive range of articles (one every week for several months) and organized several very interesting mud-related contests. It did, I think, a lot for bringing some new life to the mud-development scene.  Unfortunately Optional Realities suffered a complete database loss towards the end of the year, forcing it to sort of reboot from scratch. I hope it will rebound and that the articles can be put back online again!

Ainneve

Over summer I put out a general roll call for developers willing to lead the development of a small but fully functioning Evennia demo game - a project separate from the main Evennia development. The idea is to have something working for people to rip out and start from. The response to my request was very good and we relatively quickly ended up with two devs willing to lead and direct the effort.

The work-in-progress demo game they started is called Ainneve. It uses the base Open Adventure RPG rules and is distributed using the same liberal licence as Evennia.

Ainneve has been sputtering along throughout autumn and winter, and while progress has been a bit ... sporadic, it seems to attract new volunteers to help out. As more of the base systems gets finalized there is going to be even more "low hanging fruit" for people to jump in and help with.

EvMenu, EvMore and EvEditor, RPSystem

In July I merged some new systems into Evennia's utilities library. EvMenu is a class that builds an in-game menu. It uses multiple-choice questions for the user to navigate its nodes. I had a "menusystem" in our contrib/ folder since many years, but it was showing its age and when I found I myself didn't really understand how it was working I found it time to make something more flexible.

The EvMore is a page-scroller that allows the server to pause long texts and wait for the user to press a key before continuing. It's functionality is similar to the unix more program.

For its part, EvEditor has existed for a while as well, but it was moved from the contrib folder into the main utilility library as it turned out to be an important and common resource (it's basically a mud-version of the classic vi editor).

Finally, some months later, in September, I added the rpsystem and rplanguage contribution modules. The former is an author-stance recognition system similar to what is seen in some rp-heavy muds (where you don't know/see people's name off the bat, but only their description, until you manually assign a name to them).

The rplanguage module is used for handling foreign (fantasy) languages by obfuscating words heard by different parties depending on their relative language skills.

Python 3 ... some day

As we know, Evennia uses Python 2. In autumn of 2015 first one and then two of our users took upon themselves to help make Evennia a little more ready for running also under Python 3. After a lot of work, Evennia's core is now using code syntax that is compatible with both Python 2 and 3.

We don't run under Python 3 at this point though. This is not something under our control, but is due to Twisted not supporting Python 3 yet. But when it becomes possible, we are now better prepared for transitioning to a point where we can (hopefully) support both main Python versions.

EvCast

In September, user whitenoise started his "EvCast" video tutorial series on Evennia. There are so far two episodes, one on installing Evennia and the second on general Python concepts you need to use Evennia efficiently:

 Python.__init__ Podcast

In the end of September I was interviewed by the hosts of the python.__init__ podcast. Turned into more than an hour of discussion about Evennia. T'was fun! You can listen to it here.

Nested inlinefuncs

Towards the end of the year I got (via Optional Realities) lured to participate in another text-gaming related website, musoapbox. This is a forum primarily populated by players from the MUSH-side of text-gaming. After some back and forth they made a strong case for increasing the functionality of Evennia's inlinefunctions.

An inlinefunction is, in Evennia, a function call embedded in a string sent to the server. This is parsed and executed (no, there is no dangerous eval happening, the call is parsed as text and then called as a normal function call). It allows for text to be dynamically replaced at run-time. What the MUSHers suggested was to also allow those inlinefunc's to be nestable - that is, to implement a call stack. The nestable form of inlinefuncs was merged with Evennia master at the end of November.

Patreon

This actually happened already back in February: after some requests for more ways to support Evennia development, I opened a Patreon page. And here at the end of the year, a whole ten patrons have signed up for it. Very encouraging!

Next year

In 2016 the first upcoming thing that comes to mind is push out the changes to the webclient infrastructure. That has been a long time coming since I've been unsure of how to proceed on it for quite some time. The goal is to make the webclient a lot more "pluggable" and modular than it is, and to clean up its API and way of communicating with the server. Overall I think the web-side of things need some love.

I'll likely put together some more general-use contribs as well, I have some possibles in mind. 

We'll continue squashing bugs and work down our roadmap.

I'll also try to get an Evennia Wikipedia.org page together, if you want to look at how it's progressing and help editing it, see the Evennia mailing list for the link.


... And a lot more I don't know yet, no doubt! On towards a new year!
.
Griatch

Pushing through a straw

Recently, a user reported a noticeable delay between sending a command in-game to multiple users and the result of that command appearing to everyone. You didn't notice this when testing alone but I could confirm there was almost a second delay sometimes between entering the command and some users seeing the result. A second is very long for stuff like this. Processing time for a single command is usually in the milliseconds. What was going on?

Some background 

Evennia has two components, the Portal and the Server, running as two separate processes. The basic principle is that players connecting to an Evennia instance connects to the Portal side - this is the outward facing part of Evennia. The connection data and any other input and output will be piped from the Portal to the Server and back again.

The main reason for this setup is that it allows us to completely reset the Server component (reloading module data in memory is otherwise error-prone or at least very tricky to make generically work in Python) without anyone getting disconnected from the Portal. On the whole it works very well.

Debugging

Tracing of timings throughout the entire Server<->Portal pipeline quickly led me to rule out the command handler or any of the core systems being responsible - including both sides of the process divide was a few milliseconds. But in the transfer between Portal and Server, an additional 900 miliseconds were suddenly added! This was clearly the cause for the delay.

Turns out that it all came down to faulty optimization. At some point I built a batch-send algorithm between the Server and Portal. The idea was to group command data together if they arrived too fast - bunch them together and send them as a single batch. In theory this would be more efficient once the rate of command sending increased. It was partly a DoS protection, partly a way to optimize transfer handling.

The (faulty) idea was to drop incoming data into a queue and if the rate was too high, wait to empty that queue until a certain "command/per second" rate was fullfilled. There was also a timed routine that every second force-emptied the queue to make sure it would be cleaned up also if noone made any further commands.

In retrospect it sounds silly but the "rate of command" was based on a simple two-time-points derivative;

rate = 1 / (now - last_command_time)

If this rate exceeded a set value, the batch queuing mechanism would kick in. The issue with this (which is easy to see now) is that if you set your limit at (say) 100 commands / second, two commands can happen to enter so close to each other time that their rate far exceed that limit just based on the time passed between them. But there are not actually 100 commands coming in every second which is really what the mechanism was meant to react to.

So basically using a moment-to-moment rate like this is just too noisy to be useful; the value will jump all over the place. The slowdown seen was basically the DoS protection kicking in because when you are relaying data to other users, each of them will receive "commands" in quick succession - fast enough to trigger the limiter. These would be routed to the queue and the sometimes-delay simply depended on  when the queue cleanup mechanism happened to kick in.

Resolution

Once having identfied the rate measuring bug, the obvious solution to this would be to gather command rates over a longer time period and take an average - you will then filter out the moment-to-moment noise and get an actually useful rate.

Instead I ended up going with an even simpler solution: Every command that comes in ups a counter. If I want a command rate limit of 100 commands/second, I wait until that counter reaches 100. At that point I check when the time difference between now and when the counter was last reset. If this value is below 1, well then our command rate is higher than 100/second and I can kick in whatever queuing or limiter is needed. The drawback is that until you have 100 commands you won't know the rate. In practice though, once the rate is high enough to be of interest, this simple solution leads to an automatic check with minimal overhead.

In the end I actually removed the batch-sending component completely and instead added command DoS protection higher up on the Portal side. The Command-input is now rate limited using the same count-until-limit mechanism. Seems to work fine. People have no artificial slowdowns anymore and the DoS limiter will only kick in at loads that are actually relevant. And so all was again well in Evennia world.

Documenting Python without Sphinx

Last week Evennia merged its development branch with all the features mentioned in the last post. Post-merger we have since gone through and fixed remaining bugs and shortened the list at a good clip.

One thing I have been considering is how to make Evennia's API auto-documenting - we are after all a MUD creation library and whereas our code has always been well-documented the docs were always only accessible from the source files themselves.

Now, when you hear "Python" and "documentation" in the same sentence, the first thought usually involves Sphinx or Sphinx autodoc in some form. Sphinx produces very nice looking documentation indeed. My problem is however as follows:

• I don't want our API documentation to be written in a different format from the rest of our documentation, which is in Github's wiki using Markdown.  Our users should be able to help document Evennia without remembering which formatting language is to be used.
• I don't like reStructuredText syntax. This is a personal thing. I get that it is powerful but it is also really, really ugly to read in its raw form in the source code. I feel the sources must be easy to read on their own.
• Sphinx plugins like napoleon understands this ugliness and allows you to document your functions and classes in a saner form, such as the "Google style". One still needs reST for in-place formatting though.
• Organizing sphinx document trees is fiddly and having had a few runs with sphinx autodoc it's just a mess trying to get it to section the Evennia sources in a way that makes sense. It could probably be done if I worked a lot more with it, but it's a generic page generator and I feel that I will eventually have to get down to make those toctrees myself before I'm happy.
• I want to host the API docs as normal Wiki files on Github (this might be possible with reST too I suppose).

Long story short, rather than messing with getting Sphinx to do what I want, I ended up writing my own api-to-github-Markdown parser for the Evennia sources: api2md. Using Python's inspect module and aiming for a subset of the Google formatted docstrings, this was maybe a day's work in total - the rest was/is fine-tuning for prettiness.
 
Now whenever the source is updated, I follow the following procedure to fully update the API docs:

1. I pull the latest version of Evennia's wiki git repository from github alongside the latest version of the main Evennia repository.
2. I run api2md on the changed Evennia sources. This crawls the main repo for top-level package imports (which is a small list currently hard-coded in the crawler - this is to know which modules should create "submodule" pages rather than try to list class contents etc). Under each package I specify it then recursively gets all modules. For each module in that package, it creates a new Markdown formatted wiki page which it drops in a folder in the wiki repository. The files are named after the model's path in the library, meaning you get files like evennia.objects.models.md and can easily cross-link to subsections (aka classes and functions) on a page using page anchors.
3. I add eventual new files and commit the changes, then push the result to the Github wiki online. Done!

(I could probably automate this with a git hook. Maybe as a future project.)

The api2md program currently has some Evennia-custom elements in it (notably in which packages it imports) but it's otherwise a very generic parser of Python code into Markdown. It could maybe be broken out into its own package at some point if there's interest.   

The interesting thing is that since I already have code for converting our wiki to reST and ReadTheDocs, I should be able to get the best of both worlds and convert our API wiki pages the same way later. The result will probably not be quite as custom-stunning as a Sphinx generated autodoc (markdown is a lot simpler in what formatting options it can offer) but that is a lesser concern.
So far very few of Evennia's docstrings are actually updated for the Google style syntax (or any type of formatting, really) so the result is often not too useful. We hope that many people will help us with the doc strings in the future - it's a great and easy way to get to know Evennia while helping out.

But where the sources are updated, the auto-generated wiki page looks pretty neat.


(Image from Wikimedia commons)

Building Django proxies and MUD libraries

2015 is here and there is a lot of activity going on in Evennia's repository, mailing list and IRC channel right now, with plenty of people asking questions and starting to use the system to build online games.

We get newcomers of all kinds, from experienced coders wanting to migrate from other code bases to newbies who are well versed in mudding but who aim to use Evennia for learning Python. At the moment the types of games planned or under development seems rather evenly distributed between RPI-style MUDs and MUSH games (maybe with a little dominance of MUSH) but there are also a couple of hack-and-slash concepts thrown into the mix. We also get some really wild concepts pitched to us now and then. What final games actually comes of it, who can tell, but people are certainly getting their MU*-creative urges scratched in greater numbers, which is a good sign.

Since Christmas our "devel" branch is visible online and is teeming with activity. So I thought I'd post an summary about it in this blog. The more detailed technical details for active developers can be found on Evennia's mailing list here (note that full docs are not yet written for devel-branch).

Django proxies for Typeclasses

I have written about Evennia's Typeclass system before on this blog. It is basically a way to "decorate" Django database models with a second set of classes to allow Evennia developers to create any type of game entity without having to modify the database schema. It does so by connecting one django model instance to one typeclass instance and overloading __setattr__ and __getattribute__ to transparently communicate between the two.

For the devel branch I have refactored our typeclass system to make use of Django's proxy models instead. Proxy models have existed for quite a while in Django, but they simply slipped under my radar until a user pointed them out to me late last year. A proxy model is basically a way to "replace the Python representation of a database table with a proxy class". Sounds like a Typeclass, doesn't it?
Now, proxy models doesn't work quite like typeclasses out of the box - for one thing if you query for them in the database you will get back the original model and not the proxy one. They also do not allow multiple inheritance. Finally I don't want Evennia users to have to set up django Meta info every time they use a proxy. So most work went into overloading the proxy multiclass inheritance check (there is a django issue about how to fix this). Along the way I also redefined the default managers and __init__ methods to always load the proxy actually searched for and not the model. I finally created metaclasses to handle all the boilerplate. We choose to keep the name Typeclass also for this extended proxy. This is partly for legacy reasons, but typeclasses do have their own identity: they are not vanilla Django-proxies nor completely normal Python classes (although they are very close to the latter from the perspective of the end user).
Since typeclasses now are directly inheriting from the base class (due to meta-classing this looks like normal Python inheritance), it makes things a lot easier to visualize, explain and use. Performance-wise this system is en par with the old, or maybe a little faster, but it will also be a lot more straight forward to cache than the old. I have done preliminary testing with threading and it looks promising (but more on that in a future post). 


Evennia as a Python library package

Evennia has until now been solely distributed as a version controlled source tree (first under SVN, then Mercurial and now via GIT and Github). In its current inception you clone the tree and find inside it a game/ directory where you create your game. A problem we have when helping newbies is that we can't easily put pre-filled templates in there - if people used them there might be merge conflicts when we update the templates upstream. So the way people configure Evennia is to make copies of template modules and then change the settings to point to that copy rather than the default module. This works well but it means a higher threshold of setup for new users and a lot of describing text. Also, while learning GIT is a useful skill, it's another hurdle to get past for those who just want to change something minor to see if Evennia is for them.

In the devel branch, Evennia is now a library. The game/ folder is no longer distributed as part of the repository but is created dynamically by using the new binary evennia launcher program, which is also responsible for creating (or migrating) the database as well as operating the server:

evennia --init mygame
cd mygame
evennia migrate
evennia start

Since this new folder is not under our source tree, we can set up and copy pre-made template modules to it that people can just immediately start filling in without worrying about merge conflicts. We can also dynamically create a setting file that fits the environment as well as set up a correct tree for overloading web functionality and so on. It also makes it a lot easier for people wanting to create multiple games and to put their work under separate version control.

Rather than traversing the repository structure as before you henceforth will just do import evennia in your code to have access to the entirety of the API. And finally this means it will (eventually) be possible to install Evennia from pypi with something like pip install evennia. This will greatly ease the first steps for those not keen on learning GIT.

For existing users

Both the typeclasses-as-proxies and the evennia library changes are now live in the devel branch. Some brave users have already started taking it through its paces (and is helping to flesh it out) but it will take some time before it merges into master.

The interesting thing is that despite all this sounding like a huge change to Evennia, the coding API doesn't change very much, the database schema almost not at all. With the exception of some properties specific to the old connection between the typeclass and model, code translate over pretty much without change from the developer's standpoint.

The main translation work for existing developers lies in copying over their code from the old game/ directory to the new dynamically created game folder. They need to do a search-and-replace so that they import from evennia rather than from src or ev. There may possibly be some other minor things. But so far testers have not found it too cumbersome or time consuming to do. And all agree that the new structure is worth it.

So, onward into 2015!


Image: "Bibliothek St. Florian" by Original uploader was Stephan Brunker at de.wikipedia Later versions were uploaded by Luestling at de.wikipedia. - Originally from de.wikipedia; description page is/was here.. Licensed under CC BY-SA 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Bibliothek_St._Florian.jpg#mediaviewer/File:Bibliothek_St._Florian.jpg

Dance my puppets

In many traditional multiplayer text engines for MUD/MUSH/MU*, the player connects to the game with an account name that also becomes their character's in-game name. When they log into the game they immediately "become" that character. If they want to play with another character, they need to create a new account.

A single-login system is easy to implement but many code bases try to expand with some sort of "account system" where a single login "account" will allow you to manage one or more game characters. Matthew “Chaos” Sheahan  beautifully argues for the benefits of an account system in the April issue of Imaginary Realities; you can read his article here.

Evennia and account systems

First a brief show of how Evennia handles this. We use the following separation:

Session(s) <-> Player <-> Objects/Characters(s)

The Session object represents individual client connections to Evennia. The Player is our "account" object. It holds the password hash and your login name but has no in-game existence. Finally we have Objects, the most common being a subclass of Object we call Character. Objects exist in the game. They are "puppeted" by Sessions via the Player account.

From this separation an account system follows naturally. Evennia also offers fully flexible puppeting out of the box: Changing characters (or for staff to puppet an NPC) is simply a matter of "disconnecting" from one Character and connecting to another (presuming you have permission to do so).

The Multisession modes of Evennia

This is the main gist of this entry since we just added another of these (mode 3). Evennia now offers four different multisession modes for the game designer to choose between. They affect how you gamers may control their characters and can be changed with just a server reload. 

 

Mode 0

This is emulates the "traditional" mud codebase style. In mode 0 a Session controls one Character and one character only. Only one Session per account is allowed - that is, if a user try to connect to their Player account with a different client the old connection will be disconnected. In the default command set a new Character is created with the same name as the Player account and the two are automatically connected whenever they log in. To the user this makes Player and Character seem to be virtually the same thing.

 

Mode 1

In this mode, multiple Sessions are allowed per Player account. You still only have one Character per account but you can control that Character from any number of simultaneously connected clients. This is a requirement from MUSHes and some slower-moving games where there are communities of gamers who want to conveniently track the progress of the game continuously on multiple clients and computers. 

 

Mode 2

In multisession mode 2, multiple Characters are allowed per Player account. No Characters are created by default in this mode, rather the default command set will drop you to a simplified OOC management screen where you can create new characters, list the ones you already have and puppet them. This mode offers true multiplaying, where you can connect via several clients simultaneously, each Session controlling a different Character.

 

Mode 3

This mode allows gamers not only to play multiple Characters on the same Player account (as in mode 2) but to also connect multiple Sessions to each Character. This is a multi-character version of Mode 1, where players can control the same Character via Player logins from several different clients on different machines in any combination.



It's interesting that some of these modes may seem silly or superfluous to people used to a certain type of MU* yet are killer features for other communities. It goes to show how different the needs are for users of different game styles.

Webby stuff

Latest Evennia come with a range of improvements, mainly related to its integration with the web.

New and improved ways to expand the website/webclient

Thanks to the work of contributor Kelketek, Evennia's Django-based web system (website and webclient) has been restructured to be much easier to expand. Previously you had to basically copy the entire web/ folder into your game and modify things in-place. This was not ideal since it made it inherently harder to update when things changed upstream. Now Evennia makes use of Django's collectstatic functionality to allow people to plugin and overload only the media files and templates that they need. Kelketek wrote a new and shiny web tutorial explaining just how things work. 

Websocket-based webclient with OOB

Evennia's webclient was an ajax-based one using a long polling ("comet") paradigm to work. These days all modern browsers support websockets though, a protocol that allows asynchronous server-client communication without the cludgery of long polling. So Evennia's new webclient will now use websockets if the browser supports it and fall back to the old comet client if it does not.

The new client also has full support for OOB (Out-of-band) communication. The client uses JSON for straight forward OOB messaging with the server. As part of this, I had an excuse to go back to clean up and make the OOB backbone of Evennia more complete. The server-side oob commands are borrowed from MSDP but the server side is of course independent of communication protocol (so webclient and telnet extensions can call the same server-side callbacks). I've not yet finalized the documentation for how to use the OOB yet, that will come soon-ish.

Bringing back python memory

Lately I've done work on the memory management of Evennia. Analyzing the memory footprint of a python program is a rather educational thing in general.

Python keeps tracks of all objects (from variables to classes and everything in between) via a memory reference. When other objects reference that object it tracks this too.
Once nothing references an object, it does not need to be in memory any more - in a more low-level languages this might lead to a memory leak. Python's garbage collector handles this for us though - it goes through all abandoned objects and frees the memory for usage by other things. The garbage collector will however not do its thing as long as some other object (which will not be garbage-collected) still holds a reference to the object. This is what you want - you don't want existing objects to stop working because an object they rely on is suddenly not there.

Normally in Django, whenever you retrieve an database model instance, that exists only in memory then and there. If you later retrieve the same object from the database, the model instance you have to work with is most likely a new one. This is okay for most usage, but Evennia's typeclass system (described in an earlier blog entry) as well our wish to store temporary properties on models (existing until next server restart) does not work if the model instance we get is always different. It would also help if we didn't have to load models from the database more than necessary.

For this reason, Evennia uses something called the idmapper. This is a cache mechanism (heavily modified for Evennia) that allows objects to be loaded from the database only once and then be reused when later accessed. The speedup achieved from this is important, but as said it also makes critical systems work properly.

The tradeoff of speed and utility is memory usage. Since the idmapper never drops those references it means that objects will never be garbage collected. The result was that the memory usage of Evennia could rise rapidly with an increasing number of objects. Whereas some objects (like those with temporary attributes) should indeed not be garbage collected, in a working game there is likely to be objects without such volatile data. An example might be objects that are not used some of the time - simply because players or the game don't need them for the moment. For such objects it may be okay to re-load them on demand rather than keep them in memory indefinitely.

When looking into this I found that simply force-flushing the idmapper did not clean up all objects from memory. The reason for this has to do with how Evennia references objects via a range of other means. The reference count never went to zero and so the garbage collector never got around to it. 

With the excellent objgraph library it is actually pretty easy to track just what is referencing what, and to figure out what to remove. Using this I went through a rather prolonged spree of cleanups where I gradually (and carefully) cleaned up Evennia's object referencing to a point where the only external reference to most objects were the idmapper cache reference. So removing that (like when deliberately flushing the cache) will now make the object possible to garbage-collect.

This is how the reference map used to look for one type of Evennia object (ObjectDB) before the cleanup. Note the several references into the ObjectDB and the cyclic references for all handlers (the cyclic reference is in itself not a problem for reference-counting but they are slow and unnecessary; I now made all handlers use lazy-loading with weak referencing instead).

This is how the reference map looks for the same object now. The __instance__ cache is the idmapper reference. There are also no more cyclic references for handlers (the display don't even pick up on them for this depth of recursion). Just removing that single link will now garbage-collect ObjectDB and its typeclass (ignore the g reference, that is just the variable holding the object in ipython).
We also see that the dbobj.typeclass <-> typeclass.dbobj references keep each other alive and when one goes the other one goes too - just as expected.

An curious aspect of Python memory handling is that (C-)Python does not actually release the memory back to operating system when flushing the idmapper cache. Rather Python makes it internally available so that it does not need to request any more. The result is that if you look at Evennia with the top command, its memory requirement (for example while continuously creating new objects) will not actually drop on a idmapper flush, it will just stop rising.  This is discussed at length in this blog, it was good to learn it was not something I did at least.


Apart from the memory stuff, there is work ongoing with fixing the latest batch of user issue reports. Another dev is working on cleaning up the web-related code, it should make it a lot cleaner to overload web functionality with custom code. One of those days I'll also try to sit down and finally convert our web client from long-polling to use web sockets now that Evennia suppports web sockets natively. Time, time ...

Moving from Google Code to GitHub

A few weeks back, the Evennia project made the leap from Google Code to GitHub (here). Things have been calming down so it's time to give a summary of how the process went.

Firstly I want to say that I liked Google Code. It did everything expected of it with little hassle. It had a very good Issue system (better than GitHub in my opinion) and it allowed us to use Mercurial instead of Git for version control (I just happen to like Mercurial better than Git, so sue me). Now, GitHub is getting to be something of a standard these days. But whereas our users have occationaly inquired about us making the move, I've been reluctant to do so. 

The problem I did have with Google Code was that I got the increasing feeling that Google didn't care all that much about it. It worked decently, but it was not really going anywhere either. What finally made me change my mind though was an event just after summer last year. There was a bug in Google Code that made the links to online clones disappear. It was worse than that - creating new online clones of the main repo didn't work - people wanting to contribute using a clone just couldn't. 

This is extremely critical functionality for a code-sharing website to have! I made a bug report and many other projects chimed in seeing the same issues. Eventually the links returned and everything worked the way it had. But it took several months before this critical bug was fixed. Even then Google didn't even bother to close my issue. This suggested quite strongly to me that Google Code is not really a priority even for its parent company. It was time to consider a move.

I was never personally a fan of Git. It is undoubtedly powerful, but I always felt its syntax way too archaic and the number of ways to shoot yourself in the foot way too many. But I do like GitHub better than BitBucket (I've used both in other projects), so that's where we nevertheless were heading.

Already last year I created an Evennia "organization" on GitHub and one of our  users first helped to set up a Git Mirror of our Mercurial repo. The idea was a good one - have a mirror on GitHub, allowing the transition to be more gradual. In the end this didn't work out though - there were some issue with the hg-git conversion and the mirror never didn't actually update. When I checked back and it was three months behind we just removed that first ill-fated version. 

In the end I decided to not fiddle about with it, but to move everything over in one go.

 Converting the repository

 

I set aside a new folder on my hard drive and cloned the original mercurial repo into a new sub folder. A good idea is to set up a quick Python virtual environment for easily getting updated dependencies of build scripts. 

I initialized an empty Git repository and used a program called hg-fast-export to convert. As it turned out there were some finer details to consider when doing that:

• The most obvious one was that the conversion initially failed, complaining about the Mercurial original containing "unnamed branches". These came from a contributor who did something to spawn off all sorts of weird branches with little purpose. I should not have merged those into main in the first place, but in those days I didn't know mercurial well enough to be  concerned. In the end I simply used mercurial's MQ extension to remove the unnamed (and unused) branches so the conversion could complete.
• The second issue was that Mercurial is less stringent about its author strings than Git is. Git's author string is "name <email>". Over the years we have gotten contributions from people with all sorts of combinations of names, with or without an email address. So for this we had to supply a mapping file to the converter. It's basically a list of old_author_string = new_author_string and allows for grouping the various used names as needed (some of them were the same person using slightly different author strings). 

Once this was in place, the repo conversion worked fine. It was just a matter of changing the .hgignore file to a .gitignore file and change some code that made use of mercurial to get and display the current revision id.

 Converting the Wiki, part one

Evennia's wiki consitutes our documentation, it's some 80+ pages or so by now. Definitely not something we want to loose. Google Code use a dialect of MediaWiki whereas GitHub's wiki supports a few other formats, like markdown or reST. I needed to convert between them.

Digging around a bit I found googlecode2github. This download contains python scripts for converting the wiki as well as Issues. I didn't really get the issues-converter to work, so I had to find another solution for that (see next section).

All in all, the initial wiki conversion worked decently - all the pages were converted over and were readable. I was even to the point of declaring success when finding the damn thing messed up the links. Googe Code writes links like this: [MyLink Text to see on page]. The script converted this to [[MyLink|Text to see on page]]. Which may look fine except it isn't. GitHub actually wants the syntax in the inverse order: [[Text to see on page|MyLink]].

Furthermore, in Google Code's wiki, code blocks were marked with

{{{ 
  <verbatim code> 
}}}

In markdown, code blocks are created just by indenting the block by four spaces. The converter dutifully did this - but it didn't add empty lines above and below the block, which is another thing markdown requires. The result was that all code ended up mixed into the running text output.

I could have gone back and fixed the converter script, but I suspected there would be enough small things to fix anyway. So in the end I went through 80+ pages of fixing link syntax and adding empty lines by hand. After that I could finally push the first converted wiki version up to the GitHub wiki repository.

Some time later I also found that there is a way to let GitHub wiki pages use syntax highlighting for the language of your choice. The way to do this is to enclose your code blocks like this:


```python

<verbatim code>

```

This is apparently "GitHub-flavoured" markdown. So another stint into all the pages followed, to update everything for prettiness.

 Converting Google Code Issues

I didn't want to loose our Issues from Google Code. I looked around a bit and tested some conversions for this (it helps to be able to create and delete repos on GitHub with abandon when things fail). I eventually settled on google-code-issues-migrator.

This is a Python script that gathers all the Issues from a given Google Code project. It then uses GitHub's API to re-post the issues. It retains the issue numbers and re-maps the Google Code Issue tags to GitHub's equivalent. It didn't retain most other formatting and whereas I ended up as the creator of all issues, the converter included the name of the original author as well as a link back to the original Google Code one. I found that to be quite sufficient for our needs. 

 Converting the IRC announcer

A lot of development discussion goes on in our IRC channel #evennia on Freenode. There is an announcer bot in there that I've written, that collates information from various sources and reports it in the IRC channel:

• Repository updates
• Wiki updates
• Issue creation and updates
• Mailing list/forum posts
• Dev-blog updates (this blog)

Say what you will about Google, but they are great at offering RSS feeds to all their stuff. So my IRC bot was basically a glorified threaded RSS reader that echoed changes to the channel as they came in. This had been working nicely for years.

GitHub does offer RSS feeds to -some- of their offerings, but it's a lot more patchy. I eventually had to do quite a bit of hacking to get everything reporting the way we were used to.

• GitHub has its own IRC announcer bot that reports to IRC. The problem is that this will connect, send message and then disconnect. This causes a lot of spam in the channel. We neither can nor want to set +n on our channel to allow external messages either. The way I solved this was to expand my own custom IRC bot to sit in two irc channels. The GitHub announcer connects to only one of them (so this gets all the spammy connect messages). My IRC bot picks up the announcement and echoes it cleanly to our main #evennia channel.  It works really well.
• Issues are handled by the GitHub announcer in the same way. 
• GitHub has no automatic way to report wiki updates. It doesn't even have a proper RSS feed. However, a user clued me in on using the pipes website to relay an RSS feed from github. I then configured my IRC bot to check that RSS and report it (I also changed the IRC colours to match the GitHub-announcer ones).
• Mailing list and blog haven't changed, so those are still handled via RSS as before.

All this done, the modified IRC announcement works well.

 Closing the book on Google Code

At this point all the critical things were moved over. So after some heads-up warnings on the mailing list (and users helping to rewrite our documentation to use Git instead of mercurial) we eventually made the official move.

One thing I really dislike is when a project switches hosts and don't let users know about it in their revision history. So I made a mercurial-only last commit announcing that the repo is closed and giving the link to the new one.

The Google Code page doesn't go anywhere, but I changed the front page to point to GitHub instead. I even made an issue in the Issue tracker with a title telling people not to use that tracker anymore. Finally I re-pointed all the links on http://www.evennia.com to GitHub and made a mailing list posting. Move was officially complete.

 Converting the Wiki, part 2

At this point were were officially moved over and I started to look into getting fancy with our documentation. We have for the longest time made automated translations of our wiki for compiling by ReadTheDocs.

Getting Google Code's special wikimedia syntax into reST (that ReadTheDocs uses) used to mean jumping through a few hoops. My hackish solution worked in two steps. First a custom python script (whose originating url I can no longer find, sorry) converted the Google Code wiki to HTML. Once this was done, pandoc converted the files from HTML to reST. The result was ... acceptable. There were some minor issues here and there but mostly the result was readable.

I figured that converting from the more standard Markdown of the GitHub wiki to reST should be a breeze by comparison. Not so.

The first hurdle was that the version of pandoc coming with my Linux distribution was too old to support Github-flavoured markdown syntax. I knew from before that Pandoc worked so I didn't want to start with something else. I had to download the several hundred MBs needed by the Haskell build environment and their package manager in order to get and compile all the dependencies and finally the latest version of pandoc. To their credit it was all a very streamlined experience, it just took quite some time.

The second hurdle came when finally looping pandoc to convert all wiki files. It turns out to be that the [[Text on page|address]] syntax I had manually corrected earlier is a special syntax offered by Gollum, the engine powering GitHub's wiki behind the scenes. None of the markdown-to-reSt converters I looked at (pandoc or otherwise) even recognized this syntax as a link at all. As it turns out, normal markdown actually expects its links in the format [Text on page](address).

I was not going to go through and edit all those pages again. So my next step was to write a script to scan and replace all the [[...|...]] syntax in our wiki and replace it with the standard markdown one. After this the markdown files converted to reST quite nicely -- formatting-wise they look much better than the old wiki to HTML to reST chain I had to use from Google Code.

Problem was that when compiling these reST pages into HTML with Sphinx, no links worked.

Each individual page looked okay, just that the links were not pointing to anything reasonable. In retrospect this was not so strange. Pandoc knows nothing about the relationships between files, and clearly the simple naming scheme used for addresses is something the wiki softwares knows and Sphinx does not.

Some thinking lead to a custom Python script for renaming the link targets in the converted pages to their html page name. This needed to handle the fact that wiki links also allows whitespace. So the [Start](Getting Started) link would be converted to [Start](GettingStarted.html), which seems to be the format with which Sphinx will generate its pages.

One also needs to have a "toc" (Table of Contents) to tie all those pages together for the benefit of Sphinx. I just used a "hidden" toc, letting my converter script add this to the bottom of my normal index file. As long as it's included somewhere, Sphinx will be happy.

Originally I put the reST files in a subfolder of the GitHub wiki repo, I thought I could just point ReadTheDocs to that repo later. The GitHub wiki has a strange "feature" though. It seems to pick its wiki pages from wherever they are in the repo, no matter if they are in the root or in subfolders. Suddenly I was starting to see reST-style pages appear in the online wiki, and sometimes I would get the markdown version (the two would go out of sync). Very strange and confusing.

Since the files clearly "polluted" our wiki, I had to move the converted reST files to a separate branch of the wiki repository. This has the advantage of keeping all the support scripts and converter mechanisms separate from the normal wiki content. ReadTheDocs can luckily be set to read its information from another branch than master, so finally the latest converted wiki can again be read there!

That concludes what I think was the last main conversion effort. Phew!

 Impressions so far

GitHub is nice. The merge requests and easy way to comment on them are really good. Since people are more familiar with using GitHub overall, it does seem to be a shorter step for people to make a fork and contribute small things. Doing the same in Google Code was probably not harder per se, just something less people were previously familiar with.

Due to the modular way Evennia is structured, people are recommended to make a fresh clone of the new Git repo and simply copy their plugin source files and database back into it. So far this seems to have gone smoothly.

The GitHub issue tracker is worse than the Google Code one. It has no good way to order Issues or list them in a more compact form (nor in a matrix). Not having good issue templates is really limiting; having to reply to issues only to ask for basic info they should include in their issue is an unnecessary time sink.

I also find that there is no clear way to announce an issue change (like "Needing more information"). Tags work partly for this, but setting them is not announced anywhere as far as I can tell - they are just there.

Most things also takes so much spaaace. Overall GitHub seems designed for people with big monitors. I have those, but most of the time I prefer working on my laptop. I'm sure it's a matter of habit, but Google Code is very compact by comparison. It gave a lot better overview of things. On GitHub I have to scroll everywhere and this is true both in the repo view, wiki and issues.

These small quips nonwithstanding, I think this move will serve us well. There is a good wibe of development and continuing improvement going on at GitHub. There's plenty of help and tutorials all over. Since so many people are using GitHub, problems are more likely to have been answered before. And of course we hope this will in effect help more people find Evennia and join the fun.