Sunday, May 22, 2016

Evennia 0.6 !


As of today, I merged the development branch to make version 0.6 of the MU* development system and server Evennia.

Evennia 0.6 comes with a lot of updates, mainly in the way Evennia talks to the outside world. All communication is now standardized, so there are no particular treatment of things like text - text is just one of any standardized commands being passed between the server the client (whether over telnet, ssh, websockets or ajax/comet).

For example the user can now easily plug in "inputfuncs" to handle any data coming from the client. If you want your client to offer some particular functionality, you just need to plop in a python function to handle it, server-side. We also now offer a lot of utility functions for things like monitoring change (tell the client whenever your health status changes so it can update a health bar or flash the screen).

The HTML5 webclient has itself updated considerably. Most is happening behind the scenes though. Notably the webclient's javascript component is split into two:

  •  evennia.js, acts as a library for handling all communication with the server part of Evennia. It offers events for a gui library to plug into and send/receive. It will also gracefully degrade from a websocket connection to AJAX/COMET long-polling if the player uses an older browser. 
  • evennia_gui.js is the default front-end and implements a traditional and stable "telnet-like" interface. The html part uses uses Django templating to make it easy to customize and expand. Since this simply makes use of the resources offered by evennia.js, one could pretty easily slip in some other gui library here, or set up calls to get all sorts of interesting information from the server (which talks back using inputfuncs). 
There are a truckload of more updates and features that are covered on the mailing list.
.
Griatch

Thursday, March 24, 2016

Technical stuff happening

Hi folks, a bit more technical entry this time. These usually go onto the Evennia mailing list but I thought it would be interesting to put it in the dev-blog for once.

So, I'm now halfway through the TODO list issue of the wclient development branch as alluded to in the last post. The wclient branch aims to rework and beef up the web client infrastructure of Evennia.

The first steps, which has been done a while was converting the SSH/SSL and IRC input/output protocols to use the new protocol infrastructure (telnet and websockets was done since before). That's just under-the-hood stuff though. Today I finished the changes to the Monitor/TickerHandlers, which may be of more general interest.

With the changes to the the way OOB (Out-Of-Band) messages are passing through Evennia (see this mailing list post for more details), the OOBHandler is no more. As discussed there, the handling of incoming data is becoming a lot freer and will be easily expandable to everyone wanting to make for a custom client experience. The idea is thus for Evennia to offer resources for various input commands to make use of, rather than prescribing such functionality in a monolothic way in the OOBHandler. There were three main functionalities the OOBHandler offered, and which will now be offered by separate components:

  1. Direct function callback. The instruction from the client should be able to trigger a named server-side function. This is the core of the inputfunc system described previously.
  2. Field/Attribute monitoring. The client should be able to request monitoring of an object's database fields or Attributes. For example, the client may request to be notified whenever the Character's "health" Attribute changes in some way. This is now handled by the new monitorhandler. See below.
  3. Non-persistent function repeats. One should be able to set up a repeating ticker that survives a server reload but does not survive a cold shutdown - this mimics the life cycle of server Sessions. Scripts could do this already but I wanted to be able to use the TickerHandler for quick assignment. Problem was that the Tickerhandler in master branch is not only always-persistent, it also only calls database object methods. So I have now expanded the tickerhandler to also accept arbitrary module functions, without any connection to a database object.
The MonitorHandler 

evennia.MONITOR_HANDLER is the new singleton managing monitoring of on-object field/attribute changes. It is used like this:


MONITOR_HANDLER.add(obj, field_or_attrname, callback, **kwargs)

Here obj is a database entity, like a Character or another Object. The field_or_attrname is a string giving the name of a db_* database field (like "db_key", "db_location" etc). Any name not starting with db_ is assumed to be the name of an on-object Attribute (like "health"). Henceforth, whenever this field or attribute changes in any way (that is, whenever it is re-saved to the database), the callback will be called with the optional kwargs, as well as a way to easily get to the changed value. As all handlers you can also list and remove monitors using the standard MONITOR_HANDLER.remove(), .all() etc.


The TickerHandler

evennia.TICKER_HANDLER should be familiar to Evennia users from before - it's been around for a good while. It allows for creating arbitrary "tickers" that is being "subscribed" to - one ticker will call all subscribers rather than each object or function having its own timer.

Before, the syntax for adding a new ticker required you specify a typeclassed entity and the name of the method on it to call every N seconds. This will now change. This is the new callsign for creating a new ticker:


TICKER_HANDLER.add(interval, callback, idstring="", persistent=True, *args, **kwargs)

Here, interval, like before, defines how often to call callback(*args, **kwargs).

The big change here is that callback should be given as a valid, already imported callable, which can be either an on-entity method (like obj.func) or a global function in any module (like world.test.func) - the TickerHandler will analyze it and internally store it properly.

idstring works as before, to separate tickers with the same intervals. Finally persistent=False means the ticker will behave the same way a Script with persistent=False does: it will survive a server reload but will not survive a server shutdown. This latter functionality is particularly useful for client-side commands since the client Session will also not survive a shutdown.

... So this is a rather big API change to the TickerHandler, which will mean some conflicts for those of you relying heavily on tickers. Easiest will definitely be to simply stop the old and start new ones. It's not clear yet if we'll offer some automated way to convert old tickers to new ones. Chime in if this is something important to you.

Happening Next

The next steps involves making use of these new utilities to implement the basic OOB commands recommended by the MSDP and GMCP protocols along with some recommended functionality. We'll see how long that takes, but progress is being made. And if you are a web guy, do consider helping out.
.
Griatch

Sunday, February 14, 2016

Climbing up Branches

Today I pushed the latest Evennia development branch "wclient". This has a bunch of updates to how Evennia's webclient infrastructure works, by making all exchanged data be treated equal (instead of treating text separately from other types of client instructions).

It also reworks the javascript client into a library that should be a lot easier to expand on and customize. The actual client GUI is still pretty rudimentary though, so I hope a user with more web development experience can take upon themselves to look it over for best practices.

A much more detailed description of what is currently going on (including how to check out the latest for yourself) is found in this mailing list post. Enjoy!

Thursday, December 17, 2015

A summary of a year

As 2015 is slowly drawing to an end, I looked back through Evennia's repository to see just what was going on this year. And it turns out it was a lot! I honestly didn't remember some things happened as recently as they did.

Note: For those reading this not familiar with Evennia, it's a Python library for creating MUDs (text-based multiplayer games).
 

Making Evennia into a library

In February of 2015 we merged what was likely the biggest change happening for a good while in Evennia - the complete refactoring of the Evennia repository into a library. It used to be that when you cloned the Evennia repo, it would come with a pre-made game/ folder where you were supposed to put your custom files. Mixing the stuff you downloaded from us with your own files (which you might want to keep under version control of your own) was not a very clean solution.

In the big "library update", we instead created a stand-alone evennia program which you use to create your own "game dir" with pre-created templates. Not only does this allow you to treat the cloned evennia repo as a proper library, you can also use the same evennia install for multiple games and makes it a lot clearer just what comes from us and what is your custom code.

Below is a fun Gource representation (courtesy of Youtuber Landon Wilkins) of how the Evennia repository structure has changed over the years. You can see the latest library-change a little after the 3minute, 20 second mark.



Typeclasses into proxies

At the same time as the library change, I also completely overhauled the way typeclasses are represented in Evennia: Typeclasses are Python classes that links back to a database model under the hood. They used to use a custom overloading of varioust get/set methods, now they instead utilize django proxy models extended to support multiple inheritance. T

his radically increased performance and made the code considerably cleaner, as well as completely hid the core django model from the end user (no longer a need to track if you were dealing with a model or its typeclass - you only ever run into typeclasses). This, together with the library-model led to some changes in people's source codes.

Even so, this change led to a lot less problems and edge-cases than I had anticipated: it seems more people had issues with upgrading django than with adopting their codes to the new typeclass changes ...

Evennia Autodocs

Following the big library merger I sat down to write a more comprehensive autodoc utility. We had been distributing a Doxygen config file with the repo for a long time, but I wanted something that integrated with our github wiki, using markdown in the source (because frankly, while Sphinx produces very pretty output, ReST markup looks really ugly in source code, in my opinion).

The result was the api2md program, which is now a part of our wiki repository. It allows our source code to be decorated in "Google style", very readable output:

def funcname(a, b, c, d=False):
   
"""
    This is a brief introduction to the 

    function/class/method

    Args:
        a (str): This is a string argument that 

          we can talk about over multiple lines.
        b (int or str): Another argument
        c (list): A list argument
        d (bool, optional): An optional 

           keyword argument

    Returns:
        e (str): The result of the function

    Notes:
        This is an example function. 

        If `d=True`, something amazing will happen.

    """



This will be parsed and converted to a Markdown entry and put into the Github wiki, one page per module. The result is the automatically generated Evennia API autodocs, reachable as any other wiki page.

The convertion/prettification of all core functions of Evennia to actually use  the Google-style docstrings took almost all year, finishing late in autumn. But now almost all of Evennia uses this style. Coincidentally this also secures us at a 45% comment/code ratio. This places us in the top 10% of well-documented open-source projects according to openhub (gotta love statistics).

Imaginary realities / Optional Realities

Spring of 2015 saw some more articles for the Imaginary Realities e-zine as well as some for the newly-opened Optional Realities web forum (unrelated, despite the similar name). The latter was made by a team working on a commercial Evennia-based sci-fi game, but the forums were open for other games (of any engine) and general discussion on mud and mud design.

Optional Realities published an impressive range of articles (one every week for several months) and organized several very interesting mud-related contests. It did, I think, a lot for bringing some new life to the mud-development scene.  Unfortunately Optional Realities suffered a complete database loss towards the end of the year, forcing it to sort of reboot from scratch. I hope it will rebound and that the articles can be put back online again!

Ainneve

Over summer I put out a general roll call for developers willing to lead the development of a small but fully functioning Evennia demo game - a project separate from the main Evennia development. The idea is to have something working for people to rip out and start from. The response to my request was very good and we relatively quickly ended up with two devs willing to lead and direct the effort.

The work-in-progress demo game they started is called Ainneve. It uses the base Open Adventure RPG rules and is distributed using the same liberal licence as Evennia.

Ainneve has been sputtering along throughout autumn and winter, and while progress has been a bit ... sporadic, it seems to attract new volunteers to help out. As more of the base systems gets finalized there is going to be even more "low hanging fruit" for people to jump in and help with.

EvMenu, EvMore and EvEditor, RPSystem

In July I merged some new systems into Evennia's utilities library. EvMenu is a class that builds an in-game menu. It uses multiple-choice questions for the user to navigate its nodes. I had a "menusystem" in our contrib/ folder since many years, but it was showing its age and when I found I myself didn't really understand how it was working I found it time to make something more flexible.

The EvMore is a page-scroller that allows the server to pause long texts and wait for the user to press a key before continuing. It's functionality is similar to the unix more program.

For its part, EvEditor has existed for a while as well, but it was moved from the contrib folder into the main utilility library as it turned out to be an important and common resource (it's basically a mud-version of the classic vi editor).

Finally, some months later, in September, I added the rpsystem and rplanguage contribution modules. The former is an author-stance recognition system similar to what is seen in some rp-heavy muds (where you don't know/see people's name off the bat, but only their description, until you manually assign a name to them).

The rplanguage module is used for handling foreign (fantasy) languages by obfuscating words heard by different parties depending on their relative language skills.

Python 3 ... some day

As we know, Evennia uses Python 2. In autumn of 2015 first one and then two of our users took upon themselves to help make Evennia a little more ready for running also under Python 3. After a lot of work, Evennia's core is now using code syntax that is compatible with both Python 2 and 3.

We don't run under Python 3 at this point though. This is not something under our control, but is due to Twisted not supporting Python 3 yet. But when it becomes possible, we are now better prepared for transitioning to a point where we can (hopefully) support both main Python versions.

EvCast

In September, user whitenoise started his "EvCast" video tutorial series on Evennia. There are so far two episodes, one on installing Evennia and the second on general Python concepts you need to use Evennia efficiently:




 Python.__init__ Podcast

In the end of September I was interviewed by the hosts of the python.__init__ podcast. Turned into more than an hour of discussion about Evennia. T'was fun! You can listen to it here.

Nested inlinefuncs

Towards the end of the year I got (via Optional Realities) lured to participate in another text-gaming related website, musoapbox. This is a forum primarily populated by players from the MUSH-side of text-gaming. After some back and forth they made a strong case for increasing the functionality of Evennia's inlinefunctions.

An inlinefunction is, in Evennia, a function call embedded in a string sent to the server. This is parsed and executed (no, there is no dangerous eval happening, the call is parsed as text and then called as a normal function call). It allows for text to be dynamically replaced at run-time. What the MUSHers suggested was to also allow those inlinefunc's to be nestable - that is, to implement a call stack. The nestable form of inlinefuncs was merged with Evennia master at the end of November.

Patreon

This actually happened already back in February: after some requests for more ways to support Evennia development, I opened a Patreon page. And here at the end of the year, a whole ten patrons have signed up for it. Very encouraging!

Next year

In 2016 the first upcoming thing that comes to mind is push out the changes to the webclient infrastructure. That has been a long time coming since I've been unsure of how to proceed on it for quite some time. The goal is to make the webclient a lot more "pluggable" and modular than it is, and to clean up its API and way of communicating with the server. Overall I think the web-side of things need some love.

I'll likely put together some more general-use contribs as well, I have some possibles in mind. 

We'll continue squashing bugs and work down our roadmap.

I'll also try to get an Evennia Wikipedia.org page together, if you want to look at how it's progressing and help editing it, see the Evennia mailing list for the link.


... And a lot more I don't know yet, no doubt! On towards a new year!
.
Griatch

Thursday, November 12, 2015

MIT uses Evennia!

Evennia was recently used as a test bed to train an AI system to attempt to play a MUD as a human would - by only reading and understanding the text on the screen.

Researchers at MIT (Massachusetts Institute of Technology) recently presented the paper Language understanding for Text-based games using Deep reinforcement learning (PDF) at a conference on natural language processing. A summary is in the MIT press release.

I was contacted by these fine folks some time ago so I knew they had plans to use Evennia for their research. It's great to see they now have an article out on it! Evennia devs are also mentioned in the acknowledgements - so something for the Evennia dev community to be proud of! 

MUDs are tricky

The main complication for an AI playing a MUD is that the computer has no access to the actual game state but must try to surmise how well it's doing only from the text given (same as a human would). The researchers compare the results from a range of deep-learning neural network algorithm that they train to play.

To test their AI, the researchers first used Evennia to build a simple training "Home World": a 4-room "house" where the simple goal is to find and eat an apple while refraining to go to sleep. The room descriptions used here were pretty formulaic although not trivial to give a challenge. This they used to train their AI system.

They then took this trained neural network and applied it to the real challenge, playing the Evennia Tutorial World. You can yourself try this out in our demo install or by just running a single command when starting Evennia. They call it "Fantasy World" in the article.

The tutorial world has hand-written descriptions and often describes the exits as part of the room text. The article actually makes a comprehensive analysis of the tutorial world, including the available game states and transitions as well as the number of words and number of commands per state. Interesting stuff in itself. I presume the scientists have modified their copy of the tutorial world to provide better metrics for their analysis.

A bridge too far

As far as I understand from the article, the AI does understand to use commands with one or two arguments (like eat apple or the move red-root right), but they note that actually finding the tomb of the fallen hero (the main quest of the tutorial) is too hard for the AI:

[...]However, this is a complex quest that requires the player to memorize game events and perform high-level planning which are beyond the scope of this current work.
So instead they evaluate the AI's performance on a more mundane task: Getting across the bridge to the castle. It's not clear to me if the AI actually plays more of the game too or if their test just exposes the AI to the bridge itself. I suspect it does play more due to the examples they use from other rooms; evaluating the bridge-crossing is just a clear-cut metric to use for "success".

The MIT press release claims that the AI is also scored on how much health/magic it has, but I don't see that mentioned in the article itself (and the tutorial world only has magic if you find the hero's tomb which they claim they cannot do).

The bridge in Evennia's tutorial world is actually a single "room" that takes multiple steps to cross. At every step the room description changes to describe the progress. Random texts will appear as the bridge sways in the wind and various environmental cues are heard and seen. There is also a small chance of falling off the bridge if one lingers too long on it.

So although all you really need to do is to walk east repeatedly, I can see why this can be a challenge to a neural network having no mental image of what a bridge is. It can only work off the text it's given at any given time.

In the paper, the algorithms are evaluated both on their ability to actually cross the bridge and on how optimal their solution was, for example by not issuing invalid commands to the situation.

Beyond the bridge

The results are that after being trained on the training house setup, the AI will eventually be able to cross the bridge. The particular algorithm proposed also perform slightly better than the comparison ones (and a lot better than simple randomness).

So from the perspective of the researchers this seems to be a success. Even so, this reinforces the fact that quite some way to go before an AI can *actually* play a real MUD successfully. Using MUDs for this type of research is a good idea though, and I do hope they expand and continue this line work in the future.

Who knows, maybe the AI will even find that ancient tomb eventually!

Image from MIT news

Sunday, October 11, 2015

Illustrations and soaps

I recently made an article presenting the infrastructure and main coding features of Evennia using a series of nifty diagrams.

It can hopefully help newcomers get a feeling for the grand structure of an Evennia server/game a little easier. It's published as a feature on Optional Realities and you can find the article here: Evennia Illustrated.

I also recently joined MU Soapbox, a forum predominantly discussing MUSH games, to answer some technical questions on Evennia. Unsurprisingly, some (although so far surprisingly few) MUSHers express concern about Evennia's explicit lack of softcode (that is, the ability for players to use a safe in-game language to code their game rather than to use external Python modules). Their argument is sound:  They are used to in-game coding as a way to easily express creatitivy and easily garner help from players.

Evennia's stance here clash a bit with those opinions: Our philosophy is that our command system is powerful enough to offer players any complexity of build commands they want. The design/coding of the game itself should be done using proper coding IDEs and modern version control tools.

There is no denying that compared to a softcode-coded game, a player-level contributor to an Evennia game needs some extra tools to create and contribute code over version control. The admin also needs to check such contributions for malicious code before merging it into their running game. Now, these are differences I actually consider advantages from a code-quality perspective. And for finding help, people on average are more likely to know Python than a custom softcode language. But here opinions differ and in a given game community those language adoption numbers can be severely skewed.

So far, the MUSHers that have adopted Evennia seems to have done so very much to get away from softcode. It will be interesting to see if things like Kelketek's in-development Group building Evennia contrib will be stirring some interest from those on the fence, or if coding their entire game in softcode is indeed an irreplaceable source of their gaming fun.

Friday, October 2, 2015

Emoting system, or how to chat up a girl

A few days ago I pushed an emoting contribution to Evennia. A "contrib" is an optional plugin system that is not part of core Evennia but is meant to be easily picked up and used in people's own designs.

If you are not familiar with what an emoting system does, it is a way to decribe the actions of a character in the game. The simplest form of emote is a single command (like the command dance leading to some canned response, or in the case of a graphical game, a dance animation). This contribution offers a more sophisticated system though, allowing input like the following:

emote /me smiles at /cheerful as he sits at her table. "Hello!" he says.

Now, this includes /keywords that relate to the objects in the room. So assuming there is a very cheerful girl in the room, this string will come out as 

Griatch smiles at a very cheerful girl as he sits at her table. "Hello!" he says. 

But she will actually see only my outward appearance (the short description) since she doesn't know me. So the cheerful girl (let's say her name is Sandra) would for example see

A man in flowing robes smiles at Sandra as he sits at her table. "Hello!" he says.

The emoting system has the following features: 

  • Short description replacement in emotes and in searches, as seen above. This means that you can do look cute and the system will know what you want to look at (in vanilla Evennia you'd need to use look Sandra).
  • Multi-word searching and disambiguation. If there is a cute girl and a cute puppy both in the same room, your referencing of /cute will  give an error listing the alternatives. You can then either include more words to make your reference unique or use an index (1-cute, 2-cute) to make it clear who you mean. This mimics normal object-key disambiguation in Evennia.
  • Recognition. You can assign your own aliases to people. If Sandra introduces herself you could assign her the name Sandra and henceforth be able to reference her as such and see that name appear. But you could also name her The girl calling herself Sandra if you didn't believe that's her actual name.
  • Languages. Everything within double-quotes is parsed as spoken language (like the Hello! above). By using writing this as (elvish)"Hello!", this could be spoken in another language and those who don't speak elvish would receive an obfuscated string.
  • Masking. A person wearing a mask can force people's recognition replacement to deactivate so that they are not recognized anymore.
The emoting contrib comes as two files in evennia/contrib/: rpsystem.py and rplanguage.py. To use them fully, make your Characters and Rooms inherit from the supplied classes and/or add the new commands to the Character command set. Enjoy!

image ©Griatch, from griatch-art.deviantart.com