Sunday, April 23, 2017

The luxury of a creative community

For this blog post I want to focus on the series of very nice pull requests coming in from a growing cadre of contributors over the last few months.

Contributed goodness

People have put in a lot of good work to boost Evennia, both by improving existing things and by adding new features. Thanks a lot everyone (below is just a small selection)!
  •  Contrib: Turn-based combat system - this is a full, if intentionally bare-bones implementation of a combat system, meant as a template to put in your particular game system into.
  • Contrib: Clothing sytem - a roleplaying mechanic where a character can 'wear' items and have this show in their descriptions. Worn items can also be layered to hide that underneath. Had plenty of opportunities for extensions to a given game.
  • Contrib: An 'event' system is in the works, for allowing privileged builders to add dynamic code to objects that fires when particular events happen. The PR is not yet merged but promises the oft pondered feature of in-game coding without using softcode (and notably also without the security of softcode!). 
  • A lot of PRs, especially from one user, dealt with cleanup and adherence to PEP8 as well as fixing the 'alerts' given by LGTM on our code (LGTM is by the way a pretty nifty service, they parse the code from the github repo without actually running it and try to find problems. Abiding by their advice results is cleaner code and it also found some actual edge-case bugs here and there not covered by unit tests. The joint effort has brought us down from some 600+ alerts to somewhere around 90 - the remaining ones are alerts which I don't agree with or which are not important enough to spend effort on). 
  • The help mechanics of Evennia were improved by splitting up the default help command into smaller parts, making it easier to inject some changes to your help system without completely replacing the default one. 
  • Evennia's Xterm256 implementation was not correctly including the additional greyscale colors, those were added with new tags |=a ... |=z.
  • Evennia has the ability to relay data to external services through 'bots'. An example of this is the IRC bot, which is a sort of 'player' that sits in an in-game channel and connects that to a counterpart-bot sitting in a remote IRC channel. It allows for direct game-IRC communication, something enjoyed by people in the Evennia demo for many years now. The way the bot was defined used to be pretty hard-coded though. A crafty contributor changed that though, but incorporating the bot mechanism into Evennia's normal message flow. This allows for adding new types of bots or extending existing ones without having to modify Evennia's core. There is already an alternative IRC bot out there that represents everyone in the IRC room as a room full of people in the MUD. 
  • Evennia's Attributes is a database table connected to other objects via a ForeignKey relation. This relation is cached on the object. A user however found that for certain implementations, such as using Attributes for large coordinate systems, non-matches (that is failed Attribute lookups on the object) can also be cached and leads to dramatic speed increases for those particular use cases. A PR followed. You live and learn.
  • Another contributor helped improve the EvEditor (Evennia's VIM-like in-game text editor) by giving it a code-mode for editing Python code in-game with auto-indents and code execution. Jump into the code mode with the command @py/edit.
  • Time scheduling is another feature that has been discussed now and then and has now been added through a PR. This means that rather than specifying 'Do this in 400 seconds' you can say 'do this at 12AM, in-game time'. The core system works with the real-world time units. If you want 10 hours to a day or two weeks to a month the same contributor also made an optional calendar contrib for that!
  • A new 'whisper' command was added to the Default cmdset. It's an in-game command for whispering to someone in the same room without other people hearing it. This is a nice thing to have considering Evennia is out-of-the-box pretty much offering the features of a 'talker' type of game.
  • Lots of bug fixes big and small!
  • Some at_* hooks were added, such as at_give(giver, getter). This allows for finer control of the give process without handling all the logics at the command level. There are others hooks in the works but those will not be added until in Evennia 0.7. 
About that Evennia 0.7 ...

So while PRs are popping up left and right in master I've been working in the devel branch towards what will be the Evennia 0.7 release. The branch is not ready for public consumption and testing yet But tentatively it's about halfway there as I am slowly progressing through the tickets. Most of the upcoming features were covered in the previous blog post so I'll leave it at that.

I just want to end by saying that it's a very luxurious (and awesome) feeling for me to see master-branch Evennia expand with lots of new stuff "without me" so to speak. The power of Open Source indeed!
  

Image from http://maxpixel.freegreatpicture.com, released as public domain.

Sunday, February 5, 2017

News items from the new year

The last few months have been mostly occupied with fixing bugs and straightening out usage quirks as more and more people take Evennia through its paces.

Webclient progress

One of our contributors, mewser/titeuf87 has put in work on implementing part of our roadmap for the webclient. In the first merged batch, the client now has an option window for adjusting and saving settings. This is an important first step towards expanding the client's functionality. Other  features is showing help in an (optional) popup window and to report window activity by popup and/or sound.

The goal for the future is to allow the user or developer to split the client window into panes to which they can then direct various output from the server as they please It's early days still but some of the example designs being discussed can be found in the wiki webclient brainstorm (see the title image of this blog for one of the mockups).
 
New server stuff

Last year saw the death of our old demo server on horizondark.com, luckily the new one at silvren.com has worked out fine with no hickups. As part of setting that up, we also got together a more proper list of recommended hosts for Evennia games. Evennia requires more memory than your average C code base so this is important information to have. It seems most of our users run Evennia on various cloud hosting services rather than from a traditional remote server login.

Arx going strong 

The currently largest Evennia game, the mush Arx - After the Reckoning has helped a lot in stress testing. Their lead coder Tehom has also been active both in reporting issues and fixing them - kudos! There are however some lingering issues which appears rarely enough that they have not been possible to reproduce yet; we're working on those. Overall though I must say that considering how active Arx is, I would have expected to have seen even more "childhood diseases" than we have. 

Launch scripts and discussions

It is always interesting with feedback, and some time back another discussion thread erupted over on musoapbox, here. The musoapbox regulars have strong opinions about many things and this time some were critical of Evennia's install process. They felt it was too elaborate with too many steps, especially if you are approaching the system with no knowledge about Python. Apparently the average MUSH server has a much shorter path to go (even though that does require C compiling). Whereas I don't necessarily agree with all notions in that thread, it's valuable feedback - I've long acknowledged that it's hard to know just what is hard or not for a beginner.

Whereas we are planning to eventually move Evennia to pypi (so you can do pip install evennia), the instructions around getting virtualenv setup is not likely to change. So there is now unix shell scripts supplied with the system for installing on debian-derived systems (Debian, Ubuntu, Mint etc). I also made scripts for automating the setup and launch of Evennia and to use it with linux' initd within the scope of a virtualenv.
So far these scripts are not tested by enough people to warrant them being generally recommended, but if you are on a supported OS and is interested to try they are found from the top of the Evennia repo, in bin/unix/. More info can be found on their documentation page.

Docker

Speaking of installing, Evennia now has an official Docker image, courtesy of the work of contributor and Ainneve dev feend78. The image is automatically kept up-to.date with the latest Evennia repo and allows Evennia to be easily deployed in a production environment (most cloud services supports this). See Docker wiki page for more info.


Lots of new PRs

There was a whole slew of contributions waiting for me when returning from Chistmas break, and this has not slowed since. Github makes it easy to contribute and I think we are really starting to see this effect (Google Code back in the day was not as simple in this regard). The best thing with many of these PRs is that they address common things that people need to do but which could be made simpler or more flexible. It's hard to plan for all possibilities, so many people using the system is the best way to find such solutions.

Apart from the map-creation contribs from last year we also have a new Wildnerness system by mewser/titeuf87. This implements wilderness according to an old idea I had on the mailing list - instead of making a room per location, players get a single room.  The room tracks its coordinate in the wildnerness and updates its description and exits dynamically every time you move. This way you could in principle have an infinite wilderness without it taking any space. It's great to see the idea turned into a practical implementation and that it seems to work so well. Will be fun to see what people can do with it in the future!

Wednesday, November 30, 2016

Birthday retrospective

So, recently Evennia celebrated its ten-year anniversary. That is, it was on Nov 20, 2006, Greg Taylor made the first repo commit to what would eventually become the Evennia of today. Greg has said that Evennia started out as a "weird experiment" of building a MUD/MUX using Django. The strange name he got from a cheesy NPC in the Guild Wars MMORPG and Greg's first post to the mailing list also echoes the experimental intention of the codebase. The merger with Twisted came pretty early too, replacing the early asyncore hack he used and immediately seeing a massive speedup. Evennia got attention from the MUD community - clearly a Python-based MUD system sounded attractive.

When I first joined the project I had been looking at doing something MUD-like in Python for a good while. I had looked over the various existing Python code bases at the time and found them all to be either abandoned or very limited. I had a few week's stunt working with pymoo before asking myself why I was going through the trouble of parsing a custom script language ... in Python ... Why not use Python throughout? This is when I came upon Evennia. I started making contributions and around 2010 I took over the development as real life commitments forced Greg to step down.

Over the years we have gone through a series of changes. We have gone from using SVN to Mercurial and then to using GIT. We have transited from GoogleCode to GitHub - the main problem of which was converting the wiki documentation (Evennia has extensive documentation).

For a long time we used Python's reload() function to add code to the running game. It worked ... sometimes, depending on what you changed. Eventually it turned out to be so unpredictable that we now use two processes, one to connect clients to and the other running the game, meaning we can completely restart one process without disconnecting anyone.

Back in the day you were also expected to create your own game in a folder game/ inside the Evennia repo itself. It made it really hard for us to update that folder without creating merge conflicts all over. Now Evennia is a proper library and the code you write is properly separated from ours.

So in summary, many things have happened over the years, much of it documented in this blog. With 3500 commits, 28 000 lines of code (+46% comments) and some 25 people contributing in the last year, Openhub lists us as

"A mature, well-established codebase with a stable commit history, a large development team and very well documented source code". 

It's just words compiled by an algorithm, but they still feel kinda good!


While Evennia was always meant to be used for any type of multiplayer text game, this general use have been expanded and cleaned up a lot over the years.

This has been reflected in the width of people wanting to use it for different genres: Over time the MUSH people influenced us into adding the option to play the same character from many different clients at the same time (apparently, playing on the bus and then continuing on another device later is common for such games). Others have wanted to use Evennia for interactive fiction, for hack&slash, deep roleplay, strategy, education or just for learning Python.

Since Evennia is a framework/library and tries to not impose any particular game systems, it means there is much work to do when building a game using Evennia. The result is that there are dozens of games "in production" using Evennia (and more we probably don't know about), but few public releases yet.

The first active "game" may have been an Evennia game/chat supporting the Russian version of 4chan... The community driven Evennia demo-game Ainneve is also progressing, recently adding combat for testing. This is aimed at offering an example of more game-specific code people can build from (the way Diku does). There are similar projects meant for helping people create RPI (RolePlay Intensive) and MUSH-style games. That said, the Evennia-game Arx, After the Reckoning is progressing through beta at a good clip and is showing all signs of becoming the first full-fledged released Evennia game. 


So cheers, Evennia for turning 10. That's enough of the introspection and history. I'll get back to more technical aspects in the next post.

Thursday, October 13, 2016

Season of fixes


The last few months has been dominated by bug-fixing and testing in Evennia-land. A lot more new users appears to be starting to use Evennia, especially from the MUSH world where the Evennia-based Arx, After the Reckoning is, while still in alpha, currently leading the charge.

With a new influx of users comes the application of all sorts of use- and edge-cases that stretch and exercise the framework in all the places where it matters. There is no better test of code than new users trying to use it unsupervised! Evennia is holding up well overall but there are always things that can be improved. 
  • I reworked the way on-object Attributes was cached (from a stupid but simple way to a lot more sophisticated but harder way) and achieved three times faster performance in certain special cases people had complained about. Other issues also came to view while diving into this, which could be fixed.
  • I reworked the venerable batch command and batchcode processors (these allow to create a game world from a script file) and made their inputs make more sense to people. This was one of the older parts of Evennia and apart from the module needing a big refactoring to be easier to read, some parts were pretty fragile and prone to break. Especially when passing it file names tended to be confusing since it understood only certain relative paths to the files to read in and not even I could remember if one should include the file ending or not. This was cleaned up a lot. 
  • Lots of changes and updates were made to the RPSystem contrib, which optionally adds more advanced roleplaying mechanics to Evennia. The use of this in Evennia's demo game (Ainneve, being separately developed) helps a lot for ironing out any remaining wrinkles.
  • Lots and lots of other fixes and smaller feature updates were done (About 150 commits and 50 Issues closed since end of summer).
A fun thing with a growing user base is that we are also starting to see a lot more Pull requests and contributions. Thanks a lot, keep 'em coming!
  • Map system contrib (merged), for creating a world based on ASCII map. Talking about maps, users contributed not just one but two new tutorials for implementing both static and dynamic maps with Evennia. 
  • Webclient notifications (pending), for making the webclient show us in a clearer way when it gets updated in a different tab. A more advanced implementation awaits the webclient being expanded with a proper client-side option window; there is currently a feature request for this if anyone's interested in taking it on.   
  • AI system contrib (pending). This is a full AI backend for adding complex behaviors to game agents. It uses Behavioral trees and is designed to be modified both in code and from inside the game.
  • Action system contrib (pending). This contrib assigns the actions of Characters a time cost and delays the results of commands the given time. It also allows players to go into turn-based mode to enforce a strict action order. 
  • Lots of now closed PRs were contributed by the Arx lead developer to fix various bugs and edge-cases as they came up in live use.
The fixing and tightening of the nuts and bolts will likely continue the remainder of the year. I'm currently working on a refactoring of the way command sets are merged together (see the end of my blog post on Evennia in pictures for a brief summary of the command system). But with so much new blood in the community, who can tell where things will turn from here!

Friday, July 1, 2016

The art of sharing nicks and descriptions

In the month or so since the merger of Evennia's development branch and all its web-client updates, we have been in bug-fixing mode as more people use and stress the code.

There have been some new features as well though - I thought it could be interesting to those of you not slavishly following the mailing list.

 

Shared web login

When you are logged into the website you will now also auto-login to your account in the web client - no need to re-enter the login information! The inverse is also true. You still need to connect to game at least once to create the account, but after that you will stay connected while the browser session lasts.

Behind the scenes the shared login uses cookies linked to server-side Django sessions which is a robust and safe way to manage access tokens. Obviously browser sessions are irrelevant to telnet- or ssh connections.

Extended Nicks 

Evennia's nick(name) system is a way to create a personal alias for things in game - both to on-the-fly replacing text you input and for referring to in-game objects. In the old implementation this replacement was simply matched from the beginning of the input - if the string matched, it was replaced with the nick.

In this new implementation, the matching part can be much more elaborate. For example you can catch arguments and put those arguments into the replacement nick in another order.

Let's say we often use the @dig command this limited way:

@dig roomname;alias = exit;alias, backexit;alias
 
Let's say we find this syntax unintuitive. The new nick system allows to change this by catching the arguments in your nick and put it into the "real" command. Here is an example of a syntax putting the aliases in parentheses and separating all components with commas:

> nick newroom $1($2), $3($4), $5($6) = @dig $1;$2 = $3;$4, $5;$6

From here on you can now create your rooms with entries like this: 

> newroom The great castle(castle), to the castle(castle), back to the path(back)

Multidescer contrib

I have added a new "multidescer" to the contrib folder. A multidescer is (I think) a MUSH term for a mechanism managing more than one description. You can then combine any of these various descriptions into your "active" description.

An example of usage:

desc hat = a blue hat.
desc basic = This is a tall man with narrow features.
desc clothing = He is wearing black, flowing robes.

 These commands store the description on the Character and references them as unique keywords. Next we can combine these strings together in any order to build the actual current description: 

> desc/set basic + |/ + clothing + On his head he has + hat
> look self
This is a tall man with narrow features. 
He is wearing black, flowing robes. On his head he has a blue hat.

This allows for both very flexible and easy-to-update descriptions but also a way to handle freeform equipment and clothing. And you can of course use the nick system to pre-format the output

> nick setdesc $1 $2 $3 $4 = $1 + |/ + clothing + On his head he has a $4

This way you can clothe yourself in different outfits easily using the same output format:

> setdesc basic clothing hat 
 
 The multidescer is a single, self-contained command that is easy to import and add to your game as needed.



... There's also plenty of bug fixes, documentation work and other minor things or course.

Anyway, summer is now upon us here in the northern hemisphere so things will calm down for a bit, at least from my end. Have a good 'un!
.
Griatch

Image by ryancr (released under Creative Commons)

Tuesday, May 31, 2016

Evennia in pictures

This article describes the MU* development system Evennia using pictures!  

This article was originally written for Optional Realities
Since it is no longer available to read on OR, I'm reposting it in full here.

Figure 1: The parts of the Evennia library

evennia_infrastructureA.png

Evennia is a game development library. What you see in Figure 1 is the part you download from us. This will not run on its own, we will soon initialize the missing “jigsaw puzzle” piece on the left. But first let’s look at what we’ve got.

Looking at Figure 1 you will notice that Evennia internally has two components, the Portal and the Server. These will run as separate processes.

The Portal tracks all connections to the outside world and understands Telnet protocols, websockets, SSH and so on. It knows nothing about the database or the game state. Data sent between the Portal and the Server is protocol-agnostic, meaning the Server sends/receives the same data regardless of how the user is connected. Hiding behind the Portal also means that the Server can be completely rebooted without anyone getting disconnected.

The Server is the main “mud driver” and handles everything related to the game world and its database. It's asynchronous and uses Twisted. In the same process of the Server is also the Evennia web server component that serves the game’s website. That the Server and webserver are accessing the database in the same process allows for a consistent game state without any concerns for caching or race condition issues.

Now, let’s get a game going. We’ll call it mygame. Original, isn’t it?

Figure 2: The full setup for mygame
evennia_infrastructure.jpg.png

After installing evennia you will have the evennia command available. Using this you create a game directory - the darker grey piece in Figure 2 that was missing previously. This is where you will create your dream game!

During initialization, Evennia will create Python module templates in mygame/ and link up all configurations to make mygame a fully functioning, if empty, game, ready to start extending. Two more commands will create your database and then start the server. From this point on, mygame is up and running and you can connect to your new game with telnet on localhost:4000 or by pointing your browser to http://localhost:8000.

Now, our new mygame world needs Characters, locations, items and more! These we commonly refer to as game entities. Let’s see how Evennia handles those.

Figure 3: The Database Abstraction of Evennia entities

evennia_typeclassesA.png
Evennia is fully persistent and abstracts its database in Python using Django. The database tables are few and generic, each represented by a single Python class. As seen in Figure 3, the example ObjectDB Python class represents one database table. The properties on the class are the columns (fields) of the table. Each row is an instance of the class (one entity in the game).

Among the example columns shown is the key (name) of the ObjectDB entity as well as a Foreign key-relationship for its current “location”. From the above we can see that Trigger is in the Dungeon, carrying his trusty crossbow Old Betsy!

The db_typeclass_path is an important field. This is a python-style path and tells Evennia which subclass of ObjectDB is actually representing this entity.

Figure 4: Inheriting classes to customize entities

evennia_typeclassesB.png

In Figure 4 we see the (somewhat simplified) Python class inheritance tree that you as an Evennia developer will see, along with the three instanced entities.

ObjectDB represents stuff you will actually see in-game and its child classes implement all the handlers, helper code and the hook methods that Evennia makes use of. In your mygame/ folder you just import these and overload the things you want to modify. In this way, the Crossbow is modified to do the stuff only crossbows can do and CastleRoom adds whatever it is that is special about rooms in the castle.

When creating a new entity in-game, a new row will automatically be created in the database table and then “Trigger” will appear in-game! If we, in code, search the database for Trigger, we will get an instance of the Character class back - a Python object we can work with normally.

Looking at this you may think that you will be making a lot of classes for every different object in the game. Your exact layout is up to you but Evennia also offers other ways to customize each individual object, as exemplified by Figure 5.

Figure 5: Adding persistent Attributes to a game entity.

evennia_typeclassesC.png

The Attribute is another class directly tied to the database behind the scenes. Each Attribute basically has a key, a value and a ForeignKey relation to another ObjectDB. An Attribute serializes Python constructs into the database, meaning you can store basically any valid Python, like the dictionary of skills seen in Figure 5. The “strength” and “skills” Attributes will henceforth be reachable directly from the Trigger object. This (and a few other resources) allow you to create individualized entities while only needing to create classes for those that really behave fundamentally different.


Figure 6: Sessions, Players and Objects

evennia_session_player_character.png

Trigger is most likely played by a human. This human connects to the game via one or more Sessions, one for each client they connect with. Their account on mygame is represented by a PlayerDB entity. The PlayerDB holds the password and other account info but has no existence in the game world. Through the PlayerDB entity, Sessions can control (“puppet”) one or more ObjectDB entities in-game.

In Figure 6, a user is connected to the game with three Sessions simultaneously. They are logged in to their Player account Richard. Through these Sessions they are simultaneously puppeting the in-game entities Trigger and Sir Hiss. Evennia can be configured to allow or disallow a range of different gaming styles like this.

Now, for users to be able to control their game entities and actually play the game, they need to be able to send Commands.

Figure 7: Commands are Python classes too

evennia_commandsA.png

Commands represent anything a user can input actively to the game, such as the look command, get, quit, emote and so on.

Each Command handles both argument parsing and execution. Since each Command is described with a normal Python class, it means that you can implement parsing once and then just have the rest of your commands inherit the effect. In Figure 7, the DIKUCommand parent class implements parsing of all the syntax common for all DIKU-style commands so CmdLook and others won’t have to.

Figure 8: Commands are grouped together in sets and always associated with game entities.

evennia_commandsB.png

Commands in Evennia are always joined together in Command Sets. These are containers that can hold many Command instances. A given Command class can contribute Command instances to any number of Command Sets. These sets are always associated with game entities. In Figure 8, Trigger has received a Command Set with a bunch of useful commands that he (and by extension his controlling Player) can now use.

Figure 9: Command Sets can affect those around them

evennia_commandsC.png

Trigger’s Command Set is only available to himself. In Figure 8 we put a Command Set with three commands on the Dungeon room. The room itself has no use for commands but we configure this set to affect those inside it instead. Note that we let these be different versions of these commands (hence the different color)! We’ll explain why below.

Figure 10: The name Command “Set” is not just a name

evennia_commandsD.png

Command Sets can be dynamically (and temporarily) merged together in a similar fashion as Set Theory, except the merge priority can be customized. In Figure 10 we see a Union-type merger where the Commands from Dungeon of the same name temporarily override the commands from Trigger. While in the Dungeon, Trigger will be using this version of those commands. When Trigger leaves, his own Command Set will be restored unharmed.

Why would we want to do this? Consider for example that the dungeon is in darkness. We can then let the Dungeon’s version of the look command only show the contents of the room if Trigger is carrying a light source. You might also not be able to easily get things in the room without light - you might even be fumbling randomly in your inventory!

Any number of Command Sets can be merged on the fly. This allows you to implement multiple overlapping states (like combat in a darkened room while intoxicated) without needing huge if statements for every possible combination. The merger is non-destructive, so you can remove cmdsets to get back previous states as needed.



… And that’s how many illustrations I have the stamina to draw at this time. Hopefully this quick illustrated dive into Evennia helps to clarify some of the basic features of the system!