Sunday, October 1, 2017

Evennia in Hacktoberfest 2017

Evennia, the Python MUD/MUSH/MU* creation library participates in the Hacktoberfest 2017 (sign up on that page)! Hacktoberfest is open for all open-source projects like ours. After registering, if you make at least four Pull Requests to a public repo on Github during October (need not just be to Evennia), you win a limited-edition T-shirt!

The help Evennia out and get your T-Shirt, look at our Issue Tracker. I have marked some issues with "Hacktoberfest" but you could take on any issue you want. Take a look in particular at the Unit test issue if you are looking to get into contributing on a smaller scale while helping us tremendously.

If you have any questions on contributing (or it's your first time making a Pull Request), don't be shy to drop into #evennia on irc.freenode.net or ask in our forum/mailing list. Have fun!

Wednesday, September 20, 2017

Evennia 0.7 released

As of today Evennia 0.7 is officially out! Evennia is a Python framework and -server for creating text-based multiplayer games (MU*).

A big thank you to all collaborators that have helped with code and testing along the way!

Here is the hefty forum post that details how you migrate to Evennia 0.7.

Evennia 0.7 comes with a range of changes and updates (these are just the ones merged from the latest devel branch, a lot more has happened since 0.6 that were already in master): 
 
  • Evennia separates Player objects from Character objects in that the former is an OOC entity that can puppet one or more of the latter. The name Player was a source of some confusion since this is used differently in other code bases. Player has now been renamed to Account to make its function clearer.
  • Evennia's in-built website now uses our own theme and bootstrap under the hood to make it easier to modify as well as rescale it better on mobile devices (webclient is not updated at this point).
  • Shared logins between webclient and website, with the ability to log out of each independently of the other if so desired.
  • Prefix-ignoring - All default commands are now renamed without their @-prefixes. Both @examine, +examine or examine will now all point to the same command. You can customize which prefixes Evennia simply ignores when searching for a command. The mechanic is clever though - if you create a command with a specific key "+foo", then that will still work and won't clash with another command named just "foo".
  • Easy pause mechanisms using yield statements directly in Command code (just do yield 10 in your command code to have it pause ten seconds without blocking anyone else). You can also do retval = yield "Do you want to accept?" and have the command pause (non-blocking) for the player's input before continuing.
New contribs (optional plugins) (selection, new since 0.6 but many have been available on master branch for a while):
  • The optional In-Game-Python contrib by Vincent le Geoff allows for coding and scripting in-game using full-fledged Python to make events and triggers. It's not a safe softcode-style language (you have the full power of Python which is not good for untrusted users) but is intended for trusted builders to make complex scripting from the command line.
  • Wilderness/maps - Creation of dynamic or static maps based on ascii-maps for dynamically creating rooms and to show when moving around. (titeuf87, Cloud Keeper)
  • Full turn-based combat system, meant to expand for a desired system (battlejenkins)
  • Alternative Unix-style command base for Evennia, for those that prefer to enter commands like you do on the unix command line (with --flags etc) (Vincent le Geoff)
  • Multidescer, which together with Evennia's nick replacement system can be heavily customized in-game (me).
  • Mail - a @brandymail-style in-game mail-system (grungies1138)
  • Clothing system, for layered outfits adding to the wearer's desc when worn (battlejenkins).
A more technical list from the main announcement post: 
  • EvMenu formatting functions now part of class - EvMenu no longer accepts formatting functions as inputs, these are now part of the EvMenu class. To override the formatting of EvMenu nodes you should now override EvMenu and replace the format method you want. This brings EvMenu more in line with other services in Evennia, all of which are built around overriding. 
  • Scripts are now valid message senders - Also Scripts can now act as "sender" of a Msg, for example to a Channel or a player.
  • Due to the new prefix-ignoring above, @desc (builder-level description-setting) was renamed to setdesc to make it clearly separate from desc (used by a player setting their own description). This was actually the only clash we had to resolve this way in the default commands. 
  • Permission Hierarchy names change - To make Evennia's permission hierarchy better reflect how Evennia actually works, the old Players, Player Helpers, Builders, Wizards, Immortals hierarchy has changed to Player, Helper, Builder, Admin, Developer. Singular/Plural form is now ignored so Builder and Builders will both work (this was a common source of simple errors) Old permissions will be renamed as part of the migration process. The distribution of responsibilities has not changed: Wizards (which in some other systems was the highest level, which confused some) always had the power to affect player accounts (like an admin) whereas Immortals had server-level access such as @py - that is, they were developers. The new names hopefully makes this distinction clearer.  
  • All manager methods now return querysets - Some of these used to return lists which was a throwback to an older version of Typeclasses. With manager methods returning querysets one can chain queries onto their results like you can with any django query.
  • PrettyTable was removed from the codebase. You can still fetch it for your game if you prefer (it's on pypi). But EvTable has more functionality as well as color-support. 
  • Add **kwargs support to all object at_* hooks - All object at_ hooks, such as at_look, at_give etc now has an additional **kwarg argument. This allows developers to send arbitrary data to those hooks so they can expand on them without changing the API.  
  • Remove {-color tags - The use of {r, {n etc was deprecated for years and have now been completely removed from Evennia's core in favor of only one form, namely |r, |n etc. However, the color parser is now pluggable and the {-style as well as the %c style color tags etc can be re-added to your game since they are now in a contrib.  
  • Updated hooks for say/whisper commands - There are now at_before/after_say sub-hooks to allow say to be more easily customized. There is still ongoing discussion on the best way to handle this (see e.g. this PR).  
  • More compact distribution of ports - Before, Evennia distributed its ports widely, such as using 4000, 4001, 8000, 8001, 5001, 8022, some of which appeared close but actually was unrelated in functionality. They were also scattered throughout the settings file. The port definitions have now all moved more closer together, at the top of the settings file. Evennia will now use ports 4000-4006 by default (fewer depending on which protocols you want to start). This should make it easier to assign port ranges when wanting to run multiple Evennia instances on the same machine.  
  • Change how auto-login works - It used to be that once you logged into the website, every time you went to the webclient you would auto-login, even if you manually quite the webclient before. Only way to really log out (to change account, say) would be to first log out of the website. Now this will work more intuitively - you will still auto-login but if you log out of the webclient you "decouple" from the login state of the website and need to re-login to the webclient next time, even if you are still logged into the website.  
  • Better idle disconnects - This is a long-running pet peeve for many. Idle timeouts will now honor a particular lock. Setting this lock allows selected entities (for example staff and bots) to avoid getting timed out like everyone else.
  • A slew of bug fixes and other tweaks. See here for the list.

    Friday, August 25, 2017

    Renaming Django's Auth User and App

    Now that Evennia's devel branch (what will become Evennia 0.7) is slowly approaching completion, I thought I'd try to document an aspect of it that probably took me the longest to figure out. One change in Evennia 0.7 is that the Django model named "Player" changes name to "Account". Over time it has become clear that the old name didn't properly reflected the intention of the model. Sounds like a simple change, right? Well, it was not.

    Briefly on migrations


    First some background. A Django migration is a small Python file sitting in migrations/ sub folders throughout Evennia. A migration describes how our database schema changes over time, so as we change or update fields or add new features we add new migrations to describe how to go from the old to the new state. You apply them with the `evennia migrate` command we sometimes ask you to run. If we did not supply migrations, anyone with an existing Evennia database would have to either start from scratch or manually go in and tweak their database to match every upstream change we did.

    Each migration file has a sequential number, like migration_name.0002.py etc. Migrations will run in order but each migration can also "depend" on another. This means for example that a migration in the player/ folder (application) knows that it need to wait for the changes done by a migration in the typeclasses/ folder/app before it can run.  Finally, Django stores a table in the database to know which migrations were already run so to not apply them more than once.

    For reference, Evennia is wrapping the django manage commands, so where I refer to evennia migrate below you would use manage.py migrate in most Django applications.

    Our problem


    So I wanted to migrate the database change "Rename the Player model to Account". We had to do this under a series of additional constraints however:

    1. The Player is the Django Auth user, storing the password. Such a user Django expects to exist from the beginning of the migration history.
    2. The Player model sits in a application (folder) named "players". This folder should also be renamed to "accounts", meaning any in-migration references to this from other apps would become invalid.
    3. Our migration must both work for old and new users. That is, we cannot expect people to all have existing databases to migrate. Some will need to create the database from scratch and must be able to do so without needing to do any changes to the migration structure.
    4. We have a lot of references to "player" throughout the code, all of which must be renamed.
    5. The migration, including renames, should be possible to do by new users with newbie-level Python and database skills.

     

    Some false starts

    There is no lack of tutorials and help online for solving the problem of renaming a model. I think I tested most of them. But none I found actually ended up addressing these restraints. Especially point 2 in combination with point 3 above is a killer.

    • One of my first tries wiped the migrations table completely, renamed the folder and just assumed Player never existed. This sounds good on paper and works perfectly for fresh databases. But existing databases will still contain the old Player-related models. With the old migrations gone, this is now all wrong and there is no information on how to migrate it.
    • I tried initiating fresh migrations with a player model state so you can move an existing database over to it. But then fresh databases doesn't work instead, since the player folder is gone. Also, you run into trouble with the auth system.
    • I next tried to keep the old migrations (those we know work both for old and new databases) but to migrate it in-place. I did many different attempts at this, but every time one of the restraints above would get in the way.
    • Eventually I wrote raw SQL code in the migrations to modify the database tables retroactively. That is, I basically manually removed all traces of Player in the database where it was, copying things table by table. This was very work-intensive but overall decently successful. With proper error-checking I could get most of the migration to work from old databases as well as for new databases. The trouble was the migrations themselves. No matter how I tried, I couldn't get the migration history to accept what I had done - the dependencies now longer made sense on the database level (since I had manually edited things) and adding new migrations in the future would have been tricky.

    In the end I concluded that I had to abandon the notion that users be able to just do a single migrate command. Some more work would be needed on the user's part.

    The solution


    In the end I needed to combine the power of migrations with the power of Git. Using Git resolved the Gordian knot about the player folder. Basically the process goes like this:

    • First I copied of the `players` folder and renamed it and everything in it to accounts. I added this to settings.INSTALLED_APPS. I also copied the migrations from player and renamed everything in them appropriately - those migrations thus look like Account has always existed - so when you run the migration from scratch the database will be created normally. Note that this also means setting the Account as the auth user from the beginning of the migration history. This would be fine when migrating from scratch except for the fact that it would clash with the still existing Player model saying the same thing. We dodge this problem by the way we run this migration (we'll get to this later).
    • Next I added one new migration in the Account app - this is the migration that copies the data from the player to the equivalent account-named copies in the database. Since Player will not exist if you run this from scratch you have to make sure that the Player model exists at that point in the migration chain. You can't just do this with a normal import and traceback, you need to use the migration infrastructure. This kind of check works:

        # ...

        def forwards(apps, schema_editor):
        try:
            PlayerDB = apps.get_model("players", "PlayerDB")
        except LookupError:
            return

        # copy data from player-tables to database tables here

        class Migrations(migrations.Migration):
        # ...
        operations = [
            migrations.RunPython(forwards, migrations.RunPython.noop)
        ]


    • Now, a lot of my other apps/models has ForeignKey or Many2Many relations to the Player model. Aided by viewing the tables in the database I visited all of those and added a migration to each where I duplicated the player-relation with a duplicate account relation. So at this point I had set up a parallel, co-existing duplicate of the Player model, named Account.
    • I now made sure to commit my changes to git and tag this position in the version history with a clear git tag for later reference. This is an important step. We are saving the historical point in time where the player- and account-apps coexisted.5. The git position safely saved, I now went about purging player. I removed the player app and its folder.
    • Since the player folder is not there, it's migrations are not there either. So in another app (any would work) I made a migration to remove the player tables from the database. Thing is, the missing player app means other migrations also cannot reference it. It is possible I could have waited to remove the player/ folder so as to be able to do this bit in pure Python. On the other hand, that might have caused issues since you would be trying to migrate with two Auth users - not sure. As it were, I ended up purging the now useless player-tables with raw SQL:

        from django.db import connection

        # ...

        def _table_exists(db_cursor, tablename):
        "Returns bool if table exists or not"
        sql_check_exists = "SELECT * from %s;" % tablename
        try:
            db_cursor.execute(sql_check_exists)
            return True
        except OperationalError:
            return False


        def _drop_table(db_cursor, table_name):
        if _table_exists(db_cursor, table_name):
            sql_drop = "DROP TABLE %s;" % table_name
            db_cursor.execute(sql_drop)


        def drop_tables(apps, schema_migrator):
        db_cursor = connection.cursor()
        _drop_table(db_cursor, "players_playerdb")
        _drop_table(db_cursor, "players_playerdb_db_attributes")
        # etc

        class Migration(migrations.Migration):

        # ...

        operations = [
            migrations.RunPython(drop_tables)
        ]

    • Next I continued creating migrations to remove all the "duplicate" foreignkeys I had creater earlier in all other apps that make up Evennia. I committed those.
    • Finally I wrote a little program to rename Python code uses of "player" in to "account" (retaining capitalization, supporting renaming "a player" to "an account" etc). I spent some time on this since I wanted our users to be able to convert their own codes with a decent interface. I include this tool in the Evennia repository.

    And with this, the migration's design was complete. Below is how to actually use it ...

    Running the final migration


    In brief, what our users will do after pulling the latest code is as follows:

    1. If they are starting fresh, they just run evennia migrate as usual. All migrations referencing player will detect that there is no such app and just be skipped. They are finished, hooray!
    2. If they have an existing database, they should make a copy of my renaming-program, then check out  the tagged point in the git history I created above, a time when players and accounts coexisted in code.
    3. Since an important aspect of Evennia is that users create their own game in a "game" folder, the user can now use my renaming program to interactively rename all occurrencies of `player` into `account` (the term 'player' is so ubiquitous that they may have used in in different places they don't want to rename).
    4. Next they run migrations at that point of the git history. This duplicates player data into its renamed counterparts.
    5. Now they should check out the latest Evennia again, jumping back to a point where the players application is  gone and only accounts exists.
    6. Note that our migration history is wrong at this point. It will still contain references to migrations from the now nonexisting players app. When trying to run from scratch, those will fail. We need to force Django to forget that. So the user must here go into their database of choice and run the single SQL statement DELETE FROM django_migations; . This clears the migration history.  This is a step I wanted to avoid (since it requires users to use SQL) but in the end I concluded it must be done (and is hopefully a simple enough instruction).
    7. Next, we trick the database to again think that we have run all the migrations from the beginning (this time without any mention of players). This is done with the --fake switch: evennia migrate --fake . This fake-applies all migrations and stores in the database that they have run.
    8. However, the last few migrations are the ones I added just above. Those actually remove the player-related tables from the database. We really do want to run those. So we fake-fake undo those with evennia migrate --fake typeclasses 0007, which is the application and migration number I used to wipe players. This causes django to forget those migrations so we can run them again.
    9. Finally we run evennia migrate. This runs the previously "forgotten" migrations and purges the last vestigest of players from the database. Done!

     

    Conclusions


    And that's it. It's a bit more involved for the end user than I would have liked, and took me much longer than expected to figure out. But it's possible to reproduce and you only need to do it once - and only if you have a database to convert. Whereas this is necessarily specified for Evennia, I hope this might give a hint for other django users aiming to do something like this!

    Sunday, April 23, 2017

    The luxury of a creative community

    For this blog post I want to focus on the series of very nice pull requests coming in from a growing cadre of contributors over the last few months.

    Contributed goodness

    People have put in a lot of good work to boost Evennia, both by improving existing things and by adding new features. Thanks a lot everyone (below is just a small selection)!
    •  Contrib: Turn-based combat system - this is a full, if intentionally bare-bones implementation of a combat system, meant as a template to put in your particular game system into.
    • Contrib: Clothing sytem - a roleplaying mechanic where a character can 'wear' items and have this show in their descriptions. Worn items can also be layered to hide that underneath. Had plenty of opportunities for extensions to a given game.
    • Contrib: An 'event' system is in the works, for allowing privileged builders to add dynamic code to objects that fires when particular events happen. The PR is not yet merged but promises the oft pondered feature of in-game coding without using softcode (and notably also without the security of softcode!). 
    • A lot of PRs, especially from one user, dealt with cleanup and adherence to PEP8 as well as fixing the 'alerts' given by LGTM on our code (LGTM is by the way a pretty nifty service, they parse the code from the github repo without actually running it and try to find problems. Abiding by their advice results is cleaner code and it also found some actual edge-case bugs here and there not covered by unit tests. The joint effort has brought us down from some 600+ alerts to somewhere around 90 - the remaining ones are alerts which I don't agree with or which are not important enough to spend effort on). 
    • The help mechanics of Evennia were improved by splitting up the default help command into smaller parts, making it easier to inject some changes to your help system without completely replacing the default one. 
    • Evennia's Xterm256 implementation was not correctly including the additional greyscale colors, those were added with new tags |=a ... |=z.
    • Evennia has the ability to relay data to external services through 'bots'. An example of this is the IRC bot, which is a sort of 'player' that sits in an in-game channel and connects that to a counterpart-bot sitting in a remote IRC channel. It allows for direct game-IRC communication, something enjoyed by people in the Evennia demo for many years now. The way the bot was defined used to be pretty hard-coded though. A crafty contributor changed that though, but incorporating the bot mechanism into Evennia's normal message flow. This allows for adding new types of bots or extending existing ones without having to modify Evennia's core. There is already an alternative IRC bot out there that represents everyone in the IRC room as a room full of people in the MUD. 
    • Evennia's Attributes is a database table connected to other objects via a ForeignKey relation. This relation is cached on the object. A user however found that for certain implementations, such as using Attributes for large coordinate systems, non-matches (that is failed Attribute lookups on the object) can also be cached and leads to dramatic speed increases for those particular use cases. A PR followed. You live and learn.
    • Another contributor helped improve the EvEditor (Evennia's VIM-like in-game text editor) by giving it a code-mode for editing Python code in-game with auto-indents and code execution. Jump into the code mode with the command @py/edit.
    • Time scheduling is another feature that has been discussed now and then and has now been added through a PR. This means that rather than specifying 'Do this in 400 seconds' you can say 'do this at 12AM, in-game time'. The core system works with the real-world time units. If you want 10 hours to a day or two weeks to a month the same contributor also made an optional calendar contrib for that!
    • A new 'whisper' command was added to the Default cmdset. It's an in-game command for whispering to someone in the same room without other people hearing it. This is a nice thing to have considering Evennia is out-of-the-box pretty much offering the features of a 'talker' type of game.
    • Lots of bug fixes big and small!
    • Some at_* hooks were added, such as at_give(giver, getter). This allows for finer control of the give process without handling all the logics at the command level. There are others hooks in the works but those will not be added until in Evennia 0.7. 
    About that Evennia 0.7 ...

    So while PRs are popping up left and right in master I've been working in the devel branch towards what will be the Evennia 0.7 release. The branch is not ready for public consumption and testing yet But tentatively it's about halfway there as I am slowly progressing through the tickets. Most of the upcoming features were covered in the previous blog post so I'll leave it at that.

    I just want to end by saying that it's a very luxurious (and awesome) feeling for me to see master-branch Evennia expand with lots of new stuff "without me" so to speak. The power of Open Source indeed!
      

    Image from http://maxpixel.freegreatpicture.com, released as public domain.

    Sunday, February 5, 2017

    News items from the new year

    The last few months have been mostly occupied with fixing bugs and straightening out usage quirks as more and more people take Evennia through its paces.

    Webclient progress

    One of our contributors, mewser/titeuf87 has put in work on implementing part of our roadmap for the webclient. In the first merged batch, the client now has an option window for adjusting and saving settings. This is an important first step towards expanding the client's functionality. Other  features is showing help in an (optional) popup window and to report window activity by popup and/or sound.

    The goal for the future is to allow the user or developer to split the client window into panes to which they can then direct various output from the server as they please It's early days still but some of the example designs being discussed can be found in the wiki webclient brainstorm (see the title image of this blog for one of the mockups).
     
    New server stuff

    Last year saw the death of our old demo server on horizondark.com, luckily the new one at silvren.com has worked out fine with no hickups. As part of setting that up, we also got together a more proper list of recommended hosts for Evennia games. Evennia requires more memory than your average C code base so this is important information to have. It seems most of our users run Evennia on various cloud hosting services rather than from a traditional remote server login.

    Arx going strong 

    The currently largest Evennia game, the mush Arx - After the Reckoning has helped a lot in stress testing. Their lead coder Tehom has also been active both in reporting issues and fixing them - kudos! There are however some lingering issues which appears rarely enough that they have not been possible to reproduce yet; we're working on those. Overall though I must say that considering how active Arx is, I would have expected to have seen even more "childhood diseases" than we have. 

    Launch scripts and discussions

    It is always interesting with feedback, and some time back another discussion thread erupted over on musoapbox, here. The musoapbox regulars have strong opinions about many things and this time some were critical of Evennia's install process. They felt it was too elaborate with too many steps, especially if you are approaching the system with no knowledge about Python. Apparently the average MUSH server has a much shorter path to go (even though that does require C compiling). Whereas I don't necessarily agree with all notions in that thread, it's valuable feedback - I've long acknowledged that it's hard to know just what is hard or not for a beginner.

    Whereas we are planning to eventually move Evennia to pypi (so you can do pip install evennia), the instructions around getting virtualenv setup is not likely to change. So there is now unix shell scripts supplied with the system for installing on debian-derived systems (Debian, Ubuntu, Mint etc). I also made scripts for automating the setup and launch of Evennia and to use it with linux' initd within the scope of a virtualenv.
    So far these scripts are not tested by enough people to warrant them being generally recommended, but if you are on a supported OS and is interested to try they are found from the top of the Evennia repo, in bin/unix/. More info can be found on their documentation page.

    Docker

    Speaking of installing, Evennia now has an official Docker image, courtesy of the work of contributor and Ainneve dev feend78. The image is automatically kept up-to.date with the latest Evennia repo and allows Evennia to be easily deployed in a production environment (most cloud services supports this). See Docker wiki page for more info.


    Lots of new PRs

    There was a whole slew of contributions waiting for me when returning from Chistmas break, and this has not slowed since. Github makes it easy to contribute and I think we are really starting to see this effect (Google Code back in the day was not as simple in this regard). The best thing with many of these PRs is that they address common things that people need to do but which could be made simpler or more flexible. It's hard to plan for all possibilities, so many people using the system is the best way to find such solutions.

    Apart from the map-creation contribs from last year we also have a new Wildnerness system by mewser/titeuf87. This implements wilderness according to an old idea I had on the mailing list - instead of making a room per location, players get a single room.  The room tracks its coordinate in the wildnerness and updates its description and exits dynamically every time you move. This way you could in principle have an infinite wilderness without it taking any space. It's great to see the idea turned into a practical implementation and that it seems to work so well. Will be fun to see what people can do with it in the future!