Showing posts with label server-development. Show all posts
Showing posts with label server-development. Show all posts

Friday, January 1, 2021

Happy new years 2021! Evennia things to come this year


Another year passed with Evennia, the Python MU* creation system. The past year saw a lot of bug fixing and more gradual additions and in September we released version 0.9.5. This was an intermediary version on our way to 1.0. Time to look forward to next year. 

On my development horizon for 2021 are the main new features planned for v1.0. Some of these are rather big things I've wanted to get around to for a while. They are all happening in the develop branch of Evennia, which is currently not recommended for general use.

  • SessionDB: In the current Evennia, Sessions (the representation of a single client connection) is an in-memory entity. This is changing to be a database-backed entity instead. One will be able to typeclass Sessions like other entities for easier overriding. This change also means that there will be one single point of session-id (the django-session), alleviating some reported issues where the Portal- and Server-side sessions have drifted out of sync. It will also make it a lot easier to support auto-logins, also across server reboots. Db-backed Sessions will also simplify the Portal-Session interaction a lot.  
  • Script refactor: The Scripts will see some refactoring, mainly because they are used more as general-storage entities compared to the timers they were originally meant to be. These days Evennia also offers a range of other timer-mechanisms (tickers, delays, Events etc), so it's less important to rely on Scripts for this functionality. The most important change will be that the timer will required to be explicitly started (instead of always starting on script-creation). It will also be possible to stop the timer without the script getting deleted (so separating the timer from the Script's life-cycle). 
  • Channel refactor: The Channels will also see changes; notably to make it considerably easier to override and customize them per-caller. Today the Channel typeclass has a maze of different hooks being called, but it's harder for devs wanting users to customize their channel output. So one of the changes will be new hooks on the account/object level for allowing to format the channel output per-user. There will also be a cleanup of the existing hooks to make things clearer. 
  • New starting tutorial: As part of the new documentation, I'm writing a new starting-tutorial. This will consolidate many of the existing beginner tutorials in a consistent sequence and if following it to the end, the reader will have created a small beginner game with everything in place. I plan to make a few new contribs to support this.
  • Contrib restructure: Our contrib/ folder is getting a little cluttered. I'm investigating organizing things a little differently by at least moving things into categorized folders. This will lead to people having to change their imports, but we'll see just how it goes.
  • Documentation cleanup: There are a lot of small changes, cleanup and restructuring needed in the docs overall - many of the existing pages are auto-translated from the old wiki and need rewriting both in style and content. The whole idea of moving to the new doc-system is to be able to update the docs alongside the code changes. So hopefully the changes to Sessions, Scripts and Channels etc will all be covered properly from the onset rather than after release (as was the case with the wiki). 
  • Unittest coverage: Our current test coverate is 64%, we need to expand this. I hope to get to at least 70% before v1.0 but that is less of a strict goal.
  • Evennia PYPI package: This will be one of the last things before the release of 1.0 - Evennia will be put onto PYPI  so you can install with pip install evennia. Once we do it will simplify the install instructions dramatically for those not interested in contributing to Evennia proper.
We also have some pull-requests in the making that will be interesting to have in the system, such as Volund's plugin system, making it easier to inject custom settings on the fly (good for contribs wanting to add their own database tables, for example). 

A lot of work to do as usual! 

Thanks for the past year, everyone - thanks to all of you that contributed to Evennia with code or by reporting issues or coming with feedback. Thanks particularly to those of you willing (and able) to chip in with some financial support for Evennia development - that's very encouraging! 
And finally, a big thanks to all of you taking part in community discussions, both you asking questions and you helping to answer them tirelessly and with good cheer - you make the Evennia community the friendly, inviting place it is!

May all our game development move forward and our hard drives stay healthy for another year. 

Cheers and a Happy new year,
Griatch

Saturday, November 14, 2020

Evennia 0.9.5 released!

 

As of today, Evennia 0.9.5 is out. Evennia is a Python based library and framework for creating text-based multiplayer games (MUD/MU*). 

This is a gradual improvement halfway between 0.9 and the upcoming 1.0. So if you have been keeping up-to-date with the master branch of Evennia over the last year you will not notice much difference from this release (time to upgrade if you haven't been keeping up though!). 

While an interim release, there are still a lot of things that has happened since v0.9: 

Webclient improvements

  • Big web client improvements (courtesy of contributor friarzen) - players can now save and restore pane layouts directly in the client (so you could have a separate pane for channel chatter, another for look-returns, two input panes etc etc). 
  • The layout changes makes it easier for devs to create default layouts to offer to players of their game. People in the Evennia community have already started doing very cool stuff with this, I'll try to gather screenshots for a future blog.
  • Allow to redirect video/music to separate panes.
  • Many other fixes, such as improving the input-history behavior.

EvMenu improvements

EvMenu is a powerful system for creating in-game text menus.

  • The EvMenu class was refactored to be easier to override. For example, all messages now go through EvMenu.msg which allows for easy customization. It also defaults to sending with a type of "menu", making it easier to redirect menus to seprate panes in the webclient.
  • In a node, EvMenu is now accessed via the much more logically named .ndb._evmenu instead of .ndb._menutree (the old name still works for backwards compatibility, but is deprecated).
  • New optional EvMenu template system for quickly building simpler EvMenus without needing so much code. This makes it easy to catch and parse arbitrary input from the user and redirect to the correct node as needed. Creating menu nodes as functions still work (and is a lot more powerful), this can be mixed with templating to create different effects.

New settings

  • INLINEFUNC_STACK_MAXSIZE is an integer that allows to control how big the inlinefunc nesting stacksize is.
  • New DEFAULT_CHANNELS setting to allow customization of which channels should be initialized on startup. This can be modified after initial server start.
  • CHANNEL_HANDLER_CLASS allows for specifying an alternative to the default ChannelHandler if wanting to change how Channels behave. 
  • SERVER_LOG_DAY_ROTATION defines how many days the server log should run before being force-rotated (default is seven days).
  • SERVER_LOG_MAX_SIZE specifies how big the log must be before it auto-rotates (even if SERVER_LOG_DAY_ROTATION days has not passed yet). 
  • PORTAL_LOG_DAY_ROTATION, PORTA_LOG_MAX_SIZE - equivalent for the Portal.

 Other improvements

  • The EvMore pager saw big performance speedups, making the viewing of large numbers of entries much snappier. You can now also paginate EvTables directly and create custom pagers by override the EvMore class (useful if you want to e.g. do a EvTable per-page).   
  • Improvement to the multi-match parser: Trying to get for example 3-box will now fail with a no-found if there are only two boxes in the room (before it would show the multi-match menu).  
  • New inside_rec lockfunc to recursively check if an object is inside another. Putting this on a room will thus also check the contents of any objects in the room, not only the contents themselves. Or if you had something in your wallet (a container).
  • New $random inlinefunc for producing a random number in strings.
  • TickerHandler.add() now returns a store_key to uniquely describe the ticker just added. The TickerHandler.remove() accepts a new kwarg store_key for removing the ticker - this makes it easier to manage tickers instead having to insert the full specifications of the ticker to remove it.
  • Many fixes to the spawn command and prototype functionality. The new spawn/raw flag will now return the prototype-dict so one can manually edit and copy&paste it.
  • The evennia.GLOBAL_SCRIPTS container will now contain all global scripts, not only those explicitly created with the GLOBAL_SCRIPTS setting. 
  • The list_to_string utility converts a list to a nice string-representation, such as ["a", "b", "c", "d"] -> "a, b, c and d". The function is renamed to iter_to_string (but old name still works) and now also works with generators and will not crash even when provided a single value.
  • A lot of bug fixes and stability fixes!

See the changelog here.

 

New documentation system 

The bigger change with 0.9.5 is that we are moving to a new documentation system. The details of the long road to do this is documented in my prervious post. The point is that we are stopping the use of the Github wiki in favor of statically generated documentation hosted on github pages. At the same time we also move the old evennia.com website from Google-sites to Github. 

Check it out: 

 As for the docs, they will be maturing for a long time still. The old wiki will not be updated anymore, but it will also not be going anywhere in the short term. Version 0.9.5 of the docs is pretty much a copy of the wiki and I hope to not have to spend too much more work maintaining it since the wiki is still around.
 
New updates and documentation features will primarily be happening in the 1.0-dev version of the documentation. This will include refactoring all pages as well as a new intro-tutorial and many other things. 
 
But that's for future blogs ...


 


Tuesday, October 20, 2020

On using Markdown with Sphinx - onward to Evennia 0.9.5

Last post I wrote about the upcoming v1.0 of Evennia, the Python MU* creation engine. We are not getting to that 1.0 version quite yet though: The next release will be 0.9.5, hopefully out relatively soon (TM).

Evennia 0.9.5 is, as you may guess, an intermediary release. Apart from the 1.0 roadmap just not being done yet, there is one other big reason for this - we are introducing documentation versioning and for that a proper release is needed as a base to start from. Version 0.9.5 contains everything already in master branch, so if you have kept up-to-date you won't notice too much difference. Here are some highlights compared to version 0.9:

  • EvMore will paginate and properly handle both EvTables and database query output. For huge data sets, pagination can give a 100-fold speed-increase. This is noticeable e.g. in the scripts and spawn/list commands, once you have a lot of items.
  • EvMenu templating language, to make it easier to create simpler menus. 
  • Webclient improvements: Cleanup of interface and the ability for players to save/load their pane layouts from the client. The developer can still provide a default for them to start out with.
  • MUD/Evennia Intro wizard to the tutorial world to explain basic game controls in an interactive way. 
  • Default channels can now be defined in settings instead of having to do so from in-game. 
  • New documentation system (see below).
  • Many, many bug fixes and optimizations!

Many contributors helped out along the way. See the changelog where contributors of the bigger new features are listed.

The path to a new documentation

For many years we've used the Github wiki as our documentation hub. It has served us well. But as mentioned in my previous post, it has its drawbacks, in particular when it comes to handling documentation for multiple Evennia versions in parallel.

After considering a bunch of options, I eventually went with sphinx, because it has such a good autodoc functionality (parsing of the source-code docstrings). This is despite our wiki docs are all in markdown and I dislike restructured text quite a bit. Our code also uses friendly and in-code-readable Google-style docstrings instead of Sphinx' hideous and unreadable format. 

Luckily there are extensions for Sphinx to handle this: 

  • Napoleon to convert Google-style docstrings to reST on the fly
  • recommonmark to convert our markdown wiki pages to reST on compile-time
  • sphinx-multiversion to merge docs from one or more GIT branches into a documentation where you can select between the versions.

What could go wrong? Well, it's been quite a ride.

Getting Markdown into reST

Linking to things in recommonmark turned out to be very flaky. I ended up forking and merging a bunch of PRs from the project but that was not enough: Clearly this thing was not built to convert 200 pages of technical markdown from a github wiki.

My custom fork of recommonmark had to be tweaked a bit for my needs, such as not having to specify the .md file ending in every link and make sure the url-resolver worked as I expected. There were a bunch of other things but I will probably not merge this back, the changes are pretty Evennia-specific.

Even so, many of my wiki links just wouldn't work. This is not necessarily recommonmark's fault, but how sphinx works by grouping things into toctrees, something that the Evennia wiki doesn't have. 

Also, the recommonmark way to make a toctree in Markdown is to make a list of links - you can't have any descriptive text, making the listing quite useless (apparently people only want bland lists of link-names?). After trying to figure out a way to make this work I eventually capitulated - I make pretty lists in Markdown while using a "hidden" toctree to inform sphinx how the pages are related.

Getting the wiki into the new doc site

This required more custom code. I wrote a custom importer that reads the wiki and cleans/reformats it in places where recommonmark just dies on them. I also made a preprocessor that not only finds orphan pages but also builds a toctree and remaps all links in all documents to their actual location on compilation. The remapper makes it a lot easier to move things around. The drawback is that every page needs to be uniquely named. Since this was already the case in the wiki, this was a good tradeoff. So with a lot of custom code the wiki eventually could port automatically.   

The thing is, that even with all this processing, recommonmark doesn't support stuff like Markdown tables, so you still have to fall back to reST notation for those. And Napoleon, while doing a good job of parsing google docstrings, do not expect Markdown. So the end result is mostly markdown but we still have to fall back to reST for some things. It's probably as far as we get.

Deploying the docs 

Figuring out how to build and deploy these components together was the next challenge. Sphinx' default Makefile was quite anemic and I also wanted something that regular contributors could use to test their documentation contributions easily. I ended up having to expand the Makefile quite a lot while also adding separate deploy scripts and interfaces to github actions (which we recently started using too).

Finally, the versioning. The sphinx-multiversion plugin works by extracting the branches you choose from git and running the sphinx compiler in each branch.  The plugin initially had a bug with how our docs are located (not at the root of the package) but after I reported it, it was quickly fixed. The result is a static document site where you can select between the available versions in the sidebar.

I've not gotten down to trying to make LaTeX/PDF generation work yet. I'm dreading it quite a bit... 

Where we are

The github wiki is now closed for external contributions. The v0.9.5 of the new documentation will pretty much be an import of the last state of the wiki with some minor cleanup (such as tables). While we'll fix outright errors in it, I don't plan to do many fixes of purely visual glitches from the conversion - the old wiki is still there should that be a problem.

The main refactoring and cleanup of the documentation to fit its new home will instead happen in v1.0. While the rough structure of this is already in place, it's very much a work in progress at this point.

Conclusions

Evennia 0.9.5 has a lot of features, but the biggest things are 'meta' changes in the project itself. After it is out, it's onward towards 1.0 again!


Tuesday, April 14, 2020

Spring updates while trying to stay healthy

So, spring grows nearer for those of us on the Northern hemisphere. With everyone hopefully hunkered down and safe from the Covid-19 pandemic, I thought it overdue to make another dev blog for the progress of Evennia, the Python MU*-creation system.


The last few months have seen primarily bug fixing on the Evennia front, but it also has seen an uptick of PRs from the community and the re-opening of the develop branch in earnest. There is still quite a lot of work to do before we can add that extra 0.1 and go from version 0.9 to 1.0.

What's in a version?


For me personally, I never put much stock in the notion of versions. Evennia didn't even have versions until a few years back: We used to just have a rolling git release. But eventually it became clear that our user base was big enough that we needed to more clearly separate major (and possibly breaking) updates from what came before. So I started versioning at Evennia 0.5 and have had roughly a new release every year since (not a plan or a promise, it just happened to turn out that way).

Evennia has been useful (and been used) for game development for many years already. But there is no denying that a 1.x label tends to convey more confidence in a system than a 0.x label, that's just the way things are. So while the new version is still quite some way off, there are a bunch of changes and improvements that we want to do in this release to mark the version change in a good way.

Documentation changes


Our documentation will move away from our trusty Github wiki. Instead we will convert the wiki into a static github page built from sources inside evennia/docs/.

The advantage of the wiki is that it is a very low entry for people to contribute and fix things using Github's editing system. We have had a lot of use of this over the years and the wiki has served us well. The drawbacks are starting to get ever more noticeable, however:
  • Whereas the wiki is itself version-controlled, we cannot show multiple versions of the wiki at the same time. This makes it hard to update the documentation at the same time as non-released code is being written. This is probably my main reason for doing the change.
  • The wiki today consists of some 200+ pages. It is hard to get an overview of what is where and what needs to be updated. 
  • The wiki word-search functionality is not really great.
  • It's impossible to review changes before they go live, to check consistency and style. This has led to some documentation pages overlapping.
  • Building the documentation to local HTML or PDF is an archaic process that I doubt anyone but me has done with any regularity. 
The change so far planned is to switch to the Sphinx documentation build-system (same as Python/Django etc is using). We will use it with extensions that allows us to still use Markdown like in the old wiki. This also allows us to build a more comprehensive (and pretty) API documentation of the entire library. We have more options to add comprehensive online search functionality in this solution as well.

Furthermore, will hopefully be able to set it up so that we can maintain and publish separate documentations for each forthcoming release. That is, you should be able to read the docs for 1.0, 1.1 or the latest master development as you like (similarly to how Django does it, although probably not as fancy from the onset). 

This means that contributions to the documentation will be done as PRs through GitHub, just like when contributing any other code. While this does add a little more of a hurdle to contributions, hopefully the benefits will far outweigh those. Building the docs locally will not require a running Evennia server (unless you want the api docs) and we will try to set everything up for to make it easy to contribute.

Many of the details around the docs are still up in the air. This is still very much work-in-progress, like everything else.

Work with this has started in the static-file-docs branch of Evennia. But we have not closed the wiki either - the two will exist in parallel for now.

PyPi


As mentioned before,  we will finally start to distribute Evennia via PyPi (the Python Package Index) - that is, you will be able to run `pip install evennia`. Using GIT will no longer be a requirement to get started.

Considering how quickly people in open-source throw up their three lines of code on PyPi these days, it may be surprising Evennia is not already on PyPi. I have however felt that reading and referencing the highly-commented code is a big part and requirement for getting the most out of the library.

With the new documentation system, this would improve. And you can of course still use git and follow master branch like the good ol' days if you want!

Web Admin improvements


For the longest time, the Django-admin component has been somewhat on the back-burner. With the help of community contributors, this is improving so you will be able to do more work the Admin GUI related to creating and managing objects, tie puppets to Accounts etc.

API improvements 


Whereas the last few months have been mostly spent fixing lingering bugs, one thing planned for version 1.0 is a general cleanup of legacy strangeness in the API. For example, certain methods can return a list or a single object depending situation, which makes it hard to generalize. There are a lot of small quirks like that which we hope to address and make more consistent.

There has also been a recent flurry of contributor PRs intended to help decouple Evennia's systems and make them easier to replace for those inclined to do so. Many of this is still being worked on, but it's likely you'll be able to replace many more "core" components for 1.0 with your own variations without doing any hacking in upstream code at all.
 ... Needless to say, this is an advanced feature that most developers will never need. But Evennia was always intended to be heavily customizable and having the option is a good thing!

Another feature that will come for 1.0 is a REST-API, added by one of our contributors. This uses Django-REST-Framework and makes it easier for external programs to authenticate and fetch data out of the Evennia database (great both for external apps, websites or custom what-have-you).
At this time you can only fetch database objects via this interface, you cannot perform Command-calls or "play the game" this way (REST is a stateless concept and Evennia Commands tend to retain state).

Many other fixes and contributions


There's a truckload of stuff already available in master branch, but with the latest contributions of bigger code changes, we have started to use the Evennia develop branch again in earnest again. For a summary of the changes so far, check out the Changelog

However, unless you want to contribute to Evennia itself (or really, really want to be on the bleeding edge), you are still recommended to use the master branch for now. A lot of work still to do, as said.


Image ©George Hodan, released under CC0 Public Domain

Monday, September 30, 2019

Blackifying and fixing bugs

Since version 0.9 of Evennia, the MU*-creation framework, was released, work has mainly been focused on bug fixing. But there few new features also already sneaked into master branch, despite technically being changes slated for Evennia 1.0.




On Frontends

Contributor friarzen has chipped away at improving Evennia's HTML5 web client. It already had the ability to structure and spawn any number of nested text panes. In the future we want to extend the user's ability to save an restore its layouts and allow developers to offer pre-prepared layouts for their games. Already now though, it has gotten plugins for handling both graphics, sounds and video:

Inline image by me (griatch-art.deviantart.com)


A related fun development is Castlelore Studios' development of an Unreal Engine Evennia plugin (this is unaffiliated with core Evennia development and I've not tried it, but it looks pretty nifty!): 

Image ©Castlelore Studios
 
On Black

Evennia's source code is extensively documented and was sort of adhering to the Python formatting standard PEP8. But many places were sort of hit-and-miss and others were formatted with slight variations due to who wrote the code.
 
After pre-work and recommendation by Greg Taylor, Evennia has adopted the black autoformatter for its source code. I'm not really convinced that black produces the best output of all possible outputs every time, but as Greg puts it, it's at least consistent in style. We use a line width of 100.

I have set it up so that whenever a new commit is added to the repo, the black formatter will run on it. It may still produce line widths >100 at times (especially for long strings), but otherwise this reduces the number of different PEP8 infractions in the code a lot.

On Python3

Overall the move to Python3 appears to have been pretty uneventful for most users. I've not heard almost any complaints or requests for help with converting an existing game.
The purely Python2-to-Python3 related bugs have been very limited after launch; almost all have been with unicode/bytes when sending data over the wire.

People have wholeheartedly adopted the new f-strings though, and some spontaneous PRs have already been made towards converting some of Evennia existing code into using them.

Post-launch we moved to Django 2.2.2, but the Django 2+ upgrades have been pretty uneventful so far.Some people had issues installing Twisted on Windows since there was no py3.7 binary wheel (causing them to have to compile it from scratch). The rise of the Linux Subsystem on Windows have alleviated most of this though and I've not seen any Windows install issues in a while.

On Future

For now we'll stay in bug-fixing mode, with the ocational new feature popping up here and there. In the future we'll move to the develop branch again. I have a slew of things in mind for 1.0. 

Apart from bug fixing and cleaning up the API in several places, I plan to make use of the feedback received over the years to make Evennia a little more accessible for a new user. This means I'll also try reworking and consolidating the tutorials so one can follow them with a more coherent "red thread", as well as improving the documentation in various other ways to help newcomers with the common questions we hear a lot. 

The current project plan (subject to change) is found here. Lots of things to do!



Top image credit: https://www.goodfreephotos.com/ (public domain)

Thursday, April 25, 2019

Steaming on, eating jam

Image credit: The Smithsonian
In the last few months, development of the upcoming Evennia 0.9 has been steaming on. Evennia is, as you may know, a Python library for creating text-based multiplayer games (MUDs, MUSH etc).
But it's not all backend work! There is also some sweet game-jamming going on, I get to that at the end. 


Evennia progress towards Python3 

The regular Evennia develop branch is now running completely in Python 3. Since we are using some new features of this Python release, we will be aiming for Python 3.7 as a minimum version once Evennia 0.9 goes stable. We will also use Django 2.1 and likely Twisted 19 - so we'll be pretty much up-to-date on all our main dependencies.

Now, while the release of Evennia 0.9 is still some time away (there are a bunch of regular bug fixes and minor features that I want to get in there too (see the progress here on the github 0.9 project page), it's worth to consider how much work it'll be for you to migrate and if you should wait or jump in right now.

If you are new, I still recommend you use regular master branch Evennia (using Python 2.7). This is for which all wiki articles and documentation online is currently written after all. Once we move to python3, you'll need to convert your code ... but syntactically the two are really not that different and conversion should not be much of an issue.

Not only are there automatic converters for most stuff, you should only need to do one pass to make sure things work and then you'll be done. This article is pretty old but it serves well to identify the main differences. Later Py3 versions just adds new stuff which you would just not have had access to in Python2.7. Once 0.9 is released, we'll also make guides for how you go about converting existing code (apart from the wealth of info on this topic online).
 
That said, if you are feeling more adventurous, or really want to use Python3 right away, many of the Evennia developer regulars have been running and testing develop branch for months now. It's overall been a pretty painless transition - as said, py3 is not that different from py2.

However, develop branch has other features beyond the py3 jump. And those are not yet documented in the main docs. So you'll have to contend with that (but asking in chat/forum works of course). However while it works fine for development already, as a matter of principle I do not recommend ever using Evennia's develop branch for a production game - it can change quickly and may occationally be broken. You have been warned! 

Game Jam

We are now a little more than a week into the Mud Coders Guild's second yearly Game Jam. This year's theme is "One Room".

I really hope some more Evennia devs jump on this one because Evennia is perfect for this kind of experimental games. This is because Evennia is not imposing a game style on you - while there are default commands you are free to replace or customize the commands, objects and every aspect of your game to be as specific as you want to your game.

For example, I have a small Game Jam contribution in the works, where I very quickly reworked the default Evennia setup to pretty much make it into a different genre of game.

Usually the systems I make for Evennia are generic and intended for any game-type. By contrast, making somehing highly niched is super-fast to do: Building a whole new framework for game mechanisms along with build helpers for me to quickly make content took no more than three days.

There will no doubt be some iteration, but I hope to spend the rest of the jam time on content and gameplay. I have some RL things happening in the coming weeks (including work on Evennia proper) but if I can get the time to tie my jam entry together, I'll likely make one or two blog posts about how it was developed and my reasons for making the choices i did. Most likely the code will appear as an Evennia contribution in case people want to do something similar for their own projects.

So, busy days.

Wednesday, January 2, 2019

Into 2019!

A new year has come around and it's time to both look back at the old and onward to the future of Evennia, the Python MUD creation system!

 

 

 

 

 

Last year 

Last year saw the release of Evennia 0.8. This version of Evennia changes some fundamental aspects of the server infrastructure so that the server can truly run in daemon mode as you would expect (no more running it in a GnuScreen session if you want to see logging to the terminal). It also adds the new Online Creation System, which lets builders create and define prototypes using a menu system as well as big improvements in the web client, such as multiple window-panes (allows the user to assign text to different windows to keep their client uncluttered) as well as plenty of fixes and features to help ease life for the Evennia developer. Thanks again to everyone who helped out and contributed to the release of Evennia 0.8!

On a personal note, I spoke about Evennia at PyCon Sweden this December, which was fun. I might put up my talk and make a more detailed blog post about that in the future, but my talk got a surprising amount of attention and positive feedback. Clearly many people have fond memories of MUDs and enjoy seeing they are not only still around but are possible to create in Python!

This year

Now we are steaming ahead towards Evennia 0.9! Already we have had a large number of contributions towards this release. A coming change is a much improved central way to make typeclass information available to a website as well as central ways to do various common operations like creating characters and linking them to Accounts. Much of this functionality is today hidden in the various default commands. In Evennia 0.9 they will be moved into more easily-findable locations as library functions everyone can just trigger without having to copy functionality.

The biggest change for Evennia 0.9 is however that we will now finally move Evennia over to Python 3. This is now possible as all our dependencies are now sufficiently ported. As discussed in the community for a long time, this will be a clean break - we will not offer any mid-way solution of Python2/3 but will drop Python2 support entirely. We will at the same time move to Django 2.+. In Django 0.9 we will however probably not use any special Python3 features yet. It should hopefully not be too difficult for our users to upgrade, but we'll of course publish extensive instructions and help when that time comes.

We will likely support a minimum of Python 3.6 and maybe 3.7.  This work is currently happening in a separate branch develop-py3 which will soonish merge into the current Python2.7-based develop branch and become Evennia 0.9 when it merges into master branch at an unknown time later this year.

There are a slew of other things planned for Evennia 0.9 and you can follow the progress from our project page. If you want to help out you are of course also very welcome. If you are new and are interested in getting your feet wet in helping out with Open-source development, we have a list of good first issues you could jump into.

Onward into the new year!



Fireworks image courtesy: U.S. Air Force photo by Zachary Hada

Sunday, September 30, 2018

Evennia 0.8 released

After about a year of work and almost 540 commits from close to 20 contributors, Evennia 0.8 is out! Evennia is a Python game server for creating text-based multiplayer games (MUDs, Mushes, etc) using Django and Twisted.

Some of the upcoming improvements have been covered by previous dev blogs, such as the completely reworked server infrastructure:




as well as the new Online Creation System that allows for editing in-game prototypes using a powerful menu system:



Other improvements are in the web client, which supports split-panes out of the box. The user can split the output area in any number of panes, resize as desired and then assign different types of content to each pane. You can also have panes that absorb "all the rest" or "all" of the content.


There are still some bits which are a bit shaky and there is still much to do with the web client (for example, not that many outgoing messages yet defaults to being tagged in a way that the webclient recognizes). But it's a step forward!

There are many other improvements for developers, such as easier tools for running debuggers and a lot of new utilities and helper functions. The menu-creation tool (EvMenu) has seen a lot of improvements and also sport a few new decorators for quickly creating multi-page menus with full functionality to flip through and select large numbers of options.

The community has also chipped in with a large number of optional contributions for developers to use for their own game, such as a full turn-based combat system (divided into convenient chunks for the dev to pick and choose support for everything from magic and potions to equipment and ranged attacks).  There are also a range of helper functions for creating simpler menus and build commands as well as auditing tools and additions making better use of Django's very mature security features. 

The more detailed list of improvements and changes can be found in the announcement here. As usual, please report any issues to the issue tracker on github. 

Future

In the immediate future, we'll focus on resolving any bugs that may have slipped through the cracks and also resolve some other issues in the pipeline. 

But beyond that, work on Evennia 0.9 will begin. And before you ask - yes Evennia 0.9 is the version where we'll move fully to Python3. Our dependencies have now reached a point where this is possible and there will be no intermediary Python2/3 version. There is no timeline for the 0.9 release but it should hopefully not be too tricky for the community to make the jump when the time comes.


Saturday, August 18, 2018

Inline building in upcoming Evennia 0.8


Evennia, the Python MUD-server game development kit, is slowly creeping closer to its 0.8 release.

In our development branch I've just pushed the first version of the new OLC (OnLine Creator) system. This is a system to allow builders (who may have limited coding knowledge) to customize and spawn new in-game objects more easily without code access. It's started with the olc command in-game. This is a visual system for manipulating Evennia Prototypes.


Briefly on Prototypes

The Prototype is an Evennia concept that has been around a good while. The prototype is a Python dictionary that holds specific keys with values representing properties on a game object. Here's an example of a simple prototype:

{"key": "My house",
 "typeclass": "typeclasses.houses.MyHouse"}

By passing this dict to the spawner, a new object named "My house" will be created. It will be set up with the given typeclass (a 'typeclass' is, in Evennia lingo, a Python class with a database backend). A prototype can specify all aspects of an in-game object - its attributes (like description and other game-specific properties), tags, aliases, location and so on. Prototypes also support inheritance - so you can expand on an existing template without having to add everything fresh every time.

There are two main reasons for the Prototypes existing in Evennia: 
  • They allow you to customize individual objects easier. For example you could have a 'Goblin' base prototype with inheriting prototypes  Goblin Wizard" and "Goblin Chieftain" - all using the same Typeclass, but with different Attributes, equipment etc. 
  • Prototypes can be manipulated and scripted by builders without needing full Python access. This means that while the Typeclasses are designed and supplied by the Python developer, the builders can then use that typeclass to make very different types of object instances in-game.

 

What's new

As said, Prototypes have been around for a good while in Evennia. But in the past they were either manually entered directly as a dict on the command line, or created in code and read from a Python module. The former solution is cumbersome and requires that you know how to build a proper-syntax Python dictionary. The latter requires server code access, making them less useful to builders than they could be.

Note: If you are visually impaired, each image is also a link to a text-only version.

OLC index

In Evennia 0.8, while you can still insert the Prototype as a raw dict, spawn/menu or the new olc command opens a new menu-driven interface.

Select a prototype to load. This will replace any prototype currently being edited! ___________________________________________________________________________________________________  Select with <num>. Other actions: examine <num> | delete <num> Back (index) | Validate prototype | Quit   1: goblin_archer      5: goblin_archwizard                2: goblin_wizard                                          3: goblin                                                 4: archwizard_mixin
 
More importantly, builders can now create, save and load prototypes in the database for themselves and other builders to use. The prototypes can be tagged and searched as a joint resource. Builders can also lock prototypes if others are not to be able to read or use them to spawn things. Developers can still supply module-based "read-only" prototypes (for use as starting points or examples to their Builders, for example).

Found 1 match.   (Warning: creating a prototype will overwrite the current prototype!) ____________________________________________________________________________________  Actions: examine <num> | create prototype from object <num> Back (index) | Quit   1: Griatch(#1)

You can now also use the menu to search for and create a new Prototype based on an existing object (if you have access to do so). This makes it quick to start up a new prototype and tweak it for spawning other similar objects. Of course you could spawn temporary objects without saving the prototype as well.

The Typeclass defines what 'type' of object this is - the actual working code to use.  All spawned objects must have a typeclass. If not given here, the typeclass must be set in one of the prototype's parents.  [No typeclass set] ______________________________________________________________________________________________________________________________________________  Back (prototype-parent) | Forward (key) | Index | Validate prototype | Quit   1: evennia.contrib.tutorial_world.mob.Mob                 7: evennia.contrib.tutorial_world.objects.TutorialObject    2: evennia.contrib.tutorial_world.objects.Climbable       8: evennia.contrib.tutorial_world.objects.Weapon            3: evennia.contrib.tutorial_world.objects.CrumblingWall   9: evennia.contrib.tutorial_world.objects.WeaponRack        4: evennia.contrib.tutorial_world.objects.LightSource     10: evennia.contrib.tutorial_world.rooms.BridgeRoom         5: evennia.contrib.tutorial_world.objects.Obelisk         current: (1/3)                                              6: evennia.contrib.tutorial_world.objects.Readable        next page

Builders will likely not know which typeclasses are available in the code base. There are new a few ways to list them. The menu display makes use of Evennia 0.8's new EvMenu improvements, which allows for automatically creating multi-page listings (see example above).

There is also a new switch to the typeclass command, /list, that will list all available typeclasses outside of the OLC.

 

Protfuncs

Another new feature are Protfuncs. Similarly to how Inlinefuncs allows for calling for the result of a function call inside a text string, Protfuncs allows for calling functions inside a prototype's values. It's given on the form $funcname(arguments)where arguments could themselves contain one or more nested Protfuncs.

As with other such systems in Evennia, only Python functions in a specific module or modules (given by settings) are available for use as Protfuncs in-game. A bunch of default ones are included out of the box. Protfuncs are called at the time of spawning. So for example, you could set the Attribute 
Strength = $randint(5, 20) 
to automatically spawn objects with a random strength between 5 and 20.

prototype-key: goblin, -tags: [], -locks: spawn:all();edit:all() -desc: Built from goblin prototype-parent: None      key: goblin aliases: monster, mob attrs:  desc = You see nothing special.  strength = $randint(5,20)  agility = $random(6,20)  magic = 0 tags:  mob (category: None) locks:  call:true();control:id(1) or perm(Admin);delete:id(1) or perm(Admin);edit:perm(Admin);examine:perm(Builder);get:all();puppet:pperm(Developer) ;tell:perm(Admin);view:all() location: #2 home: #2   No validation errors found. (but errors could still happen at spawn-time) ______________________________________________________________________________________________________________________________________________  Actions: examine <num> | remove <num> Back (index) | Validate prototype | Quit   1: Spawn in prototype's defined location (#2)       2: Spawn in Griatch's location (Limbo)              3: Spawn in Griatch's inventory                     4: Update 2 existing objects with this prototype

When spawning, the olc will validate the prototype and run tests on any Protfunc used. For convenience you can override the spawn-location if any is hard-coded in the prototype.

https://pastebin.com/raw/K0a1z23h

The system will also allow you to try updating existing objects created from the same-named prototype earlier. It will sample the existing objects and calculate a 'diff' to apply. This is bit is still a bit iffy, with edge cases that still needs fixing.

 

Current status

The OLC is currently in the develop branch of Evennia - what will soon(ish) merge to become Evennia 0.8.

It's a pretty big piece of code and as such it's still a bit unstable and there are edge cases and display issues to fix. But it would be great with more people trying it out and reporting errors so the childhood issues can be ironed out before release!




Building Image: Released as Creative Commons here

Saturday, January 27, 2018

Kicking into gear from a distance

The last few weeks I have reworked the way Evennia's startup procedure works. This is now finished in the develop branch so I thought I'd mention a little what's going on.

Evennia, being a server for creating and running text-games (MU*s), consists of two main processes:
  • The Portal - this is what players connect to with their clients.
  • The Server - this is the actual game, with the database etc. This can be shutdown and started again without anyone connected to the Portal getting kicked from the game. This allows for hot-adding new Python code into the running Server without any downtime. 
Since Evennia should be easy to set up and also run easily on Windows as well as on Linux/Mac, we have foregone using the linux process management services but instead offered our own solution. This is how the reload mechanism currently looks in master branch:

Here I've excluded connections irrelevant to reloading, such as the Twisted AMP connection between Portal and Server. Dashed lines suggest a more "temporary" connection than a solid line.

The Launcher is the evennia program one uses to interact with the Server in the terminal/console. You give it commands like evennia start/stop/reload.

  • When starting, the Launcher spawns a new program, the Runner, and then exits. The Runner stays up and starts the Portal and Server. When it starts the Server, it does so in a blocking way and sits waiting in a stalled loop for the Server process to end. As the Server and Portal start they record their current process-ids in .pid files
  • When reloading, the Launcher writes a flag in a little .restart file. The Launcher then looks up the Server's .pid file and sends a SIGINT signal to that process to tell it to gracefully shut down. As the Server process dies, the Runner next looks at the Server's .restart file. If that indicates a reload is desired, The Runner steps in its loop and starts up a new Server process. 
  • When stopping, everything happens like when reloading, except the .restart file tells the Runner that it should just exit the loop and let the Server stay down. The Launcher also looks at the Portal's .pid file and sends a SIGINT signal to kill it. Internally the processes catch the SIGINT and close gracefully.
The original reason for this Server-Portal-Runner setup is that the Portal is also reloadable in the same way (it's not shown above). But over time I've found that having the Portal reloadable is not very useful - since players get disconnected when the Portal reloads one can just as well stop and start both processes. There are also a few issues with the setup, such as the .pid files going stale if the server is killed in some catastrophic way and various issues with reliably sending signals under Windows. Also, the interactive mode works a little strangely since closing the terminal will actually kill the Runner, not the Server/Portal - so they will keep on running except they can no longer reload ...
It overall feels a little ... fiddly.

In develop branch, this is now the new process management setup:


The Portal is now a Twisted AMP server, while the Evennia Server and Launcher are AMP clients. The Runner is no more.

  • When starting, the Launcher spawns the Portal and tries to connect to it as an AMP client as soon as it can. The Portal in turn spawns the Server. When the Server AMP client connects back to the Portal, the Portal reports back to the Launcher over the AMP connection. The Launcher then prints to the user and disconnects. 
  • When reloading, the Launcher connects to the Portal and gives it a reload-command. The Portal then tells the Server (over their AMP connection) to shutdown. Once the Portal sees that the Server has disconnected, it spawns a new Server. Since the Portal itself knows if a reload or shutdown is desired no external .restart (or .pid) files are needed. It reports the status back to the Launcher that can then disconnect.
  • When stopping, the Launcher sends the "Stop Server" command to the Portal. The Portal tells the Server to shut down and when it has done so it reports back to the Launcher that the Server has stopped. The Launcher then sends the "Stop Portal" command to also stop the Portal.  The Launcher waits until the Portal's AMP port dies, at which point it reports the shutdown to the user and stops itself.
So far I really like how this new setup works and while there were some initial issues on Windows (spawning new processes does not quite work they way you expect on that platform) I think this should conceptually be more OS-agnostic than sending kill-signals. 

This solution gives much more control over the processes. It's easy to start/stop the Server behind the portal at will. The Portal knows the Server state and stores the executable-string needed to start the Server. Thus the Server can also itself request to be reloaded by just mimicking the Launcher's instructions.
 The launcher is now only a client connecting to a port, so one difference with this setup is that there is no more 'interactive' mode - that is the Server/Portal will always run as daemons rather than giving log messages directly in the terminal/console. For that reason the Launcher instead has an in-built log-tailing mechanism now. With this the launcher will combine the server/portal logs and print them in real time to easily see errors etc during development.

The merger of the develop branch is still a good bit off, but anyone may try it out already here: https://github.com/evennia/evennia/tree/develop . Report problems to the issue tracker as usual.

Wednesday, September 20, 2017

Evennia 0.7 released

As of today Evennia 0.7 is officially out! Evennia is a Python framework and -server for creating text-based multiplayer games (MU*).

A big thank you to all collaborators that have helped with code and testing along the way!

Here is the hefty forum post that details how you migrate to Evennia 0.7.

Evennia 0.7 comes with a range of changes and updates (these are just the ones merged from the latest devel branch, a lot more has happened since 0.6 that were already in master): 
 
  • Evennia separates Player objects from Character objects in that the former is an OOC entity that can puppet one or more of the latter. The name Player was a source of some confusion since this is used differently in other code bases. Player has now been renamed to Account to make its function clearer.
  • Evennia's in-built website now uses our own theme and bootstrap under the hood to make it easier to modify as well as rescale it better on mobile devices (webclient is not updated at this point).
  • Shared logins between webclient and website, with the ability to log out of each independently of the other if so desired.
  • Prefix-ignoring - All default commands are now renamed without their @-prefixes. Both @examine, +examine or examine will now all point to the same command. You can customize which prefixes Evennia simply ignores when searching for a command. The mechanic is clever though - if you create a command with a specific key "+foo", then that will still work and won't clash with another command named just "foo".
  • Easy pause mechanisms using yield statements directly in Command code (just do yield 10 in your command code to have it pause ten seconds without blocking anyone else). You can also do retval = yield "Do you want to accept?" and have the command pause (non-blocking) for the player's input before continuing.
New contribs (optional plugins) (selection, new since 0.6 but many have been available on master branch for a while):
  • The optional In-Game-Python contrib by Vincent le Geoff allows for coding and scripting in-game using full-fledged Python to make events and triggers. It's not a safe softcode-style language (you have the full power of Python which is not good for untrusted users) but is intended for trusted builders to make complex scripting from the command line.
  • Wilderness/maps - Creation of dynamic or static maps based on ascii-maps for dynamically creating rooms and to show when moving around. (titeuf87, Cloud Keeper)
  • Full turn-based combat system, meant to expand for a desired system (battlejenkins)
  • Alternative Unix-style command base for Evennia, for those that prefer to enter commands like you do on the unix command line (with --flags etc) (Vincent le Geoff)
  • Multidescer, which together with Evennia's nick replacement system can be heavily customized in-game (me).
  • Mail - a @brandymail-style in-game mail-system (grungies1138)
  • Clothing system, for layered outfits adding to the wearer's desc when worn (battlejenkins).
A more technical list from the main announcement post: 
  • EvMenu formatting functions now part of class - EvMenu no longer accepts formatting functions as inputs, these are now part of the EvMenu class. To override the formatting of EvMenu nodes you should now override EvMenu and replace the format method you want. This brings EvMenu more in line with other services in Evennia, all of which are built around overriding. 
  • Scripts are now valid message senders - Also Scripts can now act as "sender" of a Msg, for example to a Channel or a player.
  • Due to the new prefix-ignoring above, @desc (builder-level description-setting) was renamed to setdesc to make it clearly separate from desc (used by a player setting their own description). This was actually the only clash we had to resolve this way in the default commands. 
  • Permission Hierarchy names change - To make Evennia's permission hierarchy better reflect how Evennia actually works, the old Players, Player Helpers, Builders, Wizards, Immortals hierarchy has changed to Player, Helper, Builder, Admin, Developer. Singular/Plural form is now ignored so Builder and Builders will both work (this was a common source of simple errors) Old permissions will be renamed as part of the migration process. The distribution of responsibilities has not changed: Wizards (which in some other systems was the highest level, which confused some) always had the power to affect player accounts (like an admin) whereas Immortals had server-level access such as @py - that is, they were developers. The new names hopefully makes this distinction clearer.  
  • All manager methods now return querysets - Some of these used to return lists which was a throwback to an older version of Typeclasses. With manager methods returning querysets one can chain queries onto their results like you can with any django query.
  • PrettyTable was removed from the codebase. You can still fetch it for your game if you prefer (it's on pypi). But EvTable has more functionality as well as color-support. 
  • Add **kwargs support to all object at_* hooks - All object at_ hooks, such as at_look, at_give etc now has an additional **kwarg argument. This allows developers to send arbitrary data to those hooks so they can expand on them without changing the API.  
  • Remove {-color tags - The use of {r, {n etc was deprecated for years and have now been completely removed from Evennia's core in favor of only one form, namely |r, |n etc. However, the color parser is now pluggable and the {-style as well as the %c style color tags etc can be re-added to your game since they are now in a contrib.  
  • Updated hooks for say/whisper commands - There are now at_before/after_say sub-hooks to allow say to be more easily customized. There is still ongoing discussion on the best way to handle this (see e.g. this PR).  
  • More compact distribution of ports - Before, Evennia distributed its ports widely, such as using 4000, 4001, 8000, 8001, 5001, 8022, some of which appeared close but actually was unrelated in functionality. They were also scattered throughout the settings file. The port definitions have now all moved more closer together, at the top of the settings file. Evennia will now use ports 4000-4006 by default (fewer depending on which protocols you want to start). This should make it easier to assign port ranges when wanting to run multiple Evennia instances on the same machine.  
  • Change how auto-login works - It used to be that once you logged into the website, every time you went to the webclient you would auto-login, even if you manually quite the webclient before. Only way to really log out (to change account, say) would be to first log out of the website. Now this will work more intuitively - you will still auto-login but if you log out of the webclient you "decouple" from the login state of the website and need to re-login to the webclient next time, even if you are still logged into the website.  
  • Better idle disconnects - This is a long-running pet peeve for many. Idle timeouts will now honor a particular lock. Setting this lock allows selected entities (for example staff and bots) to avoid getting timed out like everyone else.
  • A slew of bug fixes and other tweaks. See here for the list.

    Friday, August 25, 2017

    Renaming Django's Auth User and App

    Now that Evennia's devel branch (what will become Evennia 0.7) is slowly approaching completion, I thought I'd try to document an aspect of it that probably took me the longest to figure out. One change in Evennia 0.7 is that the Django model named "Player" changes name to "Account". Over time it has become clear that the old name didn't properly reflected the intention of the model. Sounds like a simple change, right? Well, it was not.

    Briefly on migrations


    First some background. A Django migration is a small Python file sitting in migrations/ sub folders throughout Evennia. A migration describes how our database schema changes over time, so as we change or update fields or add new features we add new migrations to describe how to go from the old to the new state. You apply them with the `evennia migrate` command we sometimes ask you to run. If we did not supply migrations, anyone with an existing Evennia database would have to either start from scratch or manually go in and tweak their database to match every upstream change we did.

    Each migration file has a sequential number, like migration_name.0002.py etc. Migrations will run in order but each migration can also "depend" on another. This means for example that a migration in the player/ folder (application) knows that it need to wait for the changes done by a migration in the typeclasses/ folder/app before it can run.  Finally, Django stores a table in the database to know which migrations were already run so to not apply them more than once.

    For reference, Evennia is wrapping the django manage commands, so where I refer to evennia migrate below you would use manage.py migrate in most Django applications.

    Our problem


    So I wanted to migrate the database change "Rename the Player model to Account". We had to do this under a series of additional constraints however:

    1. The Player is the Django Auth user, storing the password. Such a user Django expects to exist from the beginning of the migration history.
    2. The Player model sits in a application (folder) named "players". This folder should also be renamed to "accounts", meaning any in-migration references to this from other apps would become invalid.
    3. Our migration must both work for old and new users. That is, we cannot expect people to all have existing databases to migrate. Some will need to create the database from scratch and must be able to do so without needing to do any changes to the migration structure.
    4. We have a lot of references to "player" throughout the code, all of which must be renamed.
    5. The migration, including renames, should be possible to do by new users with newbie-level Python and database skills.

     

    Some false starts

    There is no lack of tutorials and help online for solving the problem of renaming a model. I think I tested most of them. But none I found actually ended up addressing these restraints. Especially point 2 in combination with point 3 above is a killer.

    • One of my first tries wiped the migrations table completely, renamed the folder and just assumed Player never existed. This sounds good on paper and works perfectly for fresh databases. But existing databases will still contain the old Player-related models. With the old migrations gone, this is now all wrong and there is no information on how to migrate it.
    • I tried initiating fresh migrations with a player model state so you can move an existing database over to it. But then fresh databases doesn't work instead, since the player folder is gone. Also, you run into trouble with the auth system.
    • I next tried to keep the old migrations (those we know work both for old and new databases) but to migrate it in-place. I did many different attempts at this, but every time one of the restraints above would get in the way.
    • Eventually I wrote raw SQL code in the migrations to modify the database tables retroactively. That is, I basically manually removed all traces of Player in the database where it was, copying things table by table. This was very work-intensive but overall decently successful. With proper error-checking I could get most of the migration to work from old databases as well as for new databases. The trouble was the migrations themselves. No matter how I tried, I couldn't get the migration history to accept what I had done - the dependencies now longer made sense on the database level (since I had manually edited things) and adding new migrations in the future would have been tricky.

    In the end I concluded that I had to abandon the notion that users be able to just do a single migrate command. Some more work would be needed on the user's part.

    The solution


    In the end I needed to combine the power of migrations with the power of Git. Using Git resolved the Gordian knot about the player folder. Basically the process goes like this:

    • First I copied of the `players` folder and renamed it and everything in it to accounts. I added this to settings.INSTALLED_APPS. I also copied the migrations from player and renamed everything in them appropriately - those migrations thus look like Account has always existed - so when you run the migration from scratch the database will be created normally. Note that this also means setting the Account as the auth user from the beginning of the migration history. This would be fine when migrating from scratch except for the fact that it would clash with the still existing Player model saying the same thing. We dodge this problem by the way we run this migration (we'll get to this later).
    • Next I added one new migration in the Account app - this is the migration that copies the data from the player to the equivalent account-named copies in the database. Since Player will not exist if you run this from scratch you have to make sure that the Player model exists at that point in the migration chain. You can't just do this with a normal import and traceback, you need to use the migration infrastructure. This kind of check works:

        # ...

        def forwards(apps, schema_editor):
        try:
            PlayerDB = apps.get_model("players", "PlayerDB")
        except LookupError:
            return

        # copy data from player-tables to database tables here

        class Migrations(migrations.Migration):
        # ...
        operations = [
            migrations.RunPython(forwards, migrations.RunPython.noop)
        ]


    • Now, a lot of my other apps/models has ForeignKey or Many2Many relations to the Player model. Aided by viewing the tables in the database I visited all of those and added a migration to each where I duplicated the player-relation with a duplicate account relation. So at this point I had set up a parallel, co-existing duplicate of the Player model, named Account.
    • I now made sure to commit my changes to git and tag this position in the version history with a clear git tag for later reference. This is an important step. We are saving the historical point in time where the player- and account-apps coexisted.5. The git position safely saved, I now went about purging player. I removed the player app and its folder.
    • Since the player folder is not there, it's migrations are not there either. So in another app (any would work) I made a migration to remove the player tables from the database. Thing is, the missing player app means other migrations also cannot reference it. It is possible I could have waited to remove the player/ folder so as to be able to do this bit in pure Python. On the other hand, that might have caused issues since you would be trying to migrate with two Auth users - not sure. As it were, I ended up purging the now useless player-tables with raw SQL:

        from django.db import connection

        # ...

        def _table_exists(db_cursor, tablename):
        "Returns bool if table exists or not"
        sql_check_exists = "SELECT * from %s;" % tablename
        try:
            db_cursor.execute(sql_check_exists)
            return True
        except OperationalError:
            return False


        def _drop_table(db_cursor, table_name):
        if _table_exists(db_cursor, table_name):
            sql_drop = "DROP TABLE %s;" % table_name
            db_cursor.execute(sql_drop)


        def drop_tables(apps, schema_migrator):
        db_cursor = connection.cursor()
        _drop_table(db_cursor, "players_playerdb")
        _drop_table(db_cursor, "players_playerdb_db_attributes")
        # etc

        class Migration(migrations.Migration):

        # ...

        operations = [
            migrations.RunPython(drop_tables)
        ]

    • Next I continued creating migrations to remove all the "duplicate" foreignkeys I had creater earlier in all other apps that make up Evennia. I committed those.
    • Finally I wrote a little program to rename Python code uses of "player" in to "account" (retaining capitalization, supporting renaming "a player" to "an account" etc). I spent some time on this since I wanted our users to be able to convert their own codes with a decent interface. I include this tool in the Evennia repository.

    And with this, the migration's design was complete. Below is how to actually use it ...

    Running the final migration


    In brief, what our users will do after pulling the latest code is as follows:

    1. If they are starting fresh, they just run evennia migrate as usual. All migrations referencing player will detect that there is no such app and just be skipped. They are finished, hooray!
    2. If they have an existing database, they should make a copy of my renaming-program, then check out  the tagged point in the git history I created above, a time when players and accounts coexisted in code.
    3. Since an important aspect of Evennia is that users create their own game in a "game" folder, the user can now use my renaming program to interactively rename all occurrencies of `player` into `account` (the term 'player' is so ubiquitous that they may have used in in different places they don't want to rename).
    4. Next they run migrations at that point of the git history. This duplicates player data into its renamed counterparts.
    5. Now they should check out the latest Evennia again, jumping back to a point where the players application is  gone and only accounts exists.
    6. Note that our migration history is wrong at this point. It will still contain references to migrations from the now nonexisting players app. When trying to run from scratch, those will fail. We need to force Django to forget that. So the user must here go into their database of choice and run the single SQL statement DELETE FROM django_migations; . This clears the migration history.  This is a step I wanted to avoid (since it requires users to use SQL) but in the end I concluded it must be done (and is hopefully a simple enough instruction).
    7. Next, we trick the database to again think that we have run all the migrations from the beginning (this time without any mention of players). This is done with the --fake switch: evennia migrate --fake . This fake-applies all migrations and stores in the database that they have run.
    8. However, the last few migrations are the ones I added just above. Those actually remove the player-related tables from the database. We really do want to run those. So we fake-fake undo those with evennia migrate --fake typeclasses 0007, which is the application and migration number I used to wipe players. This causes django to forget those migrations so we can run them again.
    9. Finally we run evennia migrate. This runs the previously "forgotten" migrations and purges the last vestigest of players from the database. Done!

     

    Conclusions


    And that's it. It's a bit more involved for the end user than I would have liked, and took me much longer than expected to figure out. But it's possible to reproduce and you only need to do it once - and only if you have a database to convert. Whereas this is necessarily specified for Evennia, I hope this might give a hint for other django users aiming to do something like this!