Bringing back python memory

Lately I've done work on the memory management of Evennia. Analyzing the memory footprint of a python program is a rather educational thing in general.

Python keeps tracks of all objects (from variables to classes and everything in between) via a memory reference. When other objects reference that object it tracks this too.
Once nothing references an object, it does not need to be in memory any more - in a more low-level languages this might lead to a memory leak. Python's garbage collector handles this for us though - it goes through all abandoned objects and frees the memory for usage by other things. The garbage collector will however not do its thing as long as some other object (which will not be garbage-collected) still holds a reference to the object. This is what you want - you don't want existing objects to stop working because an object they rely on is suddenly not there.

Normally in Django, whenever you retrieve an database model instance, that exists only in memory then and there. If you later retrieve the same object from the database, the model instance you have to work with is most likely a new one. This is okay for most usage, but Evennia's typeclass system (described in an earlier blog entry) as well our wish to store temporary properties on models (existing until next server restart) does not work if the model instance we get is always different. It would also help if we didn't have to load models from the database more than necessary.

For this reason, Evennia uses something called the idmapper. This is a cache mechanism (heavily modified for Evennia) that allows objects to be loaded from the database only once and then be reused when later accessed. The speedup achieved from this is important, but as said it also makes critical systems work properly.

The tradeoff of speed and utility is memory usage. Since the idmapper never drops those references it means that objects will never be garbage collected. The result was that the memory usage of Evennia could rise rapidly with an increasing number of objects. Whereas some objects (like those with temporary attributes) should indeed not be garbage collected, in a working game there is likely to be objects without such volatile data. An example might be objects that are not used some of the time - simply because players or the game don't need them for the moment. For such objects it may be okay to re-load them on demand rather than keep them in memory indefinitely.

When looking into this I found that simply force-flushing the idmapper did not clean up all objects from memory. The reason for this has to do with how Evennia references objects via a range of other means. The reference count never went to zero and so the garbage collector never got around to it. 

With the excellent objgraph library it is actually pretty easy to track just what is referencing what, and to figure out what to remove. Using this I went through a rather prolonged spree of cleanups where I gradually (and carefully) cleaned up Evennia's object referencing to a point where the only external reference to most objects were the idmapper cache reference. So removing that (like when deliberately flushing the cache) will now make the object possible to garbage-collect.

This is how the reference map used to look for one type of Evennia object (ObjectDB) before the cleanup. Note the several references into the ObjectDB and the cyclic references for all handlers (the cyclic reference is in itself not a problem for reference-counting but they are slow and unnecessary; I now made all handlers use lazy-loading with weak referencing instead).

This is how the reference map looks for the same object now. The __instance__ cache is the idmapper reference. There are also no more cyclic references for handlers (the display don't even pick up on them for this depth of recursion). Just removing that single link will now garbage-collect ObjectDB and its typeclass (ignore the g reference, that is just the variable holding the object in ipython).
We also see that the dbobj.typeclass <-> typeclass.dbobj references keep each other alive and when one goes the other one goes too - just as expected.

An curious aspect of Python memory handling is that (C-)Python does not actually release the memory back to operating system when flushing the idmapper cache. Rather Python makes it internally available so that it does not need to request any more. The result is that if you look at Evennia with the top command, its memory requirement (for example while continuously creating new objects) will not actually drop on a idmapper flush, it will just stop rising.  This is discussed at length in this blog, it was good to learn it was not something I did at least.


Apart from the memory stuff, there is work ongoing with fixing the latest batch of user issue reports. Another dev is working on cleaning up the web-related code, it should make it a lot cleaner to overload web functionality with custom code. One of those days I'll also try to sit down and finally convert our web client from long-polling to use web sockets now that Evennia suppports web sockets natively. Time, time ...