Lately I've done work on the memory management of Evennia. Analyzing the memory footprint of a python program is a rather educational thing in general.
Python keeps
tracks of all objects (from variables to classes and everything in
between) via a memory reference. When other objects reference that
object it tracks this too.
Once nothing references an object, it does not need to be in memory any more - in a more low-level languages this might lead to a memory leak. Python's
garbage collector handles this for us though - it goes through all abandoned objects and frees the memory for usage by other things. The garbage collector will however
not do its thing as long as some other
object (which will not be garbage-collected)
still holds a
reference to the object. This is what you want - you don't want existing
objects to stop working because an object they rely on is suddenly not
there.
Normally in Django, whenever you retrieve an database model instance, that exists only in memory then and there. If you later retrieve the same object from the database, the model instance you have to work with is most likely a new one. This is okay for most usage, but Evennia's typeclass system (described in an earlier blog entry) as well our wish to store temporary properties on models (existing until next server restart) does not work if the model instance we get is always different. It would also help if we didn't have to load models from the database more than necessary.
For this reason, Evennia uses something called the
idmapper. This
is a cache mechanism (heavily modified for Evennia) that allows objects to be loaded from the database
only once and then be reused when later accessed. The speedup achieved from this
is important, but as said it also makes critical systems work properly.
The tradeoff of speed and utility
is memory usage. Since the idmapper never drops those references it means that objects will never be garbage collected. The result was that the memory usage
of Evennia could rise rapidly with an increasing number of objects. Whereas
some objects (like those with temporary attributes) should indeed not be garbage
collected, in a working game there is likely to be objects without such
volatile data. An example might be objects that are not used some of the time - simply because
players or the game don't need them for the moment. For such objects it
may be okay to re-load them on demand rather than keep them in memory indefinitely.
When looking into this I found that simply force-flushing the idmapper did
not clean up all objects from memory. The reason for this has to do with how Evennia references objects via a range of other means. The reference count never went to zero and so the garbage collector never got around to it.
With the excellent
objgraph library it is actually pretty easy to track just what is referencing what, and to figure out what to remove. Using this I went through a rather prolonged spree of cleanups where I gradually
(and carefully) cleaned up Evennia's object referencing to a point where
the only external reference to most objects were the idmapper cache
reference. So removing that (like when deliberately flushing the cache) will now make the object possible to
garbage-collect.
This is
how the reference map used to look for one type of Evennia object (ObjectDB) before the cleanup. Note
the several references into the ObjectDB and the cyclic references for
all handlers (the cyclic reference is in itself not a problem for reference-counting but they are slow and unnecessary; I now made all handlers use lazy-loading with weak referencing instead).
This
is how the reference map looks for the same object now. The __instance__ cache is the
idmapper reference. There are also no more cyclic references for
handlers (the display don't even pick up on them for this depth of recursion). Just removing that single link will now garbage-collect
ObjectDB and its typeclass (ignore the
g reference, that is just
the variable holding the object in ipython).
We also see that the
dbobj.typeclass <-> typeclass.dbobj references keep each other
alive and when one goes the other one goes too - just as expected.
An curious aspect of Python memory handling is that (C-)Python does
not actually release the memory back to operating system when flushing the idmapper cache. Rather Python makes it internally available so that it does not need to request any more. The result is that if you look at Evennia with the
top command, its memory requirement (for example while continuously creating new objects) will not actually
drop on a idmapper flush, it will just
stop rising. This is discussed at length in
this blog, it was good to learn it was not something I did at least.
Apart from the memory stuff, there is work ongoing with fixing the latest batch of user issue reports. Another dev is working on cleaning up the web-related code, it should make it a lot cleaner to overload web functionality with custom code. One of those days I'll also try to sit down and finally convert our web client from long-polling to use web sockets now that Evennia suppports web sockets natively. Time, time ...
