A couple days ago, I finished purging the last remnants of application specific code from a project I had started 7 years ago, the ConnServer. The ConnServer (CS for short) formed the backbone of 5 mmo games I did for a couple large clients and two startups. This version of the code is missing a lot ofthe tooling that was used in production, but I can describe a bit about how it worked. The premise behind the connection server was to build a shaders environment in which the object space was partitioned into 64k partitions each with a 48bit address space. These objects were serialized and persisted in a postgresql database that was largely used as a easy way to share the contents of a node across reboots. For each partition 3+ nodes were run using a s2s gossip protocol defined an an application layer daemon which ensured that conflicts didn't arise when a node failed. The s2s layer also joined "rooms" across multiple partitions so that locality could be maintained for user data, but interactions could happen between users on different shards.

In practice, we were able to support 2k concurrent users per $500 no-name server back in 2004 for most of our games. Those with near realtime requirements and a lot of ai based behavior usually supported 200 concurrent users per CPU. The current performance on modern hardware seems to be the same per core, meaning you can run more partitions per CPU. With the advent of much faster interconnects, it should be possible to replicate partitions pretty much in real time, but I never had that need in production.

The messaging protocol grew out of using XMLSockets as bidirectional tcp channels between the backend and the clients. We also used it for mobile game clients over EDGE, and would work fine over faster networks. The only required field is msg:method where method is the string representing a registered callback function from a shared library module in C++ or Python module. Python modules use an embedded C interface and make it easy to extend without full recompilation. The C++ plugin system had a versioned dynamic loader but symbol collision would usually lead to instability.

Overall this was on of my favorite platforms to develop on.