DAVE'S LIFE ON HOLD

Fear and loathing of Nodejs

The past few weeks, I have been performance testing various applications under node 0.12.x. I've been running these tests both bare metal and on VMs, and on a couple different operating systems. And I've been steadily disappointed with the performance. What has been shocking is the amount of time spent in userspace manipulating the v8 stack. The depth of the call graph utterly boggles my mind, and also explains how a simple application handling a measly 150 requests per second uses 106MB of real RAM. When your stack size alone is measured in MB, you know there's a design issue.

At 150 requests per second, 60% of a CPU core was occupied by the single Nodejs process. While this might sound like there's room to spare, going higher produced sever latency degradations. From the standpoint of a production quality system, we need that 40% headroom to deal with load spikes and garbage collection events. From a business standpoint this also means that we need about 256MB and 1 CPU per 150 message per second load. For a game server, that processes events with an average 200ms ping time, we can support 30 concurrent users per CPU. This too is only the event broadcast network, it doesn't even cover game state modeling, transaction handling, and AI.

What disappoints me here is that ten years ago, I could support 200 concurrents per core. A $500 server and a custom messaging server written in evented C++ cost about $30/mo to rack and power in a Fremont CA data center, and could support between 300 - 800 paying customers depending on how peak was peak. So at a cost of $45/mo, we could support approximately 500 users average across multiple games.

Now, the economics of the cloud means I can support 75 users at $10/CPU/mo, or about 350 users for the same $45/mo ten years later. What I traded was 2x a year trips to the datacenter and Javascript for C++. This isn't really worth it. With the bare metal server, I could select the network hardware and tune the driver. The cost of failure was a required adding some additional standby servers, but typically that meant pressing a staging of dev box into production. The mean time between failures was pretty low, and if the machine survived burn-in it would live for 3-4 years without so much as a reboot.

So why bother with the cloud at all? Why use Nodejs? What drives these technical decisions? Honestly it is driven by who is giving you funding and who they perceive you can hire. A cost of $0.09/user/mo isn't that much of a savings over $0.28/user/mo when you have nearly no users and your dev team costs $60k/mo. If you can get to market slightly faster, and you can save a month of your developer's time, you save $60k! You would need 333k user months of savings to get the same out of the C ++ server.

And you need to be able to hire qualified developers who can grok server tuning, machine building, OS maintenance, network programming, systems programming, and game programming. Guess what? 10 years ago I could hire a handful of people with those skills, today they are even harder to find. Demand for certain skill sets price them out of that equation. In today's dollars, you will need to spend 2x that to get the same level of talent and experience. Once again your investor will want to see you hire someone today, rather than waiting for 666k user months of gameplay to break even.

Since there is a chance your game may fail to launch or may fail to attract users, you choose the tech that makes it easier to raise funding. Hip language w/ kids coming out of school who know it? Win. If you fail you failed for $120k less! That's $120k you didn't need to raise, and one fewer pissed off investor. Success! If you succeed, you end up spending $0.18/user per month. If that user has a monthly expected value of $2, you've reduced your margins by 10% but also reduced your capital investment as well. You could turn off all your servers tomorrow, and suffer no write offs. That makes it easier for an investor to walk away from a business with no capital expenses.

So why do I fear and loath Nodejs? Because it is a finance tool masquerading as a technology stack. It isn't a high performance production quality server, it is bullet point in a slide deck that helps you get funding. That market is now shifting to shitty servers written in Go, by people who previously knew Python. I could write this same blog post about the messaging server I wrote in Go a few months back. But I won't because it isn't about the technology. It is about business and the politics of financing your dreams.