DAVE'S LIFE ON HOLD

A Short History of Just Another Web Application Server

What came before...


Back in murky days 2004, before I began working on Jawas (Just Another Web Application Server), I had been building flash games with a friend for a range of web and mobile properties. These games used a rather clever bit of C++ I wrote over a 3 year period for a series of massively multiplayer game projects: the ConnServer. In modern day terms, you could think of the ConnServer as a instant messaging server married with a embedded bot engine that used 2004 era WebSockets to communicate between a smart client and a network of peers. The role of the ConnServer was to service messages sent to chat rooms which connected a group of users in a part of the game world. Each ConnServer was responsible for one shard of the game world and would cache objects in an append only memory area. Persistence was maintained throubh a blob store in a backend Postgres database. The overal architecture was a response to the problems I had encountered at the Sporting News building the first live draft applications for NFL.com and MLB.com. The ConnServer was like Jabber met Redis met Memcached before any of those were fit for production use.

Early days...


Back in the halcyon days of 2006, I was finishing up a boxing game for ESPN, which was running a the ConnServer, with all of the AI written in OCaml, and a friend of mine said something to the effect of "wouldn't it be cool if you could program the web backend in JavaScript?" And I said give me a few months...

Jawas was born over the next few weeks as I was between game projects. It was a c10k engine from the start, contemporaneous to nginx and pre dating node.js by about 3 years. It used a heavily modified SpiderMonkey build, epoll, kqueue, mmap, and dnotify. It was fast, small and in 6 months running all of production. It also demonstrated the problems with a fully event driven we stack, and continuation passing became an well rehearsed art. Since the ConnServer handled persistent bidirectional socket connection using Flash XMLSockets much like modern day WebSockets, Jawas never needed to handle that sort of load. Instead, Jawas remained a HTTP focused app, that dabbled in image manipulation, SMS, and database access via closure wrapped stored procedures.

There were a number of cool projects built on the Jawas plaform:
What was cool about the ConnServer and Jawas combination was you could build a flexible website on the Jawas platform and then script the realtime responses to various messages passed to the ConnServer bots which interacted with an in memory high performance object store. Rarely did the two ever need to do the same thing or interface with the same database tables. Usually, the ConnServer used it blob storage for all of the user interactions, and the Jawas site merely performed the typically boring bits like account management, tracking customer usage, and serving up the administrivia pages tracking sales and related information. As Flash provided a rich media frontend there was little pressure to develop most of the site in typical HTML and markup.

Second Generation...


Around 2008, I replaced the custom Mozilla SpiderMonkey engine, and replaced it with Luajit. Lua was not only substantially faster, but also more reliable under load. The Jawas application transitioned fully away from JavaScript and entirely to Lua under Evangelizmo. I made an attempt to integrate V8, but the memory management of V8 sucks and could not guarantee the cost basis Jawas was designed to meet.

Jawas like the ConnServer was designed to support 200 concurrent users making 1 request per second each on a $500 bargin basement server. The theory behind the design is that each client gets 5ms of processing and a fixed amount of RAM (5MB if 1GB of RAM + 1GB shared). The average response time was to remain under 50ms, and that included a database transaction. We could support 2k-8k concurrent users in practice, provided that they only made 1 request every 10s - 1 minute. Greater than that would increase the capital cost basis from $2.50 / concurrent user ($0.84/year annualized cost assuming a 3 year depreciation plan). If you are building games for hire that are going to live and die by their ability to scale and generate ad impressions, you need to design everything with real world costs in mind. Engineering a solution that could handle 200k active users at a time for $50,000 meant that the project with an advalue of a measly $1CPM could generally generate about $2000 per hour at peak, and buy itself back in a matter of days rather than months. When you factored in the full cost of development, a typical project that gained traction could buy itself back in a matter of a couple weeks. Since most of the games we did were for large media companies with established brands, this was important.

Evangelizmo was a startup that ultimately failed due to launching at the exact same time as the financial crisis threatened to shutdown everything. There was nothing worse than going into trials with customers and then basically being told by every VC to call back in a year as the taps were dry. Evangelizmo had a couple cool additions to the mix, including support for SMS via a GSM modem, and interfaces for several SMS gateway providers. In the Lua revision, perfomance was born out of desire to make sure that the product could scale to meet the sort of customer demand we expected from selling it as an open platform for 3rd party e-commerce platforms. We were going to opensource the integration code for several open source e-commerce offerings and make up the revenue in bulk. On the website we advertised an outrageous price simply to keep from having to service customers while we got it rock solid. You can always drop your prices to something reasonable when you know exactly how much it would cost to scale.

After Evangelizmo, I occasionally revisited the Jawas and ConnServer code, and put it up on Github (older versions of each with all of the application specific code removed). And it was pretty good.

The New Wave


Once again, I've revisiting my old friends and updating them to work with the changes in the web platform. As I'm currently typing this up on an iPad (who ever says you can't do productive things with an iPad is lying), I'm a firm believer that Flash is dead. Jawas has a new source of life in the form of being able to manage applications which exploit WebSockets and HTML5 Canvas. I'm merging the two projects into one, and removing the distinction between Connection Server and Web Server, there's only one server now: JAWAS. Also pulled into the mix, I'm adding support for more modern protocols like WebSockets rfc6455, and AMQP to connect to brokers like RabbitMQ. In addition to the Amazon AWS interfaces built into the Jawas code, I'm also adding support for Open Stack, so the server will be able to auto-scale itself and make use of services like CloudFiles for static asset storage.

Finally Jawas is getting it multi-language support back into gear. Currently my thinking is supporting a range of languages I've supported in the past in various projects. Top of my list will be:
The goal here is to make Jawas into a useful platform for gluing together all of the disparate pieces of web technologies that I use on a daily basis and make them accessible via a webbrowser. Since about 2000, most of my development servers have been hosted by Rackspace or Amazon, and my personal development boxes are currently a Raspberry Pi and a iPad. Jawas was born in the cloud and will continue to live in the cloud, and make the cloud a friendly place for developers to live. The future is bright, and Jawas is back...