A Short History of Just Another Web Application Server
What came before...
Back in murky days 2004, before I began working on Jawas (Just Another Web Application Server), I had been building flash games with a friend for a range of web and mobile properties. These games used a rather clever bit of C++ I wrote over a 3 year period for a series of massively multiplayer game projects: the ConnServer. In modern day terms, you could think of the ConnServer as a instant messaging server married with a embedded bot engine that used 2004 era WebSockets to communicate between a smart client and a network of peers. The role of the ConnServer was to service messages sent to chat rooms which connected a group of users in a part of the game world. Each ConnServer was responsible for one shard of the game world and would cache objects in an append only memory area. Persistence was maintained throubh a blob store in a backend Postgres database. The overal architecture was a response to the problems I had encountered at the Sporting News building the first live draft applications for NFL.com and MLB.com. The ConnServer was like Jabber met Redis met Memcached before any of those were fit for production use.
Jawas was born over the next few weeks as I was between game projects. It was a c10k engine from the start, contemporaneous to nginx and pre dating node.js by about 3 years. It used a heavily modified SpiderMonkey build, epoll, kqueue, mmap, and dnotify. It was fast, small and in 6 months running all of production. It also demonstrated the problems with a fully event driven we stack, and continuation passing became an well rehearsed art. Since the ConnServer handled persistent bidirectional socket connection using Flash XMLSockets much like modern day WebSockets, Jawas never needed to handle that sort of load. Instead, Jawas remained a HTTP focused app, that dabbled in image manipulation, SMS, and database access via closure wrapped stored procedures.
There were a number of cool projects built on the Jawas plaform:
- A proof of concept video game platform for ICTV, in which we used ffmpeg to generate mpeg2 streams that would allow people to game over set top boxes, and proxy YouTube videos over the device.
- A video scavenger hunt for Sony, which used Jawas for processing images uploaded from the field via MMS messages, this is where the Image Magick support came in for doing automatic cropping and scaling of the gallery view.
- A cellphone game platform using WAP and SMS for doing old school surveys and triva contents on feature phones
- A poker game or two...
What was cool about the ConnServer and Jawas combination was you could build a flexible website on the Jawas platform and then script the realtime responses to various messages passed to the ConnServer bots which interacted with an in memory high performance object store. Rarely did the two ever need to do the same thing or interface with the same database tables. Usually, the ConnServer used it blob storage for all of the user interactions, and the Jawas site merely performed the typically boring bits like account management, tracking customer usage, and serving up the administrivia pages tracking sales and related information. As Flash provided a rich media frontend there was little pressure to develop most of the site in typical HTML and markup.
Jawas like the ConnServer was designed to support 200 concurrent users making 1 request per second each on a $500 bargin basement server. The theory behind the design is that each client gets 5ms of processing and a fixed amount of RAM (5MB if 1GB of RAM + 1GB shared). The average response time was to remain under 50ms, and that included a database transaction. We could support 2k-8k concurrent users in practice, provided that they only made 1 request every 10s - 1 minute. Greater than that would increase the capital cost basis from $2.50 / concurrent user ($0.84/year annualized cost assuming a 3 year depreciation plan). If you are building games for hire that are going to live and die by their ability to scale and generate ad impressions, you need to design everything with real world costs in mind. Engineering a solution that could handle 200k active users at a time for $50,000 meant that the project with an advalue of a measly $1CPM could generally generate about $2000 per hour at peak, and buy itself back in a matter of days rather than months. When you factored in the full cost of development, a typical project that gained traction could buy itself back in a matter of a couple weeks. Since most of the games we did were for large media companies with established brands, this was important.
Evangelizmo was a startup that ultimately failed due to launching at the exact same time as the financial crisis threatened to shutdown everything. There was nothing worse than going into trials with customers and then basically being told by every VC to call back in a year as the taps were dry. Evangelizmo had a couple cool additions to the mix, including support for SMS via a GSM modem, and interfaces for several SMS gateway providers. In the Lua revision, perfomance was born out of desire to make sure that the product could scale to meet the sort of customer demand we expected from selling it as an open platform for 3rd party e-commerce platforms. We were going to opensource the integration code for several open source e-commerce offerings and make up the revenue in bulk. On the website we advertised an outrageous price simply to keep from having to service customers while we got it rock solid. You can always drop your prices to something reasonable when you know exactly how much it would cost to scale.
After Evangelizmo, I occasionally revisited the Jawas and ConnServer code, and put it up on Github (older versions of each with all of the application specific code removed). And it was pretty good.
The New Wave
Once again, I've revisiting my old friends and updating them to work with the changes in the web platform. As I'm currently typing this up on an iPad (who ever says you can't do productive things with an iPad is lying), I'm a firm believer that Flash is dead. Jawas has a new source of life in the form of being able to manage applications which exploit WebSockets and HTML5 Canvas. I'm merging the two projects into one, and removing the distinction between Connection Server and Web Server, there's only one server now: JAWAS. Also pulled into the mix, I'm adding support for more modern protocols like WebSockets rfc6455, and AMQP to connect to brokers like RabbitMQ. In addition to the Amazon AWS interfaces built into the Jawas code, I'm also adding support for Open Stack, so the server will be able to auto-scale itself and make use of services like CloudFiles for static asset storage.
Finally Jawas is getting it multi-language support back into gear. Currently my thinking is supporting a range of languages I've supported in the past in various projects. Top of my list will be:
- and maybe a Lisp
The goal here is to make Jawas into a useful platform for gluing together all of the disparate pieces of web technologies that I use on a daily basis and make them accessible via a webbrowser. Since about 2000, most of my development servers have been hosted by Rackspace or Amazon, and my personal development boxes are currently a Raspberry Pi and a iPad. Jawas was born in the cloud and will continue to live in the cloud, and make the cloud a friendly place for developers to live. The future is bright, and Jawas is back...