One Billion Processes
Back in about '94 I wrote a compiler for a language that was best described as Forth meets Lisp meets TCL. I knew enough x86 assembler to write a basic Forth from memory in a hex editor. I used that approach to build a compiler for a language with a Lisp inspired implementation. The object model was similar to TCL's with the exception that each object had it's own process slice, and ran in it's own green thread. I wrote a basic MUD in it, and some VGA graphics drawing bits, and eventually lost interest in favor of building content for lambdamoo. I have probably rewritten that original system 30 times or so by now. My disk drives are littered with compilers and assemblers for variations on the language.
There is a tension between building a system that is sufficiently low level to capture the machine interface and sufficiently high level to make programming easy. At some level of abstraction and formality, the systems becomes too fragile. Once your implementation maps to sufficiently complex code, it becomes incredibly difficult to develop accurate intuitions about how the system will behave. Below a certain threshold in the other direction you can't develop enough velocity to get anything done. The lack of metaprogramming kills any attempts to build sustainable systems. The difficulties of chasing down instruction coding issues or pipeline thrashing take too much away from building beautiful systems.
I find myself constantly on the cusp of revisiting this project. I am looking forward to a time in the next couple years where 256 core ARM chips and 144 core low power Forth chips issue in an age of ubiquitous processing. When you have thousands of cores per blade, and billions of cores per data center, and the cost of shuffling data on and off chip becomes to great, how do you program efficient systems? How do you write programs consisting of billions of tiny self contained objects, coded to run efficiently in on die RAM attached to a core with limited bus access? Well I actually have been modeling that since '94, and know how to build the tools that can build those programs. I am just waking for hardware to catch up.
Last night I was implementing a Erlang style messaging layer for Self objects. I have begun hacking my VM to support distributed messages. I am looking forward to a time when it can be made to run on lots of little ARM cores. But I need to refactor the system into smaller and smaller components. As I work on this project, fighting with 100k lines of C++, projects like Klien start looking appealing. And after a bit, returning to NewScript looks like a good idea. In any event, the next NewScript will look like Erlang meets Self meets Forth. And that may stick once the hardware catches up.