The Ideal of the Perfect Program
Prior to our weekly retrospective at work, I demonstrated some of the code metric tools I've written for myself to keep a handle on the things I find important:
- short function definitions which build upon each other
- wide flat static call graphs so that nesting never gets too deep
- avoidance of unintentional co-routines through multifunction cyclic calls
- keeping function argument count as low as possible, keeping state as localized as possible
These design goals have a set of observations attached:
- the ideal function is one line long, and is a single sentence which describes you application
- the ideal function takes zero arguments
- the ideal application is made of many ideal applications which do one thing well
These concepts immediate caused much head scratching and debate. How can you have a function with zero arguments? Does that imply state should be global? Isn't global state bad? How do you have more than one function per line? Doesn't that conflict with minimal arguments?
When you think about these statements, there is a common thread of incredulity based on lessons learned academically. Usually when you ask a question of the questioner, "Why do you believe this?" The answer invariably comes back: "I was taught that..." or " I read that..." as if mere instruction were enough to disprove repeated observation. Never have I heard the answer in the form of "because X implies Y and we observe X therefore Y". In short, the answers are merely parroting an answer devoid of understanding. For these programmers, software engineering is a religion and not a useful art or science.
An ideal function being one line long is about not just being concise, but about the geometric growth of complexity one can achieve through the repeated application of definitions. As a word may stand for complex ideas or things, like philosophy, universe, law, mathematics, repeated application of this simple grouping of operations can generate enormous complexity. Type firefox on the command line and that tiny one word program when executed can render the entire internet's contents. Then Firefox starts on Linux it invokes a function _start which invokes main() which performs all of the functions of the browser. The value of this concision is it allows you to communicate effectively with other humans within a reduced context. The computer doesn't really care about this concision, and many compilers use static analysis to unravel this factoring and serialize the instruction streams. Humans, however, need words to describe the function of an application to other humans, and the art of reading a program is learning the meaning of the set of terms that a programmer defines to describe his system. In effect, programming is an art of formal writing with regards to the description of how a machine is to approximate the metaphor constructed via the mental model of the programmer. A good programmer conveys his ideas using a formal axiomatic system that derives from the computer's base operations, but bridges the human concerns and idioms at higher levels of abstraction.
The ideal function takes zero arguments and returns a value simply because it has the lowest cognative burden. If you think about a call like socket() which instantiates a socket and returns a file id, this is an ideal function. A less ideal form is a socket(flags) call which takes a bit field of different settings. The cognative burden is much higher as we must now remember all of the flag values or look them up each time we need something. Usually, this is an argument in favor of having sane defaults. Say we wish to modify a socket to be nonblocking, we would rather say nonblocking(socket()) rather than fcntl(sock,F_SETFD, O_NONBLOCK) which is further from the ideal of zero. I an object oriented language where the receiver of a message is an implicit argument, we would have the idem we form be socket().nonblobking() where each function returns the object itself (or an error which will trap at the next message send). The goal of the zero argument function is to minimize the number of variables at each stage to eliminate cognative overhead. This allows for both more controlled testing as it is easier to control variation.
Conceptually, all functions implicitly take a variable O which represents the state of the machine and the universe in which it operates. This means all functions which have side effects have an implicit return value O'. Since nearly all useful functions require CPU pipeline, CPU caches, memory, stack, registers, and sundry system resources, the ideal function operates entirely in terms of O -> O' with O = O' in the case of a perfect ideal functions. The reason it is perfect is 100% of the utility was expressed via transient side effects and the processing returned the machine state to its original condition. Meaning you can safely run it ad nauseum, and it will always produce the same result. A perfect application is one which receives a request, processes the information, returns a response, and externalizes all of the state change. Anyone familiar with REST, will recognize that representational state transfer is an attempt to describe a perfectly functional world by preserving all of the state of the application in the messages themselves. The difficulty with REST in the real world is it becomes economically infeasable to transfer sufficient state, as the cost of following links begins to exceed the value of the data. For example, let's say you have a resource represented via a 128bit UUID, and you're going to create a link in the document using a simple relative path addressing /asset/UUID. As you're using plain text, you encode the UUID as a base64 encoded string, with a cost of 22 bytes. Then the total cost of the "/assets/UUID", link is 32bytes. If 32bytes is a significant portion of the data pointed to by the reference, then the cost of the reference can exceed the utility of the data in question. If only a portion of the data in question is being used, say a boolean field, than the cost of the reference far exceeds the cost of the data. When the roughly 1k cost of a typical HTTP request is further factored on top of that, the cost of linking is almost entirely likely to exceed the value for many small objects. It is for this reason that many applications externalize the state transitions into relational databases, violating the REST principle, but in turn more effectively utilizing references locally. Databases are effectively imperfect functions which map O -> O', and whose primary utility is via side effect in the event of a successful transaction, but act as if they are prefect functions in the event of a failure or rollback.
Composition of functions allows us to build a complex system from a set of simpler components. Saying that the ideal application is made up of many ideal applications is much like saying "apply the Unix philosophy here", but it is actually more akin to "design tools like the command line". The utility of the command line tools lie in the Unix pipe, which routes messages from one application to the next. The greatest weakness of the pipe is that it is unicast, a one-to-one delivery of data in only one direction. As you move into applications where messaging is broadcast and switched, where you can have more than one consumer for any message sent, you open up a world where you no longer need to think about an application as performing more than one task. Rather, you can design applications which each perform a single task and work in unison and in parallel to perform all of the tasks that need to be done in response to the given message. Think of any massively multiplayer game. Now imagine that you're writing the AI that controls a single NPC in a game world, in which 10,000 simultaneous users are attempting to talk to you. Your AI can be written in such a way that you can have a single instance with a very long queue of incoming messages, or you can have many instances of the same program each with a very short queue of messages from all of the parallel users. The users' interactions themselves are brokered by PC objects, that tend to exist on a 1-to-1 relationship with the user, but there is no reason that each of the uniquely specialized behaviors brokered by the UI can't dispatch to dedicated applications that relate information about game state to the user. This form of ensemble programing is terribly powerful once you adopt the mentality that an application is made up of many applications each of which only do one thing well. One of the primary disadvantages of protocols like HTTP is that they're point-to-point protocols, and not designed for programming at large. Protocols like AMQP, XMPP, and even SMTP have significant advantages over HTTP in respect to using them for interprocess communication in that they provide facilities for addressing multiple recipients with a single request. While these transactions are still largely routed over P2P connections using TCP, this is worked around at the application layer of the network.
It has been said that perspective is worth 80 IQ points and I tend to agree. One of the great traps we all tend to fall into is failing to test our assumptions. I wrote tools to test mine. I challenge mine on a regular basis by practicing my art, and attempting to revisit the assumptions I make without thinking. Articles like this also allow me to put words together to try to explain the principles I develop in my practice, so that I can better understand the experiences I've had. By communicating those experiences, I also get to challenge my own understanding by attempting to see them through other people's points of view. The weekly retrospective at work was one of my initiatives to try to get my developers to consciously do the same. Sometimes it is funny how meta this game can truly become.