One of the key concepts behind the design of any system is that of repeated application. By using an a-historical process, one can reduce risk of catastrophic failure by relying upon a general pattern of behavior being well established. When a system fails, this should be a historic event, when the normal operation is suspended and an exceptional event occured. The study of history allows us to identify root causes of failure, but can not shed light on the humdrum banality of normal modes of operation.

Well built systems treat failure as q normal mode of operation, and avoid historic failures by normalizing operation to expect everyday failures. Poorly built systems do not account for failure as a mode of operation and produce significant degradation of service as a result. Take for example a bridge in an earthquake. If tolerances do not account for a sufficient degree of lateral and vertical movement of the road way, we will read about it in the news. But if the system is well engineered and built to treat earthquakes as a normal mode of operation, then we don't see headlines about how it didn't fall down.

When thinking about your systems, it is important to think about the normal modes of operation in terms of how developers will interact with it. Engineering in levels of priviledge, roles and responsibilities, and apropos skills will ensure that development can be treated as a normal mode of operation. Limiting your junior developers to cosmetic or non-critical functionality is wise. Designing a system of interfaces to empower them while isolating the tricky bits is wiser. Ensuring that your senior developers can build infrastructure without understanding the full system is important, but so is ensuring that each can contribute in accordance their areas of expertise is more important. Designing systems where subject matter experts can exploit their skills requires a great deal of flexibility on the part of the system designer.

Finally, building teams that work well and respect each other depends upon how you engineer the political structures which guide your development process. If developers do not have choices, they can not develop their better judgement. Proscribing solutions is rarely effective, but designing systems so that the correct path is the easy path is. Without the assumption that your developers are professionals capable of making reasonable decisions at their level of expertise is critical to building a team that works. Failure to acknowledge the limitations of those individuals, however is certain doom. As with most design decisions, you want to rely upon a repeatable process, but acknowledge the peculiar context in which the entire system must function.