Modules Considered Harmful

This past week, I was reading some code ( I read a lot of code ), and noticed that someone had taken a module I had written and rewrote it, breaking the interface and all of the code I had in 8 repos in different branches. After lots of swearing ( I swear at a lot of code ), I tried to divine why they decided to make this change. And only one word came out that made any sense "modules". You see, because I had written a single regex with a paragraph describing the output of that regex, and documented the interface of the function in the documentation that was used in 8 different repos, the team decided that it would be better to use two "modules" provided by 3rd parties, and then wrap the interfaces of those two "modules" and create a new undocumented interface that broke all but the 2 of the 8 they decided to modify. Because "modules".

Now, they admitted they didn't bother to read the source code of the two modules they used to perform the same function as the single regex. After all, the amount of code in those modules added up to 26,519 lines of source code. It was much easier to include the 26,519 lines and not know how the module worked than to read a single line of code that was well documented and consistent with the non-source code based documentation for the project and the design of the API. Better yet, they had to add an additional 20 lines of code to the module to accommodate the differences in requirements between the upstream modules output and the regex. They then proceeded to add another dozen lines of code to the two modules they had bothered to modify to accommodate the changes they made to the API. Finally, the 6 other repos that relied upon the old API were left to rot, a stinking time bomb of technical debt, because "modules".

This is only the most recent in a never ending series of experiences which lead me to believe that modules are fundamentally harmful. You see the problem is that information hiding is just an excuse for ignorance. Culturally, we don't create modules to encapsulate information about processes in easy to digest units. No! We create modules so we don't have to give two shits how something works. We use modules because we simply don't give a fuck how it works as long tests pass and we get paid. You see, developing intuition about how a complex system works is hard work and requires attention to details. Designing complex systems is even harder work, as we have to apply our intuition to develop and experiment and then test our assumptions until we devise a system that actually works in reality. Worse yet, our intuitions are often wrong, and require constant revision and difficult self assessment which may prove we don't know shit about whatever the fuck we claimed we did last time period. Admitting you don't know is painful, and it is much easier to just use an off the shelf module without understanding it and pretend that someone else thought through and solved our problem for us.

Yesterday, I was debugging a project I was working on using a 3rd party module. I noticed something funky about the interface, in which I was getting unpredictable behavior from an interface that was occasionally sending 1 or more registration messages to the server. My hunch was that the retry logic was kicking in, and decided it was time to go read that part of the source code ( I read a lot of code ). Sure enough, in the retry logic there was a timeout of 50 ms for an ACK from the server. Even on localhost, whenever I was playing a video on Twitch on my 4k display and running a bunch of servers, and doing any load testing at all on localhost, I would often get worse than 50ms response times from the server on localhost due to kernel scheduling. The single hard coded 50 on one line I had skimmed over on my initial evaluation turned out to make the entire module unsuitable for real world networking. By the end of the afternoon, I had decided that I needed to rewrite large portions of the upstream module, as it would never meet our basic requirements. So the remainder of the afternoon, I spent reading two RFCs that the module implemented and realized how utterly broken the actual specification was in the first place, and discovered two other sections of the RFC which the module inverted the implementation details resulting in it being incompatible with two other libraries I was evaluating. Once again, my experience lead me to believe that the modules are harmful. Even the RFCs are so full of shit, that if you actually implement it, you'll quickly discover that they are fundamentally flawed in so many colorful ways, no real world system can implement them and work with any degree of reliability!

At this point, this may all sound like sour grapes, but I have repeatedly stated the solution to the problem. Read. Read all of the code. If there is too much code to read, throw it the fuck out and write just what you need. Stop using modules, unless you are willing to read them, understand them, and adapt them to your problem. I often find myself at night reading through Linux kernel code, or on weekend reading things like the musl libc source, and thinking about what bits I actually need. Last week, I got annoyed at having to compile 131 files (11.9 files per line of code of app), in an embedded project, to blink a fucking LED, so I read all 131 of those files. The discovery I made was nearly 1/2 of those files described virtual classes that the other of the half of the files implemented. Most of those classes were entirely unnecessary, but were compiled because of inter-dependencies between components that were entirely unused. Modules on top of modules, because modules. Doing the same thing in assembler took the same number of lines, without the additional 6,615 lines of C++ code, 28,129 lines of C, and 187,360 lines of header files. That was 222,104 lines of source if you weren't counting, just to get an environment in which we could blink an LED on a single board. I wrote 11 of those lines.

Modules are harmful because you don't need to understand how they work, or even why they are there, to get paid to use them. They multiply like bad thoughts in an obsessive compulsive suicidal mind. There is no escape from them, and they encourage bad behavior because the work they create is it's own reward. It is why "Software Developer" has become one of the most common jobs in 4 states. Modules produce more make-work, as evident by my own experience with my own developers ( who should know better ). They promote over complication, misunderstanding, and ignorance. The artificial complexity hinders the development of understanding of the necessary complexity of solving the real problem. That said, modules are great for PHBs and other people who are rewarded for managing buts in seats, or for creating the illusion of value among people who mistake bigger for better.