How to Accept Over-Engineering for What It Really Is
Over-engineering is a tough thing to discuss and there&#x27;s never going to be a suitable answer anyone agrees upon. It is also a huge topic with a scope far beyond this textbox or a medium article.<p>Anyway, among many of the big issues here are so many things are contextual - the problem domain, stakeholders, programmer abilities, business climate, etc. It&#x27;s been my experience that you cannot apply judgement to a solution without understanding as many of the variables involved as possible. Shifting these variables even slightly can change everything.<p>For example, say I build a feature A at company Y and then I change jobs next week and to company Z which is much smaller. Company Z wants the same feature I just built at company X, but of course I can&#x27;t steal it outright. Building the same thing at company Z requires different engineering decisions, given the different climate and probably different development team, programming language, and so on hypothetically. Even if I could just copy the solution, at company Y it might be perfect, while at company Z it might be considered over-engineered. It&#x27;s a combination of both objective and subjective judgements.<p>As for the concrete implementations of features, I often am driven crazy by both over and under engineering. I worked with several people in positions of power above or horizontally to me that would routinely shoot down many well-engineered features and implementations by other people because they were, &quot;Too complicated.&quot; I began to realize this was often shorthand in most cases for, &quot;I know nothing about this, my ego or political stance is threatened, I lack talent, and&#x2F;or I am simply a moron.&quot; Of course I also hit plenty of actually over-engineered things that were indeed exactly this, but in many of these cases it was obvious. Where it is less clear is anything of decent scope, challenge, subjectivity, or other hard to quantify measure. I think people can look at something like the FizzBuzz and see that the parody versions using OO 10 design patterns with abstract factories are obviously over-engineered, while it gets much harder in the context of a big project.<p>And this leads to another problem. Complexity is not over-engineering, nor vice-versa always. Nor is scope, or code size, or any of these related concepts that come to mind. A very small amount of code can be incredibly complex, a large amount of code can be simple, and so on. Either can be over or under engineered. Just look at math - some of the most useful tools are very difficult to understand, while others are incredibly simple with tons of layers, nuances, and so on. Programming is not that much different in this regard. We&#x27;ve all seen things done in a tiny amount of code we would have never thought of and maybe can&#x27;t even understand. Likewise, we&#x27;ve encountered huge amount of code that someone complains about but is written clearly, thoughtfully, and cleanly. These&#x27;s a good related talk on some of these ideas by Rich Hickey if I recall.<p>Another issue is that it is hard to quantify both knowing what and when justifies substantial engineering efforts. As I mentioned, context plays a big part, as does intuition and experience (in relation to ability, not just pure years). These are all nearly impossible to quantify, and yet must play a large role in cases where large engineering decision making. A lot of people see things they don&#x27;t understand or are new to them and just throw out the over-engineered card. Likewise, a lot of people love pulling out their favorite tool (ex: SQL, Framework of choice) and try to use it as a golden hammer. Anything else is &quot;over-engineered.&quot; These things can lead to over-engineering or at the very least, poor engineering.<p>Over-engineering is also something that is relative. It is really hard sometimes to work with someone else and to get them to understand especially those hard to quantify things such as intuition. Programming can also often highly be about ego and ownership. People will not hesitate to point at something not written by them as over-engineered because it was not done how they would do it.<p>Game programming for me again comes to mind as an area where this is a constant pain. The inexperienced game programmer will both under and over engineer things like you&#x27;ve never seen. First, they&#x27;ll try to use their favorite language to write a game. They&#x27;ll then do things when they learn a technique like OO programming and try to apply it everywhere, to everything. &quot;A monster? Obviously I should make a monster base class, then subclass this for an Orc, then subclass it again for an Elite Orc.&quot; Everything must be designed around the tools they know instead of the problem. The minute this person does enough in their career to get any real authority or a real job, they take this unchecked mentality and try to impose it on the people around them.  I see the same thing in web application and desktop programming. Someone learns lambdas, functors, currying, design patterns, IOC, monads, etc. and they must apply them to any situation that triggers this recognition. The &quot;engineering&quot; then comes from the solution and tools, perhaps the XY problem in many cases. Usually these people also fail to consider the aforementioned contextual concerns.<p>I know in game programming this happened often where some programmer would come in and replace some lengthy looking code that made heavy use of arrays and manual for loops. They&#x27;d then go in and replace it all with vectors, iterators, and all kinds of heavier stuff. They never considered that the code was engineered that way for performance or for portability or backward compatibility or some other reason. Moreover, they didn&#x27;t actually know that the seemingly &quot;simpler&quot; solution was actually making things as a whole more difficult by pushing out these concerns to be fixed or handled elsewhere (ex: making up performance somewhere else).<p>On a personal note, I am often annoyed by the over-engineered finger pointing. There are so many things like XML messes and 200 steps to setup something for a simple purpose that are easy enough to agree on as over-engineered. Most of these come down to asking yourself, &quot;What problem am I trying to solve&quot; and perhaps repeating those steps aloud. If you feel ridiculous saying them, especially to another person, something might be wrong. But if it turns out that the problem you are trying to solve really does require something that is not a sound-bite, the problem explanation will also often have to be long and cannot be skirted around without ignoring the problem.<p>Too often I see things in this industry thrown out there to &quot;fail hard,&quot; to be an MVP, to test the market, to disrupt, to fill a gap. That&#x27;s all fine, but make sure you are solving the problem and not creating new ones like security, performance, misleading people, defrauding investors, or worse because you thought engineering a proper solution is too hard or takes too long. Find the sweet spot and make informed decisions. I recommend another talk about Rich Hickey (sorry two by the same person, just memorable talks) about Hammock-driven development and thinking before acting. If more people did that, it could swing either way and maybe we would have less short-term &quot;stuff&quot; but better long-term solutions, less over-engineered messes, and  things that actually are worth building on top of as bases. As it stands now, this industry is littered with poorly engineered solutions we were stuck with because of both over and under engineering.