simpaj

Scalability is the mind-killer

Architectural choices come with opportunity costs. That cost is often a certain rigidity or inflexibility of choice, leading to a loss of velocity and creativity. You'll typically start seeing the symptoms of narrow standardization in the form of architecture-driven or DevOps-driven development, which is when teams have to shoehorn features in to the existing patterns instead of letting the problem inform the architecture.

Most notably, I've seen this in architectures that design for scalability and loose coupling up front. These organizations tend to assume a state-like view of top-down design, focusing on standardization and legibility above all - directives that are usually dictated by people that are not "in the trenches", e.g. your architects or hand-off CTOs. Not having "skin in the game" means that one won't deal with the consequences of ones decisions, leading to misaligned incentives between the decision-makers and the teams.

There's a certain backwardness to this in the thinking that formalization precedes or goes hand in hand with understanding. This thinking is deeply rooted in the high modernist belief that complex environments can be made legible through planning and systematization. The supposed benefits to this order is efficiency through global abstractions and one-size-fits-all designs, in turn making the system more predictable and reliable.

The gist of the argument against that thinking is that there is a certain overconfidence in top-down design and its abilities to capture the nuances of local phenomena. The insight is that the purported goals of said designs are rarely achieved in practice since, on a local level, people operate differently and, in the best case, will work their way around global standardizations to achieve local optimizations. It also completely discards any emergent properties of the system that arise due to interactions that are not accounted for in the design.

I won't go into the details here, and will just refer you to the writings of Jane Jacobs and James C. Scott for more details on the arguments against a high modernist world view. I wanted to draw a parallell to these ideas from a tech point of view. In technical organizations, these top-down designs impact both technical architecture and organizational design.

It is not uncommon to see the former be born out of prior experiences of engineers, as opposed to being informed by product decisions. This is usually the most insidious type of standardization, since its effects are unintentional and its proponents tend to identify with a particular way of architecting systems, rationalizing their decisions post-hoc. I call this DevOps-driven development and it is the death trap of previously succesful product companies. The urge to micro-optimize the technology leads to an inadvertent loss of feature velocity, due to the overhead of keeping up with the standards.

Organizational design is usually intentional, and therefore more in line with the high modernist ideas about legibility. The arguments for it stems from a real understanding of Conway's law, whereby an organization is split up along product boundaries to decouple teams from each other. The problem is that, while the theoretical understanding is real, the practical application of it is misguided. Shipping your distributed organization chart leads to highly fragmented products and user experiences, with high coordination costs across product boundaries to ensure consistency. Linear is an interesting example of a company that instead works backwards from the product they want to ship (i.e. their strategy), and letting that inform their organizational structure.

So, what does all this have to do with scalability? Scalability is the common denominator of the reasons given for these top-down designs. It's the be-all and end-all of standardization and global abstractions, both technically and organizationally. "If we don't do this now, we won't be able to scale later" or "why not trade a little bit of flexibility now for the ability to scale later?". Well, because that trade almost always negatively impacts the odds of getting to the point where you need to scale, mainly because it is not rooted in solving actual product problems. In fact, it can even hinder locally optimized solutions to problems if that solution does not fit in to the global structure.

That is not to say that all companies need to act as if they're pre-PMF startups in the lawless wild west, but all companies need to be mindful of narrow and early standardization. And it's not only about velocity. Standards and processes that are too numerous and too heavy act as guardrails for creativity as well, hindering thoughts from veering from the beaten path and anchoring the discussions in the familiar. This is a sort of organizational monoculture, where a group of people sets the direction early, at the expense of divergent ideas. It is not a bad thing in and of itself, but it always leads to reduced flexibility in quickly reacting to changes.

The key takeaway here is to remember the failures of high modernism. High modernism failed because it ignored local knowledge, organic growth and the adaptability of truly decentralized systems (note, systems, not architectures). Systems grow in unpredictable ways - cater to and find comfort in that unpredictability. Let those closest to the problem, the empowered product team, solve it in the way that they see fit.

Scalability, whether technical or organizational, is not an end in and of itself. It is a possible side effect of product challenges, themselves the result of some strategic insight and subsequent bet. As an engineer, be mindful of when your mind veers towards it without a clear answer to the question of what problem you're trying to solve. Otherwise you'll risk standardizing too early, throwing away local optimizations in the process and, in the long-run, negatively impacting the odds of reaching the thing you wanted to build for - scale.