There Is No Silver Bullet, Yet
Remembering Fred Brooks' iconic essay in the current times
Book reading is something that I love, and it was the most enjoyable way to spend those countless hours in traffic while on my bus commute to work when I was younger. While book reading is largely replaced now by listening to podcasts, I vividly remember the books that I’ve read. The one book that I cherished most reading cover to cover is “The Mythical Man Month”. It is the book, that is most influential and shaped my perspectives on Software engineering and system architecture.
The author of the book, Fred Brooks died last week, and that made me relive many things I’ve learned from the book. For the uninitiated, Fred Brooks had managed OS/360 (an IBM Mainframe OS that was never released, which is a precursor to OS/390 & z/OS). He was awarded ACM Turing Award for his contributions to software engineering. RIP, legend.
The book has so many interesting chapters, but the 16th chapter clearly stands out. “No Silver Bullet - Essence and Accident in Software Engineering” is legendary, and is still as relevant as it was before. As a systems architect for complex systems, I always saw managing complexity as my primary job. Despite all the technological advances, and even in 2022, handling complexity is still the most difficult part of my job.
As we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.
Today, more than three decades from when the original essay was published, the above prediction mentioned in the Intro of the essay is still valid.
Aren’t we making progress?
It would be unfair to say we are not making progress in managing the complexity. But it has been “Two Steps Forward, One Step Back”. We are able to build more complex systems than before, with much smaller teams. Out abstractions are constantly getting better. Purpose-built software systems are now more common than before. But the complexity that we are to manage now also has grown exponentially.
Fred Brooks categorized the difficulties associated with software development, into two: Essential, and Accidental.
Essential complexity is inherent in the software. Software being a conceptual medium, the complexity associated with bringing ideas into production is as complex as ever. Translating ideas into features, features into code, code into components, components into services, and services into a usable system riddled with multiple tasks, each of which is inherent and complex. Added to that communication of ideas & conceptions, dependencies, and interactions among people are essential in any complex software development.
Accidental complexity, on the other hand, comes from the way computers & tools are built. These complexities arise out of the programming languages, frameworks, and other such abstractions that we use. For example, knowing that the Web is primarily meant for documents, and is not a great medium for delivering applications, but making it the de-facto for application is indeed accidental complexity.
Accidental Complexity Is Still a Nightmare
As per Brooks, essential complexity is something that we cannot avoid. But he was very hopeful that accidental complexity can be reduced significantly over time. He hoped Object Oriented Programming, AI/Expert Systems, and Automatic Programming are the Silver Bullet hopefuls, which can bring down accidental complexity.
But I am still skeptical if that can be the case. Over the three decades, we indeed made big strides and reduced the accidental complexity. However, as we just made strides in each of the architectural styles that influenced a particular decade, new forms of accidental complexity have emerged.
With Client-server architecture [dominant architecture style in the 90s], we made progress with OOP, components, and rapid application development.
But when we had to develop for Web paradigm [dominant architecture style in the 00s], brought in a whole lot of new accidental complexity.
As we just figured out the Web, we had to deal with “Digital First” and “Cloud Native” architectures now and had to build for the internet scale, cloud-native platforms, and changing user experiences.
The goalposts have been constantly moving. The net result though, we are yet to conquer the accidental complexity.
Is there a way out of Accidental Complexity?
I expect over the next couple of decades, the accidental complexity continue to be the same or grow. Why do I say so?
The tools that we use, and abstractions that we deal with are simply unsustainable. Let us take a case of writing a simple web application using React JS. It all starts with what React framework to choose, Next.js, Gatsby, or Remix? Then we need to deal with NPM modules, CSS frameworks, build tools, CSR vs. SSR, API integrations, OAuth, and so on. Integrating them and putting them together along with the tools, add to significant accidental complexity, and we are nowhere close to eliminating it.
Even if we can reduce it in the next decade with better abstractions, there are new architectural paradigms emerging. Heard of Web 3.0, and Web 5.0? Whether these buzzwords are real or hype, we are certain that the current Web paradigm is not how users will interact with computing in the future. Ambient computing, Metaverses, will require drastically different engineering methods, programming environments, abstractions, and tools. Each of these will add to the complexity.
While Brooks has been hopeful of Sliver Bullets for eliminating accidental complexity, we are nowhere close. Here is my prediction- “Over the next two decades, there is no single development, in either technology or management technique, which by itself promises order-of-magnitude improvement in managing accidental complexity”.
How about Essential Complexity?
Brooks proposed four ideas to handle essential complexity.
Exploiting the mass market to avoid constructing what can be bought.
Using rapid prototyping as part of a planned iteration in establishing software requirements.
Growing software organically, adding more and more function to systems as they are run, used, and tested.
Identifying and developing the great conceptual designers of the rising generation.
Brooks got most of the ideas right, which is kind of amazing.
Cloud Computing and SaaS software is essentially the first idea in action now.
Not much needs to be said about the importance of Rapid prototyping, MVPs, and how these are helping us deliver better software, and improve certainty.
Microservices architecture, Event drive architecture, and APIs have fundamentally changed the way we architect systems. Instead of the architecture, we are growing, and evolving the architecture.
However, the fourth point is where we haven’t made much progress. In my view, we have moved backward. It is unfortunate that we despise modeling languages and design patterns but rely on the heroism of the 10x developers. The culture of heroic developers, and throwing money at engineering, will only produce systems that accrue technical debt and are unmaintainable over the longer term. The leadership backlash is already in action with all the public dirty linen washing going on with Twitter. Cindy Sridharan has a few interesting tweets that are worth reading.
My Silver Hopefuls of the future
The accidental complexity that we currently manage is simply unsustainable as we go into the future. The platform abstractions that we are dealing with are still primitive, despite the major strides we’ve done. The tooling that we use is still as primate as it can get. Here are my hopefuls that can help us reduce the complexity.
Standardized Platform Abstractions - Despite VMs & Containers being good abstractions, software engineering teams still spend a significant amount of time on taking the code to production. PaaS systems like Cloud Foundry and Serverless Platforms like Lambda are good starting points, but there are too many variants and too many choices. The K8S ecosystem is fragmented and is driven by the interests of the CNCF members, and they are hardly building enough abstractions. Take the case of the Serverless Working Group, the group has seen barely any activity in the past year. This is a case in point, too much choice is always not good, and the interests of platform providers are not always aligned with consumers. There has to be an urgent call for action, for the software consumers to come forward, and not rely on the platform providers to standardize and build better platform abstractions.
Smart Tooling - It is appalling to see the tooling that we use is stuck in the mid-90s. Devs using system logs to debug software in this age is disappointing. Part of the problem here is that building tools to handle diverse software ecosystems is a complex undertaking. Added to that, the fact that Developers are not willing to pay for tooling has stalled the tooling advances. Kite AI coding assistant is shut down yesterday, citing the unsustainable business model. While I am not positive, I hope there will be tooling that offer a better developer experience, and bring in AI to make them productive. Smart tooling will be the single most enabler for developer productivity.
Better Programming Abstractions - Like tools, programming languages are stuck in the past. We are making barely any advances. The programming language continues to use the same abstractions that we used in the 90s. Languages like Ballerina and Elm are refreshing, but still a very small niche. It is also time to look beyond programming languages. Combining the higher level of abstractions offered in LC/NC(low code/no code) and bringing them into programming languages will definitely help. Programming languages that can offer better domain abstractions, and better component models are the need of the hour.
Modeling and Visualizing the Systems - After the UML over-engineering, the software modeling unfortunately has become an anti-pattern and is seen as old-fashioned. There is hardly any common language now software teams can use to model, visualize systems and communicate the design. I am hoping that there will be innovation in this space over the next decade that will help us model complex systems easily.
What are your views on software complexity?
As a closing note, I recommend everyone who works in software engineering or is remotely associated to read Mythical Man Month (affiliate link), or just Chapter 16. Let us continue the conversation. What are your views on software complexity? What are your silver bullet hopefuls for the next decade? Please leave comments with your views.
Sad to hear the passing of such an influential software engineer. As relatively new into the industry i have been reading about Fred Brooks Essential / Accidental complexity definitions only recently; and it is what drove our team to design and launch bondy.io, an open source application network platform that radically simplifies how applications and services communicate with each other. Building distributed applications is complicated, we believe there is an easier way. End users just wants it to work, always, and everywhere. Accidental complexity is dragging us all down, we believe there is an alternative.
I dont see much hope in reducing software complexity as its driven by demand and most businesses thrives on complexity to create differentiation. Also we seem to have addressed most of the "low hanging fruits" in automating manual business processes over the last few decades, we are left with increasing complexity which requires more compute, storage and networking power. Demand for distributed computing is becoming more prevalent with globalization, and it spawns evolution of multiple programming and platform stacks supporting varying scenarios, geographies and regulations. Since design-time components are driven by the run-times, they are not getting any simpler either. Most of the attempts at LC/NC tools has been quite limited in its impact and I dont expect it to make much difference as the demand changes faster than the speed of developing these tools.
Interestingly I also notice that any trend that seems to simplify aspects of software development are short-lived and will quickly turn complex in a couple of "Kubecons" (:p) and will end up needing more efforts and money to leverage its new features.