The people who create software generally refer to themselves as software engineers, and yet if they graduate from university, it is typically with a degree in computer science. That has always felt a little strange to me, because science and engineering are two pretty different disciplines – yet we for the most part seem to take such an obvious contradiction for granted. However, I think there is something uniquely magical about software, and part of that magic might stem from this tension in how we define it.
At first glance, software seems like a straightforward engineering practice. After all, it exists within a fully known universe – the computer – of our own making. This is in stark contrast to disciplines that are more obviously science, like biology or physics, which exist within domains that we did not create and do not fully understand. Unlike computers, which are fully understood, those disciplines are in large part about the search for understanding. Science, in other words, is often occupied with the process of discovering how something works – which we don’t need to do with computers.
Since we understand everything about the computer and don’t need to discover its workings, software seems much more straightforward. With nothing to discover, we might say that software is the engineering practice of combining and assembling what is available from the complex system of computing in order to manifest a given vision.
However, I don’t believe that is the full story. I want to suggest that the relationship between vision and engineering in software is often intertwined and bidirectional rather than linear, and that the whole of software development is actually full of discovery even though it exists within a completely known universe.
Consider these computer graphics animations from the era when I was first getting interested in software:
These are from an extremely primitive era, and yet they are beautiful and compelling. The interesting thing to me about these animations is that they are not actually “animating” – at least not in the conventional sense, where the value of each pixel is calculated and rendered 30 times a second. It certainly looks like that’s what is happening, given all the dynamic movement, but that would not have been feasible based on the hardware constraints at the time.
Instead, these were actually images that made use of a technique known as color cycling. Color cycling exploited the fact that graphics systems like VGA used “indexed” color. Indexed color systems were a memory optimization technique for representing each pixel on the screen with a single 1-byte color value rather than a full 3-byte RGB value, along with a separate “color pallet” that mapped these 1-byte color values to a full 3-byte RGB value.
Someone who understood how indexed color systems worked realized that it would be possible to create impressive (for the time) video effects by carefully specifying color values for an image and then intelligently cycling the 256 colors in the color pallet to create the appearance of animation.
Combining tricks like this are what enabled the famous Amiga boing ball demo:
I think this was a somewhat non-obvious result. Indexed color systems weren’t created for this purpose, but software developers who understood them were able to use them to create something unexpectedly great.
Computing in general is a long history of non-obvious results. I mean, if I had been there at the beginning of it all, I’m not sure I would have predicted that by quickly turning electricity on and off, we could get… all this?
Much like with pure science disciplines, software’s history has been a history of discovery. However, while understanding is the object of discovery in biology or physics, understanding and engineering is often the basis for discovery in software. While software development is indeed the process of combining and assembling resources to manifest a given vision, deep understanding of the resources and all the ways to use them also inform the vision in a bidirectional relationship. It’s not always as simple as starting with a vision and then using the resources available to build it. They often emerge together or in an intertwined way, and things continually leap forward over and over.
If we consider the animations above to be a software product, it is clear that the product did not begin with a vision that someone had for animations which looked this way. It was an understanding of the engineering that informed the vision, and vice versa.
I’m using color cycling as an example because it’s visual and so old as to be uncontroversial, but it’s a pattern that I’ve seen appear again and again throughout the course of my involvement with software: deep understanding of tools and technology yields a new vision, unexpected results, insight, or a solution of some kind (big or small!). That doesn’t always mean a new product or demo, and more often than not is simply an advance within a project that only a small group of people will ever take note of, but it almost always starts with the understanding part.
I feel like we might be losing sight of the understanding part.
Abstractions: Building More, Creating Less
Software development has the benefit of relying on abstraction layers. It would be incredibly arduous to have to implement all of the logic associated with storing data to disk every time one needs to store data to disk, so abstracting that out into a reusable storage interface like an RDBMS or a KV store is, of course, greatly valuable. The same can be said for the value of an HTTP library, sockets interface, C compiler, a game engine, or any other abstraction layer.
However, there are two ways of interacting with an abstraction layer: as shorthand for an understanding of what it is doing on your behalf, or as a black box. I think engineers are most capable and most effective when abstraction layers serve as shorthand rather than a black box. Computing is so complex that these interfaces are never perfectly clean (and never will be), so to use them as a black box without understanding what happens within them can lead to negative outcomes which compound with the size/complexity/scale of the application.
Most people who work with software generally have a sense of this. But to me, there also seems to be something more going on, which is related to that intertwined relationship between engineering and vision: when we see everything as black boxes, it can be difficult to have any vision at all.
For instance, sometimes we create abstraction layers that allow people to create things on top of them explicitly without having to understand anything beneath them. We call those “platforms.” The expectation is that when we create abstraction layers like that, we should see an explosion of creativity, since now people can focus only on the creative aspects of what to build rather than the technical aspects of how to build it.
This seems obviously true for some endeavors outside of computing: a filmmaker can use a camera to make great films without understanding in detail how to build the camera, and there is probably not much of a predictable relationship between knowledge of how the camera works and quality of the resulting film. We may be tempted to extrapolate from this into the world of computing and say that by creating something like a game engine, we should expect to see an explosion of creativity in games, because now game makers can focus on the creative aspects of building games without having to understand the technical aspects of how to build a game – in the same way that a filmmaker doesn’t have to build a camera.
However, I think the problem with metaphors like these is that a camera is not actually an abstraction layer for a film. A story does not simply sit on top of a camera. There is a lot of other really complicated stuff that a filmmaker really does have to understand about making a film that is necessary in order to bring a story to life.
When we apply this logic in the realm of computing by creating explicitly black-box abstraction layers, I think we do actually see an explosion of more stuff, but it is often a lot more mediocre stuff.
As we’ve made it easier to build games, we have certainly seen more of them. But the number of highly rated games (in this case as recorded by metacritic) does not seem to be increasing over time.
To me, in complex ecosystems like computing, there just seems to be some kind of ongoing relationship between deep understanding of how the tools we use to create things function and the quality or creativity of the output we get as a result.
I think we can only truly create when we truly understand what we have to create with.
When Teams Become Tupperware: Sealed Tight and Stale Inside
Maybe that much is obvious, but I think this applies in all kinds of meta ways to the software industry as a whole. For instance, I think many of the dynamics that make a good engineer are also the dynamics that make a good engineering organization.
Engineering organizations today have ballooned to huge numbers of people, but these huge engineering organizations don’t exactly have a reputation for high velocity output. Some of this is the result of what happens with products at scale: it is just fundamentally faster and easier to iterate, improve, or change a product with 100 users than it is to do the same with a product of 100 million users.
However, having seen inside some large engineering organizations, I don’t think this explains everything.
Most people managing or building engineering teams today have a familiar management philosophy. The general thesis being that hierarchical decision making – where information flows upwards in the form of reports, summaries, and executive summaries, and then flows back downwards in the form of budgets, headcount, and priorities – is too slow.
Instead, it is now more common to think that organizations should be composed of autonomous teams, and the function of leadership in an organization like that is to create “alignment” across all these teams by ensuring that they are pulling in the same direction towards the same vision:
--->
These two ways of thinking about organizations (autonomous vs hierarchical) seem really different at first glance. However, in one respect, they are both essentially the same:
Both assume a relatively unidirectional relationship between vision and engineering. The substantial difference is the way execution is organized, but neither leave a lot of room for that bidirectional relationship between engineering and vision to emerge.
To exacerbate this, organizing engineers into silos is very similar to creating abstraction layers for an organization, almost like a microservices architecture. Unfortunately these abstractions are almost always treated as black boxes, both by peer teams and by leadership teams, and much like in software, living in a world of black box abstractions can make it difficult for anyone to have the insights that are necessary for truly excellent outcomes.
A thought experiment I like to apply to an engineering organization when considering its efficacy is to imagine that it is Skype at the dawn of “mobile.” Imagine that you’re running Skype, that you have all these people siloed into autonomous teams, and that you have the foresight to see how important mobile is going to be. So you make sure that everyone is aligned on mobile as the key, number one, absolutely existential priority for the organization.
Even with everyone aligned, adapting to mobile is a big lift. The existing product is 100% organized at every level around a desktop model. The UX is centered around the concept of being “online” or “offline,” which no longer exists in a mobile environment. The communication protocol is a semi-distributed mesh that is dependent on being able to run in the “background” on a desktop OS, which won’t work at all on a mobile device. The code base is written in a way that is not deployable on mobile OSes, etc etc.
Clearly things need to change. Almost everything needs to change. But the change that every team needs to make is dependent on the change that every other team needs to make. The product vision itself is intertwined and bidirectionally informed from the engineering, but if everyone in the organization sees every other part of the organization as a black-box abstraction layer, I don’t think a change like that is going to happen.
While seismic shifts like the mobile revolution may seem rare, I think organizations ill-equipped for radical change are likely underperforming even in stable times. Just like with software, where deep understanding is often the basis for discovery, an organization has to truly understand itself to be primed for innovation. When teams operate as impenetrable black boxes, vision becomes myopic, and potential withers. Despite having immense talent, the large engineering organizations of SV often structure themselves in ways that isolate expertise. As a result, they’re probably sitting on a huge amount of untapped potential – creative and technical ideas that never connect.
Fruit loops and false idols
I’ve noticed that aspiring entrepreneurs often fixate on studying the current habits of successful entrepreneurs, hoping to mimic their achievements. The problem with this logic is that, right now, whether Jeff Bezos starts every day with a cold plunge at 5am or a bowl of fruit loops at 10am probably has minimal bearing on the outcome of his companies. If imitation is really the goal, people should probably be looking at what he was doing in 1994 – which was probably just working obsessively to the detriment of all else.
I think there is a similar fallacy for how we consider engineering organizations. Many of today’s “best practices” have been drawn from long-established internet companies like Google. However, the problem with copying their current practices on the basis of their success is that most of those companies found near-invincible business models that basically printed money, and so almost any organizational or management practice developed or selected at random likely would continue to be “successful” to some degree. Without much in the way of selective pressure, I think the practices and cultures that have developed at these companies more often reflect optimizations for the benefit of various decision makers rather than optimizations for actually creating something great. The most obvious example being the size of the organizations themselves, which I think is much more often the result of the benefits that accrue to the hiring manager for having more “reports” than to any actual relationship with capacity, ability, or velocity.
The people from those companies became the management hires at new companies, and the people who learned there have become the management hires at other new companies, and eventually we have come to forget that there was any other way of doing things before all this.
If you’re building a company or an engineering team today, I think the real opportunity lies in understanding that you’re not bound by the conventions that have been passed down as “standard.”
The qualities that make a good engineer are often the same ones that create a good engineering organization. Both start with deep understanding as the basis for innovation – cultivating the curiosity to look inside the black boxes. Both recognize that vision and engineering are often deeply intertwined and mutually informative rather than linear. The magic of both software and software organizations comes from those moments where insight into how something works sparks entirely new ideas about what it could become.
