Technical Debt: How to Duct Tape Strategically
A system that has been built “exactly how it should be”, has no technical debt; it is easy to add new features because the architecture is simple, and well-engineered to meet specification. With new projects, this comes for free, since the simplest system is no system at all. As the codebase grows, requirements change and external pressures develop. At this stage, implementing a new feature “the right way” may require rebuilding of components of the system to serve a new purpose, integration with other systems, or coordination with other people.
In this environment, the “duct tape” solution that delivers the requested feature, but leaves the design and simplicity of the overall system worse off, can seem very appealing. The result is that the feature is delivered more quickly, at the cost of the future when the system needs to be re-architecture to remain simple and well-engineered; in other words, technical debt has been incurred.
What’s so bad about technical debt?
It can be tempting for teams to take on technical debt since it builds a reputation of “getting things done”. As time goes on, however, the debt has to be repaid, at which time even simple feature requests can take a long time, as the feature needs to be built alongside large-scale architectural changes. This makes the already notoriously difficult job of software estimation, almost impossible. It is the unpredictability of when the debt will need to be repaid that makes holding onto a large amount of technical debt dangerous. It is this unpredictability which has lead some to argue that a better analogy for technical “debt” is an unhedged call option. In the face of changing software requirements, it is unclear when and how the technical debt will have to be repaid.
It’s not just the unpredictable payback of technical debt that makes it undesirable. Just like in the mechanical world, duct taped systems are generally more fragile and complicated, than well-engineered systems. The addition of new features is more error-prone since it is harder for engineers to understand how each component of the system works together when duct-tape solutions fill the codebase. Additionally, communicating the need, and effort required for changes to the system to support (even basic) new features to a less-technical audience, becomes almost impossible
One of my favourite videos about software engineering called Microservices, underscores this point well. The video is trying to make a point about poorly designed microservices, and does a good job of describing what it is like to add features to a system with a high level of technical debt:
Product Manager: Why is it so hard to display the birthday (of the user) on the settings page? Why can’t we get this done this quarter?
Software Engineer: Look…I’m sorry, we’ve been over this…it’s the design of our backend. First we have something called the BINGO service…we get the user’s ID out of there. Then from BINGO we can call PAPAYA and MBS, to get that user id and turn it into a user session token. We can validate those with LMNOP, and once we have that, we can finally pull the user’s info down from RACCOON…
(rant continues)
We’re blocked ok…You sad little product manager. You think you know what our users want? You know nothing of my pain.
It’s not all bad though
Technical debt, like duct tape, isn’t all bad though. The below image of aircraft ground crew performing pre-flight repairs to a plane with duct tape (or likely Speed Tape) was shared widely across the internet in 2015 in disbelief. If sub-optimal, short-term solutions are acceptable in aviation, where safety is paramount, then it’s likely we have use for them in software engineering too.
For many years, Facebook infamously had the internal motto “Move fast and break things”, to encourage a culture of rapid innovation. Engineers at Facebook incurred technical debt in order to build new features and encourage growth; and the rapid growth of Facebook in its early days cannot be denied. This doesn’t mean that Facebook didn’t have to pay back this debt eventually though. A post on the Facebook engineering blog describes the process well:
“Facebook.com launched in 2004 as a simple, server-rendered PHP website. Over time, we’ve added layer upon layer of new technology to deliver more interactive features. Each of these new features and technologies incrementally slowed the site down and made it harder to maintain. This made it harder to introduce new experiences. Features like dark mode and saving your place in News Feed had no straightforward technical implementation. We needed to take a step back to rethink our architecture.”
It’s certainly possible that if Facebook had decided to “move slow and fix things” in the early days, with limited developer resources, that they may not have exploded in the same way that they did; maybe a competitor would have emerged, or Myspace would’ve pivoted? For startups, a duct-tape solution can be critical to proving a market, getting product-market fit, or growing quickly enough to stay afloat.
The decision of how much technical debt to take on, like financial debt, comes down to the amount of risk that the business is comfortable to bear; a short-term gain, for an unpredictable future productivity loss. Famously, the founders of Airbnb funded the company with a baseball card book full of credit cards, an incredible risk no doubt, but one that paid off for them. Brian Chesky describes the experience in his lecture at Stanford:
So we ended up funding the company with credit cards and you know those binders kids in school put baseball cards in? We had to put them in those, we had to put them somewhere. That’s how many credit cards we had and we were completely in debt.
Putting aside business concerns, there are purely technical reasons for taking on tech debt. It’s important to remember that writing software is a process of discovery, as much as it is a process of invention. In general, the exact requirements of a piece of software are never truly fixed, and the challenges that will need to be overcome are often unknown. Creating systems with a lot of duct tape can mean getting the software in the hands of a real user or client in order to discover the “true” requirements of the software. Additionally, it can mean hitting nasty technical problems caused by bugs or missing features in external dependencies sooner rather than later in the project. At the very least it can help risky software endeavors “fail fast”, before they run wildly over-budget. On top of that, re-designing to eliminate technical debt every step of the way can mean jumping back and forward between broad architectural changes; sometimes letting the duct tape accumulate a bit gives a clearer picture of how the system needs to evolve.
What this all means
Technical debt is real, it’s unpredictable, and it’s what makes changes that should be easy, incredibly difficult. What really sets it apart from financial debt is the lack of visibility since it takes skill and willingness to identify. Additionally, the need to remove technical debt can be difficult to communicate to less-technical audiences. It’s for these reasons that technical debt has such a bad reputation in the world of software engineering. Effectively managed, however, duct tape solutions allow teams to rapidly discover requirements, test ideas, uncover challenges, fail fast and grow.
Managing technical debt requires understanding that it is unavoidable and unpredictable. Emphasis should be on understanding how and why it is being created:
- A lack of resources?
- Poor planning?
- Unrealistic expectations on the team?
In doing this, the team can understand the level of risk that they are taking on; the acceptable amount of duct tape to safely get the plane to the next destination.
Tom Clancy is a Software Engineer at Spatial Partners
At Spatial Partners we discover and deliver full solutions for any complicated data challenge. We are your problem-solving partner, and we thrive on complexity.
