Guest post by Natalie Warnert
By definition, debt is incurred when we get something upfront and pay for it later. My father used to have an anecdote where he’d say, “I’ll gladly pay you Tuesday for a hamburger today” (and how I wish you could hear the inflection he would say it in). When we want something fast and now but don’t want to pay now, that cost sneaks up on us later with, you guessed it, interest!
Software product development is no different. We incur many types of debt when we decide to pay later in the interest of going quickly (pun intended). Technical debt is acquired by making trade-off decisions to release products or features faster with the mindset of “we’ll make it better later” and “we just need to get it out there now to compete.” The idea is to make money now and we will pay those debts later with our fancy new feature profits. But we forget about the law of compound interest — or any interest at all for that matter and it sneaks up on us as an unrecognizable monster, much larger than we remembered setting aside before.
The interest on our technical debt increases from many different areas. There is the time wasted by trying to go back and fix things we should have done right in the first place but are no longer closely acquainted with (i.e. switching context). There is the time spent refactoring code to make it simpler to support and simpler to add on to and scale as our customer base, load, and performance needs grow. There is the time spend writing automated tests for existing features and new ones as they’re developed to ensure they work and don’t stop anything else from working in the process. There is the money spent on infrastructure that is aging to either support it when it’s no longer under warranty from the manufacturer or to buy new emergency infrastructure quickly and working overtime to get it set up and configured correctly. The above examples have different levels of predictability and demand different levels of regular attention. In fact, most are not preventable, only maintainable.
Most important to consider with technical debt is the cost of delay and opportunity cost of doing technical debt work when we could be doing other valuable things – if we would have paid more of it up front, it would have been cheaper but now we’re stuck with a much higher premium because of we delayed it. Supplementing the former is the hard to quantify goodwill lost when our systems aren’t performing for our customers and the somewhat easier to quantify lost revenue of when we lose a customer or can’t sign one in the first place. This money lost is just one metric we can point to when challenged about return on investment from paying down our technical debt over new feature work.
An example of cost of delay and compounded technical debt interest was I was working with a large cloud services company and I saw the results first-hand in a spectacular way. They were the epitome of a feature factory: always working on new cool things to disrupt the market at the expense of maintainability, quality of their platforms, and the detriment of their customers and their business’ bottom line.
It seemed like business was booming. Sales people were signing contracts left and right and teams were always busy trying to deliver on what had been promised from them (albeit without their input). This went on and on as the factory kept churning out new features, but underneath the infrastructure was creaking and the assembly line’s velocity (aka the development teams) was slowing down.
Every time something was touched it seemed, two other things broke. It reminds me of a tweet I once read that went something like: “99 little bugs in the code, 99 little bugs. Take one down, patch it around, 128 little bugs in the code.” One of the code bases was actually referred to as “the hairball” for obvious reasons. It wasn’t just velocity that was starting to slip…those contracts that were signed and arbitrary dates for new custom features were slipping too and accounts were going into the red at an alarming rate.
“We need to develop faster!” said Product and Sales. “We need to slow down to speed up,” said Engineering. They were risking losing deals, getting sued for breach of contract delivery, and most importantly further harming their reputation. The small customers who had made the company successful were leaving in droves and now all they could do was re-prioritize to save the large customers. Finally, everything came to a grinding halt after multiple system failures on release day when our technical debt plus interest was sent to collections. It turns out they hadn’t even been making the minimum payments and were overdue in a horrible way and wow did the collectors come calling.
The CEO stepped in and stopped everything. She took the content authority to sign contracts and deals away from everyone but herself. She reviewed and reprioritized every outstanding contract to decide what was important to deliver on and what was not. Most of it was deferred. The remaining teams (most of the company) were tasked to work on technical debt. All of the things that would enable eventual new feature work were suddenly resurrected from the depths of a Jackson Hole sized backlog across the company.
The edict stated that for an entire quarter nothing but technical debt, re-building out the environments, platforms, and refactoring the codebase was the priority. All new work was to wait until further notice. It was so bad, the “feature freeze” ended up spanning half a year instead. While some customers didn’t understand and left their contracts, some did allow the company to eat their humble pie before serving up the promised new features (now a bit cold) later than initially agreed on.
I wish I could say that this was an anomaly, but the same company had done something similar only about a year and a half earlier. While customers love new features and don’t notice if you pay down the interest on your technical debt, they do notice if you don’t. Make payments regularly on that interest and make sure they are more than the just the bare minimum (15-25 percent of your work each iteration is a suggested technical debt type work ratio). While this will never get you to balance zero, it will stop your company from screeching to a grinding halt as the train they’re driving on runs out of track and products crash into nothing in front of your customers.