Welcome to the OnTechnicalDebt Expert Interview Series. This interview focuses on how Technical Debt can be categorized to help business organizations face the challenges of taking on and paying down Technical Debt in their own software projects, particularly on how it can be used to communicate to non-technical stakeholders.
This month we interviewed Steve McConnell – CEO and Chief Software Engineer at Construx Software, and famous author of many software development books. Steve is a thought leader in Technical Debt and has been pushing the metaphor to the next level since his blog post in 2007 on categorizing and managing technical debt.
He is interviewed by Dave West – CPO at Tasktop, an ALM platform provider, formerly a Forrester Researcher and co-author of the ForresterWave: agile development management tools. In this interview Dave questions how we can categorize technical debt in a pragmatic and simple way that can help business technology professionals use the technical debt metaphor in an agile environment
1. Technical Debt: the Business vs. Technology
Dave West: Ward Cunningham in 1992 coined the term technical debt as a way to describe the consequences of poor software architecture and bad coding. What is your opinion on the way the term has been picked up by the industry? How useful do you think this term is for technology professionals?
Steve McConnell: I think it’s a tremendously useful term. The notion of technical debt has probably been around since a few months after the first piece of software was ever written. But I think that both technical staff and business staff have not had an easy way to talk about the problem. The technical debt concept raises awareness of a cluster of issues that are often buried, intangible, or hard to talk about. The idea makes the technical decision making process much easier, building a bridge between technical staff and business staff.
Dave West: By providing a common language around an area that is often misunderstood by business and technology people, does the technical debt metaphor help to create a balance between technical decisions and business decisions?
Steve McConnell: I don’t think that I would call it a balance. It’s more a matter of integrating business considerations into technical decision making and vice-versa. At the end of the day, all decisions made in this context are business decisions. However, businesses have been flying blind for a long time when it comes to technical debt. The metaphor therefore helps the business and technical staff have a concrete and open conversation about the technical path to follow that will make the most sense for the business.
Dave West: Businesses tend to focus on short-term decisions; however the ‘debt’ metaphor implies long-term perspective. Is the technical debt metaphor about understanding the implications between the short-term business decisions versus the long-term ones?
Steve McConnell: The distinction between short-term and long-term is an extremely important distinction to make in this context. In technical debt, the short-term connotation is often seen as “bad” but I do not buy into that. I think people often see it this way because decisions are being made without full understanding the consequences. Technical staff should build on the technical debt metaphor as a way to talk to business staff to explain that if they take on short-term technical debt, they will need to pay it off or else it will end up costing the business on the long-term. For example: “If we spend X weeks working on this particular infrastructure area, it will allow us to add features A, B, and C. Although the work itself does not show immediate benefit, it will open the door for other work that will produce business benefits later on.” Now this becomes a productive discussion and we have a reason for the business to engage.
2. Strategically Managing Technical Debt
Dave West: Often software business organizations perceive technical debt to be a negative value and try to avoid it at all costs. Is this the correct approach to have? Should we remove all technical debt and avoid having a continual overdraft?
Steve McConnell: I think that even though people might push the technical debt metaphor too far it is still a notion that can be extended quite a long way. Businesses have different philosophies as to how much debt they can safely take on. Some businesses adopt a very purist approach where they say we are never going to take on any debt. Others might take less of a hard line on that and believe that debt is a useful tool as long as it is taken on safely and is serving a valid business purpose.
In my experience, this position is quite polarized. Technical staff prior to being exposed to the ins and outs of technical debt often have the attitude that the good amount of technical debt is zero. That is largely because they do not understand the business benefits of what it means to take on debt. At the same time, business staff think we can load up technical debt because they never truly see the consequences. But those consequences are there… they are just never expressed in a way that the business staff can engage with.
Dave West: There are several opinions out there about categorizing technical debt. For example, in 2007 you put together a model where you categorized unintentional vs. intentional technical debt. And Martin Fowler in 2010 made the distinction between deliberate vs. reckless in the technical debt quadrant. How has your thinking on categorizing technical debt evolved since then? Is there a “standard” set of categories that most companies should use in managing technical debt?
Steve McConnell: I think that each of the models that you mentioned have good aspects to them. But I am not sure that we have gotten to the point where we have the perfect taxonomy of technical debt. There are 2 points in a project where we want to use the technical debt categorization:
When making a decision to take on technical debt: at this point we would need to rely on a taxonomy that gives guidance as to what is good debt and what is bad debt
When we’re making a decision about whether to pay down the technical debt: at this point we would rely on a taxonomy that gives guidance as to what is the good reason and basis to pay the debt down.
I think that one of the ways my understanding of this problem has evolved since my blog article in 2007 is the deeper understanding of all the different categories of unintentional debt. A lot of the time we want to make software development decisions as consciously as possible but there are all kinds of moments where the debt is just handed to us.
E.g. A company invests heavily on V1 of a framework; however V2 turns out to be a lot better. It wasn’t a bad decision at the time but the landscape changed under them, so they end up with debt that they took on unintentionally
E.g. A company that acquires another company but realizes that their codebase is not perfect, and end up acquiring a significant amount of technical debt
E.g. You start initial work in new technology area where you are being conscious about the decisions you are making but still climbing the learning curve. A year after the project, you realize the work you did was not great even though it was not intentional.
If we have a look at the intentional side of technical debt: I think that this is where project teams get into trouble… If the intention is to be sloppy (we don’t have time to dot every “i” and cross every “t” on this release, let’s just get it out the door!) then this is a recipe for accumulating technical debt that is virtually impossible to correct afterwards. I think we need to be taking on technical debt in a deliberate way, were we can identify specific pieces of work that constitute technical debt.
3. Visualizing Technical Debt
Dave West: One of the biggest challenges that IT organizations have when it comes to technical debt is visualizing it. Are there any best practices that can help organizations visualize and track the debt load that they have already incurred? And what is the measure you present when talking about technical debt (dollars, time, days)?
Steve McConnell: We’ve seen it be done in a few different ways. When talking about intentional debt, it is pretty easy to come up with a measurement: if we decide to take on debt, it is because we assume that the quick and easy shortcut is faster than the proper way of doing things. This assumption cannot come up unless there is some sort of measurement or estimate for how long it will take to do things both ways. I’ve seen a few cases where companies convert technical debt to dollars, but more often I see them track it as a number of bugs and defects. Companies do this in several different ways:
Putting an estimate into their defect tracking system to track how long the debt is in the system. When the debt ages more than 6 months, it is elevated to severity 1 and should be removed as soon as possible
Looking at their software development budget and understanding what percentage of the budget goes into new development and functionality, and what percentage goes into keeping the system going (which is classified as technical debt)
Tracking velocity in a Scrum environment: when the velocity starts to drop, they will attribute that as too much debt which is causing a drag on the project. They can therefore devote an iteration to reduce technical debt with the expectation of their velocity to increase.
Ultimately, the maintainability of the software over time is an important factor, whether that is in the area of Scrum (velocity) or at the enterprise level (change requests for business).
Dave West: Who are the stakeholders in a typical IT organization that need to be concerned with technical debt? Is this a universal measure, or should it only be aimed at the CIO and VP of engineering?
Steve McConnell: I think that technical debt as a notional concept is useful across the board. I have had the experience of software companies taking the metaphor too far, saying for example that they would like to track their technical debt on their balance sheet as a numeric value. The technical debt concept is a house of cards: the numbers we are using to represent technical debt are only estimates on how much path A would take versus path B. Some organizations are good at calculating this estimate, but others are fair, at best. Looking at the foundation of this house of cards and understanding what the technical debt notion means, I think that it is a helpful concept to start a discussion rather than giving out specific numbers and putting them into a spreadsheet.
4. Technical Debt Lifecycle
Dave West: At the end of the day, the end goal is to be able to work on reducing your technical debt in order to focus on the overall health and software quality of your systems. Would you agree with this?
Steve McConnell: This is a valid approach. Keep in mind that a lot of the decision-making on whether you will or will not pay down your technical debt will depend on the interest payment of that debt. For example, I am currently in the process of financing my house right now, with an interest rate of 3,5%. At this rate, I don’t think I care if I ever pay back my interest. If the rate was at 12%, then I would care more as the interest would be onerous. I think the same analogy applies for software technical debt – if the penalty you are incurring on the project is small, then it might not make sense to pay back the debt; if on the other hand you are doing additional work because you haven’t paid off that debt, then that is a good reason to remove your technical debt.
Dave West: Looking at the overall life of a software system, most organizations do not have an end of life for their applications: when it gets too horrible, they just “put the system down”. Is technical debt a characteristic of the overall life of the software?
Steve McConnell: I certainly see cases where the technical quality degrades so much over time that the system has to be put down. More commonly however, a system is originally developed for a certain design envelope and as the years go by there are new uses that weren’t anticipated in the original design. I have honestly been impressed to see how companies have been able to extend their system over the original design envelope. But at some point over the years, you will realize that these systems are just not useful anymore. And we have to put the system down. This brings me to an interesting point when comparing technical debt with financial debt: when you retire the system, you retire all the debt with it. But when you retire financial debt, there are always strings attached… Therefore using the term “bankruptcy” to describe the retirement of a software system is not the best financial analogy. In my opinion retiring a system is more like winning the lottery!
5. Technical Debt Key Point
Dave West: Are there any final words of wisdom you would like to give for those that are tackling technical debt on a daily basis?
Steve McConnell: I have been working with this concept for the last 20 years now. But it has been a lot more active over the past 5 years. Today as I understand it, the concept is 1/3 helpful for technical decision making and 2/3 helpful as a communication tool between technical staff and business staff. I would encourage people to try it, we have had good results.
Erik Oltmans, an Associate Partner from EY, Netherlands, spoke at the Software Intelligence Forum on how the consulting behemoth uses Software Intelligence in its Transaction Advisory services.
Erik describes the changing landscape of M & A. Besides the financial and commercial aspects, PE firms now equally value technical assessments, especially for targets with significant software assets. He goes on to detail how CAST Highlight makes these assessments possible with limited access to the targetâ€™s systems, customized quality metrics, and liability implications of open source components - all three that are critical for an M&A due diligence.