Climbing Out of Technical Debt

© 2002 Johanna Rothman, www.jrothman.com

Have you ever had a conversation like this one?

Vice President: In the last release, you were able to bring the release date by over a month by cutting the testing. Do that again, ok?

Project Manager: Sorry, boss, we can�t do that. For the past three releases we�ve shortchanged the design work and the testing. We can�t cut any time from this project. In fact, it�s time to re-architect the system, redesign it, and do the testing we�ve put off.

Vice President: NO WAY!

Project Manager: Hey, that�s just the way it is. We haven�t taken the time to finish this product yet, and our shortcuts have come home to roost. Our technical debt is too high to work this release the way we�ve worked the others.

Technical debt, as defined by my colleague Dave Smith, is the debt a company �owes� to a product they persisted in shipping in an incomplete or unstable condition. This is a situation referred to in Hunt and Thomas�s book The Pragmatic Programmer as �software rot� or by Gerald M. Weinberg in Quality Software Management, Volume 4 as �design maintenance debt.� The software isn�t necessarily rotten�the product is incomplete.

As the technical debt increases, the load on the customer support staff becomes overwhelming, and the developers have trouble adding or changing system features. When developers are stuck on new development and support is overloaded, you face difficult choices about what to do now: have developers support the current product; ignore the current product and continue with development; some combination of support and development; or stop developing and fix what you�ve already got.

Project managers, development managers and test managers often can see this coming. They know there is a window of opportunity to fix the product before the technical debt overwhelms the company�s ability to do new product development.

Look for these signs of technical debt:

• You ship a product and then have to put out a point release before it�s even left the CD duplication house.
• Testing time increases disproportionately to the development time.
• The developers tell you that part of the product needs to be re-architected, because the current architecture doesn�t support the current requirements, never mind the additional requirements.
• Developers refuse to touch a part of the product saying, �No one but Fred can touch that. I know Fred left three years ago, but no one else can work on it, because we don�t understand it.�
• Developers spend more time supporting current customers by solving customer problems, fixing defects, and answering customer questions, than they spend developing new product.
• You hire more testers than you have developers, and you�re still not sure you�ve tested the product.
• Your developers threaten to leave because all they do is �maintenance�.
• You ship the product not because it�s ready, but because the developers� families have abducted them to go on vacation, or the developers are too exhausted to continue the crunch mode you�ve been in.
• You stop testing because you don’t want to find more defects.
• Your cost to fix a defect continues to increase, from release to release. (If you�d like to see a picture of this dynamic, check out Weinberg�s Quality Software Management, Volume 4, reference below.

Recognize your technical debt, and start planning what to do. Before you can plan, you need to know a little about how you came to have this technical debt problem. These questions may help you understand how your product acquired its technical debt:

How do you staff projects? Do you assign enough people at the time they are needed, or are some positions never assigned or assigned late to a project? If you never assign enough testers, or you don�t have an architect, you will incur technical debt. Don�t blindly accept people starting on your projects late, because you can�t make up time in a project. Have people start when they�re supposed to start, or recalculate the project schedule.

How do you design your projects? Do you first architect the entire product for its entire lifetime? Do you iterate on the design for several projects? If you don�t architect the product, and update the architecture as necessary, or if you don�t do a high level design before starting to code, you will not have a congruous product. One way to recognize an incongruous product is when you have multiple ways of doing the same thing. At one organization, the developers couldn�t agree on how to open and save data files. Since they couldn�t agree, every time a developer wanted to open or save their data file, they had their own open and save functions. When they moved from a command line interface (CLI) to a GUI interface, they had no easy way to integrate the open and save functionality.

How do you test the product? Do you have a way to start testing the product at the beginning of the project? Do you only have manual tests through the graphical user interface? If you start limited testing late, you�re guaranteeing yourself technical debt. You have many opportunities to test early and often: requirements inspections, design reviews, code inspection or peer review, unit tests, daily builds and smoke tests (tests to verify the daily build works at least a little bit), peer review of fixes in addition to system test. Decide what makes sense for your environment, and add test capability as early in the project as possible.

How do you organize your projects? Do you have functional managers who hand off the product from the analysts to the developers to the testers to the manufacturing folks? If so, no one has the same view of what the product should be. At a minimum, assign an overall project manager or program manager, so someone is pulling the entire project together. Projects without project managers are highly risky, and in my experience, always fail. In addition, projects without a sponsor or champion have a difficult time finishing.

How do you know what done means? Do you have some objective criteria to know when the project is complete? If not, you�re more likely to release an incomplete product. Develop release criteria for every project, so that everyone on the project, and the project sponsor understands what your company will receive for its project money.

How do you run your projects? Do you stick to only one type of lifecycle and one process? You may need to choose other lifecycles and/or processes to avoid future technical debt. Many project managers are comfortable with a particular lifecycle and/or process, and it can be difficult for those project managers to switch to a different lifecycle for a project. If you�re used to a waterfall lifecycle, you�ll have to learn to plan for an iterative or incremental lifecycle differently. And, you�ll have to monitor your projects differently.

If you�ve changed your lifecycle, you may need to change your process. What you do and when you do it will make the lifecycle successful and keep down technical debt. If you�ve never planned when to do inspections and reviews, start with something small, such as peer review of defect fixes during system test. I�ve found that a nightly build and smoke test helps find defects faster, and helps everyone realize when the product is not ready to ship.

Although it feels as if you�re moving faster when you take shortcuts on a project, in reality, your technical debt prevents you from making project progress. You might be able to shortchange your product on one or two releases, but at some point your product technical debt is like credit card debt � if you don�t pay it off in a planned way, the debt buries you.

Observe and measure your products, to know when you should act to reduce or avoid technical debt.

For more information, see

Hunt, Andrew and David Thomas, The Pragmatic Programmer. Addison-Wesley, Boston, MA 1999.

Weinberg, Gerald M., Quality Software Management, Volume 4. Dorset House, New York, 1998.