How 19 Software Engineering Teams Deal With Technical Debt
The concept of being in debt of any kind is typically not attractive, so the term “technical debt” can sound intimidating. But in the same vein that monetary debt can be beneficial to building credit, a company’s technical debt can signal growth. And when tech debt is accrued and “paid back” intelligently, it can lead to some major wins for companies.
But what does technical debt look like from company to company? And how do different businesses leverage their debt to evolve? We asked engineers at some of the nation's top tech companies to share their stories.
What is Technical Debt?
WHAT IS TECHNICAL DEBT AND HOW DOES YOUR TEAM DEFINE IT?
CARLOS ASMAT, LEAD ROBOTICS ENGINEER
Piaggio Fast Forward’s Lead Robotics Engineer Carlos Asmat said technical debt shouldn’t come as a surprise to anyone at a tech company. Accounting for this debt “is more of an art than a science,” he said, and is a necessary part of good business strategy.
Any time corners are cut, a team acquires technical debt. On the software side, it might mean hardcoding a parameter that should be more readily configurable. Inevitably, someone will need to come back and redo the entirety of the work that was partially done. This saves time on a fraction of the task now, at the expense of needing to redo the entire task in the future. Just as with real debt, it is more expensive in the long term than working with the available budget.
MICHAEL KESSLEMAN, VP OF PRODUCT MANAGEMENT
Forward Financing’s VP Product Management Michael Kessleman, Senior Product Manager Ali Fawaz, and Software Engineer Team Lead Kelvin Ma provided insight into the merits of having some strategic technical debt at their fintech company. They said that the business has dedicated action plans on estimating, planning out and paying back technical debt built into its infrastructure.
We define technical debt as the expected cost to rework existing features or components of a system to better support the longer-term needs of the organization. Technical debt is incurred when the business is facing either an opportunity or an immediate problem such as an unanticipated edge case that is disrupting the current system.
Sometimes, we build a feature fast, fully knowing that it may need to be replaced later. When features are built quickly to meet an immediate business need, we plan to incur some short-term technical debt. In this instance, a quick fix relieves pressure on the current business environment while giving the development team the time to more strategically “pay down” the technical debt later. The cost of the quick fix can be easily weighed against the benefit that the feature delivers to the business.
As we continue to grow, we are willing to take on technical debt in order to meet the changing landscape and needs of the business. When technical debt is incurred as a result of exploring a new opportunity, the strategy is to minimize the investment until the value of the feature is proven, ultimately avoiding the cost of over-engineering. In this respect, technical debt is a great counterbalance to over-engineering, particularly when the additional cost is high and the expected benefit is still unknown.
In both instances technical debt allows us to meet the needs of the business while making appropriate plans to go back and review the feature to make sure that it is built soundly, to scale and that it will perform optimally.
TOORAJ HELMI, ENGINEERING LEAD
Fast-moving startups like Builder, formerly known as Engineer.ai, sometimes have tough choices to make when it comes to technical debt. Do you continue to move quickly, even if that sometimes means taking on more tech debt? Or do you slow things down and give your team time to iron out issues that could create problems down the road? It’s not an easy call, to say the least. Here’s how Builder Engineering Lead Tooraj Helmi and his team approach it.
Technical debt refers to the prioritization of short-term gains despite the longer-term troubles that may arise. It’s the understood tradeoff that results when our team decides to experiment and gather knowledge in support of immediate agility rather than plan and build with future efficiency top of mind. As a startup with global offices and engineering teams working together across the world, we are naturally prone to technical debt.
Just recently, my team here in LA had a dependency on a product being designed and developed by our team in India. Their part of the project was going to take three months, so we had two choices: put things on hold and wait for them to complete it, or create a simplified version of the product to fill the gap and move things forward for the time being. After a cost and benefit analysis, we chose the second approach, prioritizing short-term progress even though we knew that the placeholder product would be disposed of in a few months.
JOSHUA RHOADES, VP OF ENGINEERING
Technical debt isn’t something that can be effectively dealt with on a one-off or as-needed basis. Truly getting a hold of it requires companies to have processes and language in place that ensures everyone is on the same page and knows what their responsibilities are. BlackLine VP of Engineering Joshua Rhoades shared the concrete steps his company is taking to both address existing tech debt and reduce the amount they accrue going forward.
Over the last two years, we’ve put a drastically heavier focus on technical debt and how it impacts us in all its forms. From our perspective, technical debt is anything that forces hard limitations onto the system or our users, degrades delivery efficiency, or artifacts that adversely impact the ability to deliver on business needs in an efficient and high-quality fashion.
CHRIS DOWNS, LEAD PLATFORM ENGINEER
Mythical Games might be a young startup — it emerged from stealth in late 2018 — but the video game technology company is filled with a host of industry veterans, so it’s no wonder the team has a mature take on technical debt. Rather than trying to avoid it at all costs, the company acknowledges and accepts that tech debt will occur. Instead of asking engineers to hide the debt they’ve accrued, the video game technology company encourages them to ticket and track it.
Lead Platform Engineer Chris Downs shared more about Mythical Games’ proactive approach to tackling technical debt.
At Mythical, technical debt is defined as pieces of a feature or system that are moved to the backlog to be finished later after the core functionality is delivered. While these pieces may be necessary for ease of future refactoring or stability, the deliverable will still work without them. An example of technical debt is documentation. Well-documented systems make onboarding new engineers and clients much easier, but it’s easy to push that into the backlog and hope you find time to do it later.
JACKIE BENOWITZ, SENIOR SOFTWARE ENGINEER
It should surprise no one that when describing technical debt, Jackie Benowitz, Squarespace senior software engineer, relies on a well-known theory of statistical mechanics/thermodynamics. After all, Squarespace’s mission is turning complicated back-end systems into simple, elegant visuals.
We spoke with Benowitz about why, like entropy, technical debt increases over time and what their team does about it.
Technical debt is the cost incurred when you prioritize factors like speed to market over building perfect solutions. Technical debt occurs similar to entropy. It accumulates as product decision changes and/or evolves. These costs include but aren’t limited to tangled code, unhandled edge cases and error states, and increased time maintaining and onboarding new developers to the system.
For example, we needed to prioritize iterating quickly as we wrap up a yet-to-be-released product. As a result, we decided to migrate off GraphQL for our API contract and use REST instead. Since deciding on GraphQL, our product has changed drastically. The aggravation it caused us did not outweigh the onboarding costs, iterating costs, and lack of out of the box observability REST APIs have here at Squarespace.
BRAD MOORE, VP OF ENGINEERING
Not having technical debt doesn’t mean you’re beating the system. Rather, it means you’re potentially not producing code quickly enough or iterating as necessary. Brad Moore, vice president of engineering at Quartet Health, gave us a few examples of situations where technical debt might surface as part of the overall process.
We think of technical debt as the backlog of work created whenever we do not achieve a desired quality standard on our engineering delivery. It may be the result of a deliberate compromise or an unforeseen side effect of a decision. As it accrues, technical debt results in higher costs of maintenance and enhancement.
An example includes code that might exceed a desired level of complexity (meaning the code performs exactly as specified, passing all tests), but is expensive and risky to maintain over the lifecycle of the product. Another example is code has a dependency on a third-party library version that is currently end of life. In this case, the code performs as expected but the third-party library must be upgraded to ensure ongoing support.
JYOTI MOKAL, VP OF CONSUMER ENGAGEMENT
When it comes to technical debt, Zipari understands the importance of making the right kind of compromises. Jyoti Mokal, VP of consumer engagement products at the technology company, told us about how they believe openly talking about critical code requirements improves the process from beginning to end.
"It’s the reality of coding. Whenever you code a product, you are faced with constraints, including time, resources, and what’s known and predicted about user interaction."
Technical debt describes the costs of future work created by making coding choices to deliver working product in the short term. It’s the reality of coding. Whenever you code a product, you are faced with constraints, including time, resources, and what’s known and predicted about user interaction. For example, when there is a time constraint, engineers must make compromises, choosing to write simpler pieces of unoptimized code. This, in turn, produces technical debt; time and resources that will be owed following the initial production completion.
UDI PLADOTT, SOFTWARE ENGINEER
Engineers like Udi Pladott know that when it comes to software programming, updates and installation, if you’re not ahead you’re behind. But when foundational data programs mature in the equivalent of dog years (ie. quickly!), it can be impossible to maintain compatibility. Pladott explained how online invitation company Paperless Post handles tech debt and optimizes for the future.
Technical debt is any development work that is in our backlog as a result of either conscious, calculated shortcuts we took while developing and delivering more urgent product functionality, or as a result of engineering oversights that were identified after the code was deployed to production. For example, we built our product on a particular version of an open-source or third-party library (e.g. Ruby on Rails 4.x) and then continued developing our code without devoting the necessary time and resources to keeping our code compatible with version updates of that external library. As a result, at some point, our product will be running on technological foundations that may be several years old.
ANTHONY MENDONCA, VP OF ENGINEERING
People thinking about undergoing various cosmetic treatments and surgical procedures turn to RealSelf’s all-in-one information resource and marketplace. The company’s online forums offer feedback and reviews for treatments and more than 20,000 vetted doctors and healthcare professionals who supply them. With that information, users can book their treatments through RealSelf’s portal with more confidence.
VP of Engineering Anthony Mendonca acknowledges that tech debt is a fact of life in any engineering organization, and its prudent allowance can actually increase agility over time.
We recognize that every line of code our team writes has a half-life attached the minute it’s committed to our codebase. We define technical debt as anything introduced to our code base or processes that significantly reduces the agility of the team. This includes more than just lines of code, but also our deployment process or how the team triages and prioritizes bugs. It is a pretty broad definition, which intentionally leaves room for conversation about impact. Ultimately, technical debt for us is a useful analogy to have a discussion about short and long-term risk to the business.
ARTHUR KEPLER, SITE RELIABILITY ENGINEERING TEAM LEAD
Going through cancer treatment is hard enough without the interminable bureaucracies surrounding our healthcare systems. Navigating Cancer’s patient relationship management solution is designed to manage communication and data exchange between oncology doctors and patients, and the company recently raised $26 million in funding to build out its team and add new features.
Site Reliability Engineering Team Lead Arthur Kepler encourages his developers to own specific projects, which encourages them to monitor and proactively take care of technical debt.
Technical debt is a consequence of all software development, just as dirty dishes happen when making a meal. Practices to ensure periodic “clean-ups” and code reviews are essential to manage technical debt before it obstructs change and undermines morale. Technical debt can also exist in more than one variety, like defects internal to a codebase or deficiencies around maintaining a product within its ecosystem of services.
The internal variety may be due to rapid development of an MVP that needs refinements later on, or code produced without test coverage. There are times that rapid delivery outweighs maintainability, but often maintainability should be a primary concern with code that powers a business service — especially when considering factors like employee turnover.
Ecosystem defects are often related to the planning and architecture at the outset of a project. For example, at more than one company I have inherited a data ETL service home-brewed by former staff in a non-concurrent, dynamically typed language which did the job for small sets of data. When a new customer asked us to load 40 times more data, the ETL process would have taken 27 hours per day. This is an example of technical debt which had to be paid by changing underlying tools and languages, which is not a small ask to the business, despite the dividends it ultimately paid.
CHEN YANG, ENGINEERING MANAGER
Postmates connects millions of users with its network of couriers, who deliver goods and meals from stores and restaurants around the country. Postmates is also working on an autonomous delivery robot that can safely navigate city sidewalks. The company is headquartered in San Francisco — with a sizeable PNW office in Bellevue — and has raised more than $900 million since it was founded in 2011.
Engineering Manager Chen Yang said his team is always careful to document tech debt, and devotes at least 10 percent of engineering time clearing it.
In my team, technical debt is defined as something that we should do but didn't do. It can be planned or unplanned. For example, we used to have a few tests that need to be run manually for each production deployment. Automating these would require non-trivial effort — requiring a new test platform and so on — while running manually took five to 10 minutes. The same applies to services or code that are unused or has become obsolete over time.
LORENZO PASQUALIS, VP OF ENGINEERING
DreamBox Learning’s advanced edtech software starts from the assumption that math education isn’t a one-size-fits-all field — each individual student learns in his or her own way. The company’s software — currently in use by more than 3 million students across the North American continent — measures tens of thousands of data points for each student, and adapts in real-time to suit the needs of each child.
VP of Engineering Lorenzo Pasqualis said tech debt is inevitable, but his team’s goal is to add to it with “eyes wide open” in terms of the clearing work it will require later on.
For my team, technical debt is necessary work that has been put on the back burner in order to achieve some important goal expeditiously. Some examples might include known non-critical bugs; shortcut implementations that need to be revisited; messy code that needs refactoring; missing error conditions handling; missing automated/unit tests; simplistic logic; and unoptimized algorithms.
CHRIS CAVAGE, DIRECTOR OF ENGINEERING
Installing any kind of underground infrastructure — be it utilities, fiber optics, telecommunications, gas lines, or water pipes — usually involves massive disruption for the communities they serve. To limit the impact of installation and maintenance, Kent-based Digital Control has created one of the world’s most advanced devices — dubbed “Falcon” — for locating and directing underground horizontal directional drill heads.
Director of Engineering Chris Cavage outlined the multiple ways in which tech debt accrues, and his team’s collaborative approach to clearing it.
Technical debt can be anything from a bad design decision early in the process to a drastic change in product requirements late in the game, causing a need to refactor code without the allotted time by management to go in and fix it. You know it when you see it, and everyone on the team starts to refer to it as “that code.” An example of this was when we rolled out our last Falcon Locator, and the protocols to the microcontrollers weren’t plumbed in correctly because of a desire to save time by reusing older protocols — which ultimately cost us time in patchwork programming until we finally just rewrote the entire section.
ANDY PIRO, VP OF ENGINEERING
Ballard-based Pluto builds technology that allows geographically dispersed users to hang out in virtual reality. With an alpha now available on Steam, users can see digital renderings of one another’s faces and hands — the most crucial tools for in-person communication — and hear their voices. The startup has raised north of $13 million since it was founded in 2015, and is currently refining its product for a rollout to the masses.
VP of Engineering Andy Piro said the reduced speed of work caused by unforeseen tech debt is the easiest way to measure its long-term impact.
We usually identify it when a design, code sample or PR comes with the caveat, “this isn’t the ideal way to do this,” or “this is what we have now but we should go back in and fix it later.” Falling into the trap of thinking, “this is the quicker way to do this,” when really it’s just the quicker way to get your code committed at this moment — while knowing it’s slowing things down in the long run and requiring work in the future that will be harder and take longer.
We have a web application that multiple native applications use and communicate with using a generic IPC interface. A couple of years ago, a single platform-specific method was added to this interface “because it’s the easiest way to get this one feature in there right now for this one platform”. Fast forward to today, and the method is still in there and is confusing, hard to use and fragile to any platform other than the one it was specifically designed for, but we still have it in there because it’s not trivial to refactor out at this point.
BRIAN GAMBS, CHIEF TECHNOLOGY OFFICER
Having friends that “get” you is a wonderful thing. Bright Health’s engineering and product teams work together so closely that Chief Technology Officer Brian Gambs said “the trust is built into the relationship.” This allows the teams to openly discuss how to best tackle technical debt together in a way that benefits them both, and the business as a whole.
To us, technical debt means less-than-ideal choices in software design, development or coding. Technical debt can be good or bad. A team takes on good technical debt when it deliberately takes shortcuts, puts hacks in place, or uses quick-to-implement but non-scalable approaches in order to “fund” higher velocity work or put more focus on what is most important. A team takes on bad technical debt when it cuts corners without purposeful intent — things done by accident or not for a deliberate purpose.
The use of vendor systems is an example of where we have incurred technical debt over the last four years. While we know that we will need to replace many of those systems in order to achieve our ultimate goal, we never could have built all that technology from scratch in such a short time.
TRACEY BERG, PRESIDENT
TIM SMISER, DIRECTOR OF ENGINEERING
Technical debt at Cerity is handled through a detailed set of principles, according to company President Tracey Berg and Director of Engineering Tim Smiser. The leaders at the workers’ compensation solution provider describe a formulaic approach to compiling and disposing of technical debt that is surprisingly simple: solution architecture.
According to Berg, prioritization is a daily activity in any company. It is impossible to do everything we want at the time we want to do it; trade-offs are almost always necessary. Sometimes, time to market is more important than implementing the ideal technical solution. Other times, budget constraints may limit an organization from implementing the preferred approach. In these situations, a less-than-optimal solution is put into practice and technical debt is created. Historically, it was common to leave this sub-optimal solution in place. This results in long-term maintenance, flexibility or usability issues. If not managed, the amount of technical debt can grow so large that it seems insurmountable to address.
At Cerity, an example of technical debt is our portal login process. While most of the user experiences in our applications are custom developed, we leveraged the user interface provided with our authentication technology. It works and does what is needed from a security perspective, but the screen is not consistent with the look and feel of the rest of the application. We chose to leave it this way temporarily because our front-end development team had higher priority work to do. In doing so, we created technical debt. It was the correct priority choice at the time but it is important we prioritize fixing it at some point in the future.
LUKUS REINDL, LEAD SOFTWARE ENGINEER
Going to the dentist is scary enough without having to deal with payment plans and insurance costs. Lead Software Engineer Lukus Reindl shares how he works with technical debt so that OrthoFi’s users are always smiling.
At OrthoFi, we believe that the state of your architecture is akin to your physical health and fitness. That is, having a plan to account for tech debt would, in essence, be the equivalent to eating your veggies and exercising.
To us, technical debt can be defined and categorized through different reasons: scoping considerations as it relates to market delivery, the need to keep your tech stack current as new technologies emerge, and accounting for the scalability of your product as your business needs change and/or your client base grows over time.
No matter the reason, it is important to our team that we have plans to address tech debt so that we can support our growing business needs and, ultimately, delight our customers.
LEE GONZALES, VP OF ENGINEERING
As the saying goes: if it ain’t broke don’t fix it. When it comes to the residential mortgage industry, however, Maxwell knows there’s a lot to fix. Vice President of Engineering Lee Gonzales shares how he manages technical debt to keep the business growing.
Technical debt is any deficiency in a system that makes it harder to change, improve or maintain. Concretely, technical debt comes in at least three main forms.
Technical debt does not always start out as technical debt. Often, well-written code poorly maintained can turn into technical debt. Or code can turn into debt if it once served a purpose but is now no longer used because the business shifted or technical needs changed.
Technical debt can also take the form of a wise investment to achieve a business outcome, but much like taking out a loan to fix your car so you can still drive to work, it is best used only in a pinch or when there is an exogenous need, like an external deadline. Generally speaking, the lean thinkers have this right: go slow to go fast.
Technical debt can also be due to omission. Most often, this takes the form of not writing tests. What we usually see is that our test code coverage is too low. Or that our integration tests are faulty, flaky or incomplete. But like flossing, you just have to do it. Other forms of missing code can take the form of missing error handling, logging to enable observability, or metrics to properly understand system performance.
WHEN TECHNICAL DEBT DOES OCCUR, WHAT PROCESS DOES YOUR TEAM USE TO MEASURE AND MANAGE IT?
CARLOS ASMAT, PIAGGIO FAST FORWARD
Acquiring technical debt needs to be a conscious decision, as opposed to the result of someone getting lazy with a task. This is decided during planning sessions and the time for the work is accounted for in consequence. However, evaluating the consequences of debt, much like estimating the time a task takes, is more of an art than a science. It needs to be accounted for down the line in the overall schedule and not ignored, or else it creates more technical debt.
MICHAEL KESSLEMAN, FORWARD FINANCING
We consider technical debt as a sign of growth and a necessary tool in most software development. We’ve seen significant growth over the past few years, which requires us to take on technical debt and review some of our systems to be sure they can handle the requests, keep up with the velocity of business growth and continue to plan for future growth.
When the business requests a new feature or capability, our team will gather the requirements and estimate the effort. When the estimation is much higher than anticipated, it’s usually a clear sign of some outstanding debt on our technical “balance sheet.”
The reasons for reducing technical debt are usually to either increase the overall stability and flexibility of the existing system, or to improve the productivity of the development team. For our team, taking on technical debt is a responsible action plan which allows us to plan proactively.
However, technical debt can have some significant implications for any business if it is not effectively measured and managed. So, our development team goes through an assessment process and acts proactively to allocate time to “pay down” technical debt during sprint planning and backlog grooming.
TOORAJ HELMI, BUILDER
If there is an understood long-term disadvantage involved, we always conduct a cost and benefit analysis to see if the short-term benefits outweigh the costs. We consider the cost of reintegrating with the new system, the cost of retraining employees and customers, and the cost of transitioning from the sub-optimal product. If it makes sense to move forward regardless, we then make sure the decision to increase the debt is communicated to stakeholders who might be impacted by it.
JOSHUA RHOADES, BLACKLINE
One thing I have been really happy to see here are delivery teams calling out technical debt before it occurs, but that’s ideal and not always reality. The first-order problem is how to identify technical debt across the organization in a common language. Having a shared language has made us much more adept at recognizing technical debt. However, we all recognize that technical debt is part of delivering software. When we choose a particular piece of debt for a business goal, it’s important that we do so in a conscious manner, and that we address the debt in subsequent iterations.
We are also beginning to define types of technical debt: architectural, inadvertent versus induced, by originating source, aging and so on. Each type of debt will have assigned “owners.” Ownership is a vital concept to successfully tackling debt. Once categorized by type, these items will then go through a lightweight prioritization process based on impact, cost and value of improvement.
Our product teams are keenly aware of tech debt, with a dedicated 20 percent focus of sprint tickets targeted at technical debt. However, there is something that is often forgotten when it comes to technical debt: feature debt. What features are of low value or use, or are duplicative or cannibalizing other features? Is the UX causing inefficiencies to our users? Are we missing features? Feature debt and technical debt are often related or able to influence each other.
CHRIS DOWNS, MYTHICAL GAMES
We feel that it is important to accept that technical debt will ultimately occur. Instead of punishing engineers for creating technical debt and making them feel that they need to hide it, we promote transparency and try and ensure that any known debt is ticketed and tracked in Jira so that a project manager can make sure it is slotted in at a later date. When a milestone containing the integration of a bunch of features is approaching, we will often dedicate a sprint to polishing and technical debt repayment to try and keep the debt from rolling over to the next milestone.
JACKIE BENOWITZ, SQUARESPACE
Technical debt isn’t necessarily measured on our team, but rather prioritized. During retros and milestone whiteboarding sessions, we work together to raise the important pain points and weigh what needs to be addressed immediately. Often, these larger debt payments are added to our sprints and estimated exactly like a product task.
"Technical debt isn’t necessarily measured on our team, but rather prioritized."
Inspired by the campaigns team, we’ve dedicated one day per sprint to tackle tech debt any engineer wants to address. We continuously evaluate whether to address tech debt now or later. We constantly ask whether paying down some tech debt is worth pushing back our release date.
BRAD MOORE, QUARTET HEALTH
At Quartet, technical debt is managed in a backlog just as we manage product features. Most importantly, our engineers transparently communicate our technical debt backlog to product management, so that it is prioritized.
JYOTI MOKAL, ZIPARI
A finished product with no technical debt would be, by definition, perfect in every functionality, every use case. This is simply impossible. Technical debt occurs in every software development process. You can measure it by tracking what has been prioritized for completion and anticipated needs after initial production. At Zipari, we manage our tech debt by encouraging our engineers to talk about it in the open. As long as engineers meet our customers’ most critical requirements, they are empowered to build technical debt into products, record it, and keep moving toward deadlines. That helps ensure we make production timelines and enables us to schedule debt pay-back, so we can manage evolving product priorities.
UDI PLADOTT, PAPERLESS POST
Any technology company that has been around for several years will incur some technical debt. We are no exception. To deal with our own tech debt, every sprint, each team dedicates at least 10 percent of its time to working on issues that are not on the front line of its feature development and bug-fixing objectives. A lot of this is about technical debt. However, some tech-debt tasks may be too large or complicated to address in one sprint, so a team may take on some pieces of debt as long-term projects, breaking them down to sprint-size iterative steps.
ANTHONY MENDONCA, REALSELF
We understand that not all technical debt is bad and that it’s a natural bi-product of software development. We instead focus on understanding the impact to team agility — and how prudent debt can actually temporarily increase team agility if executed properly. A/B testing and early release prototyping help us get to market sooner, gain consumer insights and ensure we're solving the correct problem. Iterating along the way ensures we can fine tune the right solution, with less technical debt in the long run. However, if we ignore the accrual of reckless or inadvertent debt, we will encumber ourselves with maintenance and decrease flexibility of our codebase over time. Ultimately, we do our best to create a culture of continuous improvement and empower our teams to do what’s right while acknowledging that not all debt is created equal.
ARTHUR KEPLER, NAVIGATING CANCER
Technical debt is a challenging phenomenon to measure, as real-world software is complex by nature. As with financial debt, technical debt compounds over time. Additions of new business logic and handling special cases can exponentially add complexity to a codebase. The most immediate way to identify technical debt is to survey the developers working in a codebase. The approach to quantifying technical debt therefore should begin with developers continuously identifying technical debt into stories, labelled as “technical debt”. By tracking these stories at the rate they are “accrued” and “paid off”, they can give an indication of progress.
Another method to analyze “hotspots” of technical debt is examining stories and tasks that far exceed their estimate at the outset of a project. This should be an important part of retrospectives closing each sprint. Finally, a more detailed approach is to examine the version control history for changes that span numerous concerns and layers of a product — an indication of tight coupling and rigid complexity. Technical debt analysis tools such as Code Maat, Codescene, and SonarQube use similar methodologies.
Ownership of a product or service codebase is very important, as the owners will be empowered to guard against debt and decay of a code base, especially when many developers are contributing. Adequate planning, architecture, and code reviews are essential to this process, as well as ensuring test quality. If you’re at an organization with an SRE team, involve them in your code reviews and testing processes — they will thank you for it, as will customers for the improved reliability.
CHEN YANG, POSTMATES
The first step is to document it. When we add tech debt to our backlog, we also try to estimate impact. It can be the extra effort that will accumulate for the next six months. In the example I mentioned earlier, it will be multiplication of 10 minutes with the number of deployments in six months. The next step is to prioritize this together with everything else during sprint planning. We usually make sure there’s at least some tech debt work in each sprint.
LORENZO PASQUALIS, DREAMBOX LEARNING
Technical debt is part of life — the goal is not to have zero technical debt. Instead, the goal is to have a handle on what the debt is, and a plan to pay it down in one way or another. In my teams, we work using a scrum process, and we use Jira to manage backlogs. Technical debt work items are filed as Jira stories, and they get prioritized along with sprint work. Our technical leads are responsible for scheduling technical debt reduction work items in the sprints, and negotiating with the business product owners sprint-by-sprint.
CHRIS CAVAGE, DIGITAL CONTROL
If the technical debt is small enough that a developer can just knock it out during his or her day-to-day sprint tasks without schedule impact, we’ll usually just advocate for that — assuming regression tests are in place and as separate commits, of course. Otherwise, larger chunks of technical debt usually find their way to our Jira boards and get triaged during our typical backlog refinement meetings. Most team-members will have a sense of tech debt’s magnitude to properly assess the story or task and address it in the next sprint or so.
ANDY PIRO, PLUTO
We typically create backlog tasks to track things that should be refactored and they are uncosted using planning poker. We will take on refactors when working on code in the same area as the refactor and since they are uncosted but still obviously take time and effort, the overall velocity of the team will go down. In the case of unknown tech debt — that is, tech debt we didn’t explicitly know about or track — we will typically underestimate the amount of work for a given task so again you’ll see a dip in the team’s velocity. This velocity reduction is the clearest way to measure the impact of tech debt, but we stay vigilant in accepting lower velocity sprint-to-sprint if it means we are improving things by paying off some of that debt.
BRIAN GAMBS, BRIGHT HEALTH
Our approach to managing technical debt is based on a strong partnership between the engineering and product teams. Together, we work with the rest of the company to define our shared priorities. We are fortunate to have a working relationship with a product team that understands that protecting the quality of our software is important to the long-term velocity of the engineering team and thus to the success of our business. So, when we plan our roadmap, we can have honest conversations about the tradeoffs that occur when we take on technical debt. The trust built into the relationship, and a shared belief that the quality of the software matters and requires investment to maintain, makes it easier to manage technical debt and use it for leverage against velocity rather than only as a nagging burden.
TRACEY BERG & TIM SMISER, CERITY
Smiser: Technical debt is managed in the same way as other work — it’s put into our work management system and prioritized in backlog discussion. It’s easy to let technical debt operate outside of the work that is customer-driven. The result of this approach is incomplete technical debt resolutions and frequently late delivery on other commitments. Technical debt needs to be worked as a priority alongside other priorities. And just as any other work effort has a plan for delivery along with associated measures for delivering, the same approach needs to be applied for technical debt to ensure its completion.
LUKUS REINDL, ORTHOFI
Using the Spotify model, we logically divide our ownership of an architecture area. As part of the team’s accountability areas, there are defined engineering metrics that we use to monitor and measure tech debt.
The three metrics we focus on include user story cycle time, human time spent on release activities, and a ratio of new feature development vs. maintenance on the product. We balance throughput with measures of quality and stability, including metrics around test automation coverage and webpage performance. Finally, we continually measure infrastructure cost and usage projections regularly, given we are a cloud-hosted infrastructure (AWS).
As guidance to the teams, we believe that metrics should not be used for judgments, but are rather used as a means to identify where we should be making proactive investments.
LEE GONZALES, MAXWELL
Much like measuring productivity, it can be very difficult to properly measure technical debt writ large in your systems. In other instances, technical debt is signaled via more and more bugs emanating from an area of your systems. Or maybe like eggs that have gone bad, there is a particularly foul smell you start to notice in parts of your codebase — if that occurs, schedule time to have a senior engineer investigate and make a recommendation on what to do in order to prudently handle it. Since it’s hard to judge the positive or negative long-term effects of technical debt, pay it down. The best course of action is to slowly and incrementally pay it down based on the hot spots in the system and/or areas where frequent updates are required. However, always remember that infrequently changed code which looks suspicious, but never breaks, can be just fine.
Lastly, when measuring technical debt, you should be aware that there is a dark form of technical debt. Senior engineers whisper about it in hushed tones, tiger teams get formed around it, and businesses fail because of it. It is the worst and most mind-bending form of technical debt. A bit like dark matter or a coal fire underground, this is technical risk below the surface of clear observation. This form of technical debt arises from unforeseen dependencies, creeping system complexity, or the violation of deep system constraints due to basic, system-wide assumptions. Examples include assumptions like, “We will never have a million customers so an 8 KB customer cache in memory is fine.” This type of landmine usually remains invisible until you have a snafu. This is what you want to avoid.
WHAT PROACTIVE MEASURES DOES YOUR TEAM TAKE IN MINIMIZING TECHNICAL DEBT?
CARLOS ASMAT, PIAGGIO FAST FORWARD
When wanting to move fast and forward, acquiring technical debt is inevitable. An approach we take is to acquire debt in domains outside of our core competencies, like on tasks that will need to be outsourced. This uses the resulting work as a draft for the requirements of the tasks to be performed in the future. Ultimately, this allows us to be more impactful but invest time in the areas where the most value can be added.
MICHAEL KESSLEMAN, FORWARD FINANCING
We have multiple processes set in place that will help us to proactively minimize technical debt including sprint planning, proper grooming of feature work before development begins and test coverage. There are velocity assessments — if our team is losing velocity, this is a good indicator that tech debt may exist. We have automated checks for things like test coverage and code smells that help us to uncover any problems that may implicate a deeper issue or uncover any weaknesses in design. We have a tech council — a meeting that includes our longest-tenured engineers who have historical knowledge of system architecture as well as some subject matter experts that are most familiar with particular features. This meeting allows us to assess technical debt at a macro level and helps us to collectively plan large initiatives.
TOORAJ HELMI, BUILDER
We want to provide the best start-to-finish service to our customers, so our team plans for long-term optimization as much as we can. Rather than focusing on a client’s product in isolation, for example, we always try to look at the end-to-end production chain as a whole and consider what gaps may arise. If we spot something, we immediately gather the relevant stakeholders together, review our choices and come up with the optimal plan to minimize the technical debt.
In cases where technical debt can absolutely not be avoided, we build the short-term products with maximum reusable components. Using an architecture that allows for decoupling helps us easily eliminate the sub-optimal products as needed.
JOSHUA RHOADES, BLACKLINE
We’re big believers in code reviews, as well as leveraging automated static analysis on merges so that the machines focus on the common areas. This helps us get to qualitative, rather than quantitative, reviews. But real proactivity is preventing tech debt before it even gets into the codebase. We focus on reusability of components and functions, and working in properly encapsulated and separated classes and methods.
Reusability, as well as hygienic code, tends to force a thought process that avoids tech debt, since a reusable method that is also properly encapsulated and abstracted is much easier to refactor without downstream impacts. Likewise, our minimum of 80 percent unit test coverage helps immediately identify breaking changes or problems in that code, which in turn gives us greater confidence in being able to update it without worrying about unknown impacts.
We also make use of heavy linting and intellisense in our integrated development environments so that our engineers get as much helpful real-time feedback as possible, such as memory leaks or scope issues. This allows us to focus our attention and build scalable, performant, resilient and maintainable code that does not couple us to bad smells or bad design decisions.
CHRIS DOWNS, MYTHICAL GAMES
We try and figure out the root cause of the most common types of technical debt and provide tooling and processes to make them easier to avoid. One example is API documentation. We have made use of OpenAPI annotations in our Java APIs to add programmatic generation of documentation, allowing engineers to write documentation right alongside their code.
Another pain point has been code and deployment consistency across different services. We are making use of Maven archetypes to templatize the application portion of our Java services and Helm charts to templatize the deployment of these services to Kubernetes. These improvements vastly decrease the amount of boilerplate and copy and paste we need to add to a new service and this keeps things consistent across services to speed up onboarding.
JACKIE BENOWITZ, SQUARESPACE
Due to managing past tech debt, we’re now looking at building features in a new light. Before jumping into features and larger tech debt payments, we write out technical implementation plans as RFCs (review for comments). RFCs allow us to propose ideas clearly while getting ideas from reviewers. This helps expose possible debt we’re taking on, but historically track the decision of why we’re OK with that.
We’re constantly juggling what’s the right technical decision and what can we do right now to create the feature fast. These RFCs help answer questions we ask ourselves every day. What are the tradeoffs for doing the faster path? What small changes can we make that won’t keep us drowning in tech debt in the future? A lot of times, especially nearing the finish line of a project, taking on tech debt has a high reward, and that’s OK!
BRAD MOORE, QUARTET HEALTH
You can minimize unintentional technical debt by taking steps to prevent it from happening through good engineering practices such as reviews, static analysis, and training. Regardless of what prevention is in place, technical debt will still accumulate. It’s most important to continually work on paying down the debt to keep the code healthy. This requires continual negotiation between engineering and product management.
JYOTI MOKAL, ZIPARI
At Zipari, our engineers know that accruing this sort of debt is normal and necessary. The key is to track it and keep it under control. Our leadership understands that writing code against a deadline requires compromises. Documenting that debt shifts the perception that technical debt is a sign of sloppy coding and creates a shared understanding of immediate and longer-term costs and benefits to technical choices. This reduces the pressure for engineers to hide their debt and reduces the chance for bugs, broken products, and unhappy clients.
UDI PLADOTT, PAPERLESS POST
The best way to avoid tech debt is to write good code. No code makes its way to production without peer review. However, our goal isn’t to categorically avoid technical debt, but to be smart about how much of it we take on. If we take a shortcut so that we can get a product out and start learning about it faster, we’ll think about how quickly we’ll need to pay back that debt and how much harder it will be to do that work down the road. We try to think through complicated projects collectively in order to come up with the optimal solutions, balancing the speed of development with the long-term sustainability of the code.
ANTHONY MENDONCA, REALSELF
Our iterative development process focuses less on minimizing technical debt, and more on understanding the impact on balancing short and long-term results for our customers. In order to do this sustainably, we must balance time to market, value creation and the end-to-end quality of the consumer experience among other things. We’ve adopted a framework to help set expectations across product, design and engineering so we know what trade-offs we’re making on any given project.
When working on a problem we aren’t yet confident is valuable to our consumers, we rapidly iterate through a series of A/B tests. The longer we invest in building, the more we risk creating something for a problem that doesn’t even exist. In these cases the team is empowered to accept prudent technical debt to reach consumers as quickly as possible.
Finally, once we have high confidence in our technical solution, and have validated it with customer insights, we schedule the work to pay down any technical debt that will adversely affect agility over the long-term.
ARTHUR KEPLER, NAVIGATING CANCER
Service-oriented architecture is my top recommendation for enhancing maintainability of a software project. When done properly, SOA’s loose coupling of components provides the ability to empower change affecting the smallest necessary scope. Testing is the other lynchpin of empowering fearless change. Untested code is dangerous to change; there is no way to ensure regressions haven’t been introduced.
Finally, the best source of insight on technical debt are those who work closest to the codebase. Trust your developers, listen to their concerns and encourage blameless retrospectives for identifying proposals for maintenance projects. Discuss what debt was added and what has been paid off over the previous sprint — and keep track of it rather than letting it become a hidden trap later.
CHEN YANG, POSTMATES
We enable static analysis tools like SonarQube to run automatically for new commit and inspect existing code base from time to time, on top of all the test coverage tools. During our quarterly planning, we also set aside at least 10 percent of our time to work on tech debt items. Our backlog is one source of this work. We do routine house cleaning, like keeping our software version up to date, aggressively depreciating obsolete software or hardware and auditing all errors in the logs. Postmates has a very strong infrastructure team that promotes good practices and provides common tools across the company, but every team does its due diligence to keep the tech debt minimal.
LORENZO PASQUALIS, DREAMBOX LEARNING
We try to not take too many shortcuts, and we have a dedicated QA function that helps us decide what the quality bar of our code and product needs to be. Deciding to put some items on the back burner to add to the technical debt pile is a compromise we make with eyes wide open, knowing exactly the risk we are taking and the issues that we are going to cause in the future if we do not address the debt.
CHRIS CAVAGE, DIGITAL CONTROL
We do what we can upfront to avoid technical debt and the engineering time it takes to address it. Our most successful methods have included integrated code reviews with our commit process and proper software whiteboarding design sessions before starting major feature work. We’ve been pretty successful leveraging Perforce Helix Swarm and piping code reviews into our source control process on adopting projects.
ANDY PIRO, PLUTO
We talk through technical design of new features and cost them out as a team, and the majority of the time we also pair program and require approval of PR’s. The more engineers involved in software design, the less likely you are to deliberately take shortcuts or accidentally make bad design decisions. We also make use of automated test suites, CI and weekly deployments, and keep PR’s small to make sure we are making small, solid incremental changes. When refactoring existing code our philosophy is to leave things a little better than you found them, so we’re always striving for continuous improvement.
"The more engineers involved in software design, the less likely you are to deliberately take shortcuts or accidentally make bad design decisions.”
BRIAN GAMBS, BRIGHT HEALTH
The most important thing that we do to minimize technical debt is to be thoughtful about our overall product development process and technical roadmap in order to control how and when we take on “good” technical debt. In our engineering process, we build in mechanisms that allow us to avoid bad, unintentional technical debt. For example, our check-in, merge and pull request process includes code review, and we have a culture of collaborative architecture and design that helps all of our teams make good engineering decisions at critical moments along the development path. That helps us avoid rework, which is one of the primary sources of technical debt.
TRACEY BERG & TIM SMISER, CERITY
Berg: The most important approach to minimizing technical debt is to understand when technical debt is being created and create a plan for addressing it, and there are some important steps in doing that.
A solution architecture is a critical part of understanding when technical debt is created — it should outline the ideal technical solution. Technical debt is created when the “ideal” is not fully implemented. The creation of this solution architecture enables these tradeoffs to be identified and discussed.
When a decision is made to go forward with technical debt, the decision-makers need to understand the implications of their decision. What manual workarounds will need to be in place? How much throw-away work will be done as a result of this decision? What flexibility or speed to market are we gaining or losing? What additional risks is the organization taking on as a result of this decision? Then it’s necessary to immediately put the technical debt into the work queue for prioritization and ensure there is an allocation of resources to address high priority technical debt.
LUKUS REINDL, ORTHOFI
This is typically done by leveraging our cross-functional architecture “chapters” to organize our organizational technical roadmap into engineering OKRs, used to align efforts on a quarterly basis. Each of our Agile teams carve out approximately 20 percent of their team capacity to work against these items.
Teams are additionally encouraged to manage their team-level tech debt roadmap. If the teams have additional capacity left over within their 20 percent allocation, they are encouraged to execute against this roadmap. They can also argue that their team roadmap initiatives should take priority over engineering OKRs, which is a welcomed debate as part of quarterly alignment.
Finally, not all tech debt should be considered part of our 20 percent allocation. Items such as scoping decisions should continually be vetted and collaborated upon between product, engineering, and our business partners, using transparency to help guide good technical business decisions.
LEE GONZALES, MAXWELL
Leave code better than you found it and steadily pay your debt down by allocating a steady and consistent budget to its eradication or management. Scheduling 20 percent of a team’s time is often a good guideline. Good situational awareness also helps, as can hiring experienced senior people who can spot the patterns of technical debt, or sniff out the scary smells of dark debt.