The Tech Debt Trap
 | This is an essay. It expresses the opinions and ideas of some mediawiki.org users, but may not have wide support. Feel free to update this page as needed, or use the discussion page to propose major changes. |
There is an inconvenient choice we (programmers) are often faced with when trying to develop a new feature, or improve an existing one: when we encounter technical debt, do we take the time to fix it, or do we work around it? Fixing it takes time - potentially a lot; and we may not even be sure how to do it. But working around the debt adds to the problem.
Far too often, the pressure to "deliver the feature" or "getting shit done" wins out: we go for the quick solution, debt or no debt. But this is short-sighted. It adds to the overall debt of the code base, the overall overhead the organization has to deal with. Accumulating technical debt means feature development becomes more and more difficult, and more and more nasty hacks become necessary to "deliver". That's the "interest" we pay on the "debt".
Causes
[edit]One cause for this lies in management culture: Product managers often feel responsible to push their product forward, but feel little responsibility for the code base as a whole – or are assumed to think that way, so tech debt isn't even brought up when discussing priorities. Another cause can be the attitude of engineers: green fields are more fun than brown fields. Designing a component or feature from scratch is more immediately gratifying than digging into old code in order to refactor it. Indeed, refactoring that old code may be utterly beyond the skills of a new hire who got tasked with implementing some new feature.
But I believe the underlying fundamental issue is unclear ownership. Programmers constantly work with layers of abstraction (interfaces): they build their own layer on top of an existing layer, with the top layer being the user interface. But often, the underlying layer, the foundation that we are trying to build our feature on, has problems. It may offer insufficient flexibility, or insufficient guarantees, or may simply be very hard to use. So, whose problem is that? Who needs to fix it?
Currently, the answer is frequently "nobody, let's work around it". That of course means we accumulate more debt, and pay more interest eventually.
Solutions
[edit]This approach is not sustainable. Instead, the owner of the problematic component needs to be found, and a solution prioritized and scheduled. This may block feature development, or a temporary workaround allows feature development to continue, until the proper solution is put in place later. The problem here is that "later" tends to become "never" - it's a matter of management discipline to make sure that we eventually get to the chores we pushed back for the moment. We are then creating "acceptable debt", and must have a payment plan, and stick to it.
Who is responsible for cleaning up technical debt?
[edit]So, who is responsible for cleaning up the technical debt that is holding up feature development? Who owns that old crusty code? And if there is no clear owner, who should own it?
I see two cases, a "vertical" and a "horizontal" one:
- In the "vertical" case, a team is trying to develop or improve a user facing feature, perhaps in JavaScript, and is hitting difficulties with the backend that exists for this functionality. Perhaps the goal is a more interactive UI for user preferences, but the API for getting and setting preferences is not sufficient. In that case, the team responsible for implementing the UI should take ownership of the backend module, and improve it to fit their needs. If they do not have the skill set necessary, they have to import these skills from elsewhere, by learning, or by borrowing an engineer. In fact, doing both at once seems like a good idea: the person who knows the old code can help with the refactoring, and at the same time transfer knowledge to the team taking ownership.
- In the "horizontal" case, the team building the "higher" feature hits a problem with a "platform" component that serves many very different use cases. Examples of such horizontal, cross-cutting components would be a database abstraction layer, an API framework, logging infrastructure, a dependency injection mechanism, etc. Ownership of such a horizontal should lie with a single team, and should not be divided between multiple teams that use that component, to ensure consistency and avoid the problem of "everyone doing their own plumbing". In that case, the team working on the feature would specify a need, and ask the team that owns the platform component to fulfill that need. If this would block feature development for too long, this indicates that resources should be shifted from feature development (which in this scenario would be waiting for the platform team) to the platform team (which is presumably over-committed and under-resourced in this scenario).
In both cases, it is clear who is responsible for paying the debt, and there is a clear plan for when it is done, and what depends on it. Currently, this is often missing. The attitude that we have to get feature X out, and can worry about improving our code base later, has led to a situation where feature development has become more and more difficult. Which of course makes it even more attractive to use hacks and workarounds to get feature Y out the door, making the problem even worse.
We have to stop that vicious circle somewhere. We have accumulated quite a bit of debt, we need to start paying it back. And that means all engineers, all teams, not just some poor souls in a "make everything better" team. Yes, that means feature development will be slower for now. But only this way can feature development become faster and easier in the future!