Addressing technical debt is a severely under-valued need of most company codebases. While everyone agrees that it is important, somehow it never quite gets the time it deserves . I want to take a look at why this occurs, what can be done about this and what some of the unintended side affects of this can be.
Why is refactoring needed? Simple. Unrefactored code causes slower delivery of buggier features. This isn't news nor is it profound. Everyone agrees that fast delivery of working features is important. Yet, somehow, sparse hours are given to paying back technical debt.
Its not that time isn't planned for doing this. The common approach is "We'll give a couple of hours/days towards the end of the sprint, but ..." (and here's the kicker) "we just need to make sure feature XYZ is finished first." I will let you in on a secret, between production fires, bugs and XYZ growing in scope, that refactoring time will never see the light of day.
Furthermore, the worst time to refactor is after adding new code that hasn't been QA'ed. It is similar to building sandcastles by the sea. The time to start moving around major chunks of code is after everything has been signed off as working as expected. That way you know when your cleanup work breaks something. Also, this prevents you from being tempted to back out all your refactoring if feature XYZ does pass QA./
This puts the developers in a horrible position. They have two options.
One, steal away time from other assignments they have or work on this during nights and weekends. This sucks. It tells the developers that their internal company needs are not important. Also, (and often not considered) this precludes sufficiently complex pieces of code from ever being unraveled. If it takes more than a couple of hours to do, developers will never be able to do it. The areas of the codebase that need working on the most, will rot.
The other option, is that the developers don't even try to refactor anything. This will lead to a lot of "damp" code (the opposite of DRY) that takes longer than necessary to produce and will likely be buggy. This won't be because the developer can't do it. It is because the developer can't do it on top of the Rube Goldberg codebase that has arisen. This is the best way to take a top shelf developer and make him feel like a failure. (hint: If you start seeing your best developers leave, look at the codebase.)
Often this is where most development departments end up.
Others will be fortunate enough to finally gain buy in to get an entire sprint dedicated to refactoring. Here to, I would like to caution the would be team. Product and Marketing will be all too keen on seeing this as there opportunity to have you fix ever minor bug and one-off that they has ever bothered them. The second you crack this door open, expect a flood to wash away this opportunity.
A refactoring iteration should be for the developers and the developers alone to set priorities. They write the code. They know where things are the messiest. They know where the biggest "bang for refactoring buck" lies. Trust and empower them.
Now, with all that said, we can't spend all of our time perfecting the codebase. Companies need to make money; they need to test business plans. So, where is the balance? There is no magic formula, but let me propose the following two rules of thumb.
1) One of every every eight sprints should be a refactoring sprint. From experience, this seems to be a good pace to fight programatic entropy. Also, refactoring has a rather close parallel to the seventh habit and should have the proper proportion of your time.2) After a company pivot has been proven. If your company has tested the waters and decided that it is time to change directions, then it is time to prepare the codebase for this. Batten down the hatches. Those first months, as a new idea is being actualized, being able to rapidly flesh out the idea is crucial to maintain momentum. If you are slowed down by the needless legacies of prior pivots, you are not even giving this new idea a fair chance to flourish.
The quality of the codebase a developer works in can make the difference between their job being something they do versus something they love. Do what you can to make sure your developers have a job they love.
What do you think? Are there other ways to tackle technical debt? What are some other roadblocks to look out for on your way to a refactoring iteration? Are there any other rules of thumb that can be applied?
If someone came over to your house and built a deck, then left your yard strewn with the sawed off ends of boards, nails, and sawdust, you wouldn't pay their invoice.
ReplyDeleteCompanies that refuse to allow for refactoring iterations are insisting on leaving the cut up boards, nails, and sawdust in their yards. And when people start stepping on nails, they regret it.
Code isn't something you can clean-as-you-go. That's for McDonalds kitchens.
Mixed feelings... the metric of money over looks the value of this activity. It'd take a long term outlook and a commitment to quality over profits in many instances. Still, as a programmer, I realize all too often many projects are built even as their specs change, and I wish we could refactor more often, I'd probably have more pride in my work if we did.
ReplyDeleteC'mon Aaron, its not rocket science...
ReplyDeleteNick, I think that is the problem. Their is a perception that refactoring code is not a value creating activity. I think that having code that is maintained is in the companies best interest. Specs do change; company's entire focus change. If you have a lot of legacy janky code they have to carry with them, they will take longer to get profitable (if ever).
ReplyDeleteBrent, you know me entirely too well. =D
ReplyDeleteI have to agree with Aaron's comment to Nick. It's not a question of commitment to quality over profits, it's that technical debt can slow down development of new features. This means that the ROI of addressing technical debt is much harder to calculate than the ROI of a new feature.
ReplyDeletePerhaps that gives us some new insight into how to estimate the costs of technical debt (and the expected value gained from resolving it)!
To start, estimate how much the TD in a given module is slowing down new work on that module - maybe you think it slows down any work by 20%. Then, estimate how much time you expect to spend working on that module for a given time period - say the next 2 sprints. If you think it slows down your work by 20% and you expect to spend 40 hours in the next 2 sprints, then the value of resolving it is 40*.2, or 8 hours. If it will take you 6 hours to resolve it, then fixing the technical debt is worth the time.
This would suggest another rule of thumb - technical debt in an module that is being actively developed is far more expensive than debt in a module that will be dormant. That also adds weight to one of the previously suggested rules of thumb - if you are about to shift development priorities, it makes sense to evaluate the areas of code with the most active development and eliminate costly technical debt.
Finally, one commonly overlooked cost of technical debt is the escalating cost of fixing a given bug. A bug caught in the design phase (catching an error in the spec) is much cheaper to fix than catching it after development work. Catching a bug while developing a feature is cheaper than having QA find it just before release. Catching a bug in QA is FAR cheaper than letting your customers catch it. Aside from the costs to your tech support department from dealing with annoyed customers, the cost to your reputation and your customers' trust is immeasurable. Good design & well-factored code is one of your best lines of defense against this cost.