Ljubljana, Slovenia

Technology debt, part one

In the late 90's, after 96-97, I was a part of the demo scene in Slovenia. This was a somewhat informal gathering of like-minded hackers, programmers, artists, musicians who liked to have fun. There was a vibrant BBS scene at that time, where at least three or four BBS were active, one of them my own. I'm not even sure why we had BBS now, but before IRC and demo parties it seemed to be a good way to get a C++ compiler and read several tutorials and e-zines.

Amongst the digital underground, a few books and other materials were widely circulated. The memorable ones include the Phrack ezine, the Anarchist cookbook, and the Hacker Manifesto. Among the more comedic literature dealing with computers, you could read the adventures of the Bastard operator from Hell, and should absolutely take a look at the Tao of Programming (book).

To quote the latter:

A well-written program is its own heaven;
a poorly-written program is its own hell.

There is sometimes a very fine line between a new feature and technology debt. The difference between them, in my opinion, is that a feature consists of a requirement, which wasn't expected or provisioned when an application was in it's design stage. Technology debt is already a feature, which somehow slipped past some kind of quality assurance or just normal load or sanity checks, and presents a problem in it's current state. This feature needs to be adapted, rewritten or completely reimplemented from the architecture standpoint.

Sometimes, technology debt can be mild. A normal thing for beginner developers is to use the same server for development and production, and more often than not this can lead to problems. It is a form of technology debt, as accepted practice between serious programmers is at least some kind of deployment mechanism. I know some people, who have at least 4 levels of deployment: internal development, internal testing, wide testing/pre-production, and production. We are currently using 4 levels of deployment for our biggest client, come to think of it.

So, deployment often includes several servers. This is already something that needs to be maintained. Think of configuration files for your databases, key value stores, file servers, web servers, load balancers, and probably custom software. Like the Tao says:

Though a program be but three lines long,
someday it will have to be maintained.

Here the technology debt can start to build up. You have to think about backups, system restores, maintaining configuration. If you're growing you also need to think about infrastructure, in software and in hardware. Is the PHP you're running a good choice or should you research LUA, Erlang or other software? While some things are basic and can be solved with a small amount of work, others are large and keep coming to bite you in the ass.

Many start-ups over the years accrued technology debt. Some handled it better than others. A good example of handling technology debt would be Twitter, and a bad one would be MySpace. The companies design choices allowed them to solve new problems in a certain way, and eventually, a design choice they made led one of them down a dead-end street. In a way, it is Darwinism at work.

It is not the strongest of the species that survives, nor the most intelligent, but the one most responsive to change.

A lot of technology debt can be avoided with careful planning in the design stage of software development. Some things can be anticipated. If you're developing a new feature, you can think about speed, scaling, volume of content, and other qualities and requirements. While you're obviously not clairvoyant, you can predict some reasonable guesses what kind of problems your project is going to run into.

In one of our cases, we created a CMS for a Client. Over several years, the CMS was being extended with new features, adding the Clients every new requirement. The problem was that one of these requirements required us to create a copy of the CMS. But the CMS deployments needed a lot of common data. The part which was the eventual downfall of the CMS was not predicting which data needed to be shared between CMS deployments, and then maintained on each of them.

In the end we ended up with about 6 copies of the CMS, which were taking all our time just to maintain, let alone develop new features. Our only option was to build a new CMS. We are using the new CMS without problems for the last several years now, even reducing system requirements because of better design decisions. But like with the first CMS, we will hit a wall.

The wall will most probably be database performance. Several content tables have already filled up several 10 million rows and are not used just for analysis. This is real world data, which is used and updated often. Design choices can be modified, caching layers can be added and we can be good for another 3-5 years. If the content will grow faster, our only option will be to rewrite it completely.

One of my personal opinions is that software needs to be developed with a life-span in mind. Usually, I try to plan and envision what the state of an application will be after 3 years. If the database will grow, we plan for it. If we expect to add a caching layer because of load, we plan it in a way that won't kill us trying to develop it a few years from now. These are not unique things to us, they are common for pretty much any software developer.

So, basically, to avoid technology debt, we think about this:

  • What will be the state of the software and it's data in 3 years
  • If we need to add caching, what can we prepare now
  • Are there moving parts, which we can plan to make replaceable?
  • Intelligent code re-use
  • Backward compatibility or deprecating features
  • API's, DAL/DAO objects, software relationships and dependencies
  • Establishing best practices for development

Using this approach, we can see which software is going to be problematic. By implementing these best practices for development, we maintain a quality code base. Because of this, several of our clients use our software for more than 8 years now. While some software needs upgrading, most of it is still future proof.

- Tit Petric, to be continued

While I have you here...

It would be great if you buy one of my books:

I promise you'll learn a lot more if you buy one. Buying a copy supports me writing more about similar topics. Say thank you and buy my books.

Feel free to send me an email if you want to book my time for consultancy/freelance services. I'm great at APIs, Go, Docker, VueJS and scaling services, among many other things.