We’ve previously described two evolutionary processes, to a Monolith and to a Bundle. We’ve seen that given enough time, they will evolve to it. We’ve dug into our application’s Event Stream to see its causes, actions and events. The third evolutionary process towards a Legacy is on the same Event Stream. The only difference is that the causes and events are external to our application and to us, and are out of our control.
Impermanence
The simplest example to begin with would be the one where no Change was introduced at all as there was no Cause. No product change request, no bug, no refactor, nothing. That was the story of the HTML to PDF at RapidAPI (2021). It was a very small AWS Lambda Function that did exactly one thing. It had generated invoices to customers as a PDF from an HTML. It was coded in 2018 and was only about 20 lines of code. No one had to touch it. It lived and performed 100% correctly years after the developer had left the company. A perfect example of “it ain’t broken don’t touch it”.
The code was in Node.JS 8.1. The version’s official end of life was 31 of December 2019. As there was no Cause to change the code, there was no need to update the Node.JS version. All is good. AWS Lambda’s support for Node.JS 8.1x ended on March 8th 2020. As long as there is no Cause to redeploy the Lambda we are all good.
That is until January 2021. A Cause occured, a business and technical need to set up the platform in another AWS Region. This required a redeployment of all of RapidAPI’s applications to it. Deploying the Lambda “as is” had failed. We’ve updated the Lambda runtime and the code to Node.JS 14.x. Alas, no PDF was generated at all. The current PDF converter package version did not support 14.x. It had to be updated to a newer one which does not support AWS Lambda runtime straight out of the box. It took us a few good days of logging, tracing, debugging to get it all worked up again. Almost as many days that it had taken to code it for the first time.
Suddenly by no one’s fault or someone’s wrongdoing, in January 2021 this Lambda function entered a state of Legacy. Prior to it, it was in a non-supported state, entered due to an external event, the end of life event. Due to another external event, the AWS Region deployment, it has entered the state change to a Legacy.
It’s best if we’d review this process of state change with a state machine.
Once successfully deployed it has exited the Legacy State back to the initial Stable State. Meaning it once again has the potential to reenter a Legacy State in the future. Given enough time it would eventually happen. How come?
Node.JS 14.x was the latest version RapidAPI worked with in January 2021. The 14.x version entered a Long Term Maintenance status in October 2021. It would reach EOL support in October 2023. The AWS Lambda Runtime for it would be deprecated around Mid-2024.
We already know today what will be its state in 2024, the non-supported one. Once someone will try to redeploy it in 2024, maybe to set up a new development or staging environment, it will reach and fulfill its potential and will become a Legacy again. We will come back to this story in the following series of “The Inevitable”, where we would see why the decisions done here were 100% correct.
Falling Behind
We’ve just reviewed why and what happens when a very small and well isolated component reaches a Legacy state. Let’s have a look now at what happens with much larger applications, systems, and requirements. The consequences intensifies.
This true story is set in 2011. My first job was at eWave, an IT company mostly doing out source projects. My title was “enterprise search expert” and I was no expert at anything. I was just a junior with zero practical experience with a lot of somewhat unique theoretical knowledge. I was dropped into a project in its early stages, a proof of concept of a search engine (based on Microsoft FAST) that would help the Israeli government find and track the black markets, tax evasions and money laundering. It was estimated that these three caused losses of taxes of about 6Bn$ which back then was about 6% of Israel’s annual budget.
I remember the day I stepped into the government agency’s IT department in Jerusalem. The first task at hand was to evaluate the integration effort required between their systems and the search engine. In 2011 the most common framework in Israel was .NET and the latest version, which eWave was working with, was 4.5. So no wonder that my eyes went numb when I thought the agency is working with 2.0 (Released 5 years before). I was a bit of a young showoff back then so I sarcastically asked “what are you working with DOT NET TWO POINT OHHH?”. They had laughed, but not because of courtesy to my non-funny wits but because they were working on .NET 1.1 – which was released 9(!) years before. The real funny thing was they have been planning to upgrade to 2.0 soon for the last 5 years. We had finished the PoC around 2013 and they were still with 1.1.
The search engine’s client was only compatible with .NET 3.5 and above. We had to entirely re-code it. Instead of 2 days of assimilating the already existing client, it was around 15 days of coding ending up with a less reliable client then the hard tested one. That is 13 days wasted due to usage of legacy technologies (~5.5% of a developer’s annual work time). An inefficiency caused by Legacy. Imagine all the things the agency had to code again only because they were falling behind on their .NET version. Imagine the things they can not do at all.
Technology, like time itself, keeps moving forward. If you allow yourself to lag behind you’d be creating inefficiencies which would be decreasing your Velocity. Instead of channeling the effort towards business oriented goals, you’d be wasting them on catching up – fixing bugs and adding features that already exist in a newer version [see We Were Born to Run: [In]stability and Velocity].
The Billion Dollar Break
It took 2 years to finish the PoC, not successfully. The nation’s tax data was stored in a database called Adabas. When Adabas was first introduced it was revolutionary. It was one of the first commercially available NoSQL databases, an inverted index non relational one. It has been in existence even before the term NoSQL was coined!
Unlike the .NET framework they did keep using its latest version. Pushing it forward from the early version of the 1970s… all the way to… the 1990s. The year was 2011. The agency found itself trying to integrate with a 25 years old database. The agency wished to allow advanced search and analytics on a database without JOIN capabilities. In order to run the required SQL queries they had purchased and installed an SQL Gateway on top of this NoSQL database. That gateway allowed us to index the entire DB to the search engine.
We managed to make it work in the PoC environment. When the SQL Gateway was first installed on the production database, it completely crashed as the recall and the complexity of the queries were too large for it to handle. The entire project was abandoned. The development of a major crime prevention tool has failed due to a legacy database. A loss of billions of dollars for the country.
It may sound like the engineers are entirely at fault. That can not be the entire truth as we’ve seen that Legacy is caused by external events that are beyond our control. They do share some responsibility. They may not have been prepared enough for the fact that time moves forward. Which we’d be talking about in the next chapter.