At the end of the previous chapter we spoke of erosion, a process that would cause a demise or decay when left unattended or incorrectly attended. Lack of awareness of these processes leads to unrealistic expectations or unattainable values. We argued that awareness of these processes would lead us to avoid non-beneficial actions, in advance. Which is exactly the goal of The Beneficial Way Framework.
If we could define similar erosive processes to software engineering and application development, we’d have a rule of thumb on how to identify non-beneficial actions. Luckily, we already did so. The series of Wasteful Applicative Evolution has already defined 4 erosive processes (and there could be more that we haven’t noticed, so it is extensible):
- Each and every application has the potential to become a Monolith, it evolves to it. With enough time given, it eventually will. Giving in to it would mean to do our best at any given time.
- Each and every application has the potential to become a Bundle, it evolves to it. With enough time given, it eventually will. Giving in to it would mean practicing small changes, small deployments with proper monitoring.
- Each and every application has the potential to require a Refactor, it evolves to it. With enough time given, it eventually will. Giving in to it would mean to continuously [From Nothing to Something ref] refactor.
- Each and every application has the potential to become a Legacy, it evolves to it. With enough time given, it eventually will. And we’re about to see what giving in to this means.
No application can be constant and no application can run forever as long as these processes exist. These erosive processes will always exist because what feeds and moves them is the Change itself, the work that we do every day. And Change will never cease as long as we wish to keep the business moving forward. Which we do. Not only that, Change is happening faster than before. Problems grew bigger and technologies shifts faster than ever.
Any task given that has an intent to put a stop to any erosive evolutionary process (“we’re going to refactor this once and for all!”) has the potential to be non-beneficial and avoiding it should be discussed. And when an erosive evolutionary process comes to its full potential, there is no need to be alarmed.
Now that the theory is set, some examples of the Eventualism analysis, practice and its consequences would do better in understanding its principles.
We’ve previously talked about RapidAPI’s PDF Generator to describe the evolutionary process towards a Legacy state. It was a three years old AWS Lambda function that failed to redeploy because Node.JS v8 runtime was no longer supported by AWS. It would be an amazing example as all actions made by the developer were correct, unintentionally correct and yet.
When coded in 2018, the developer had no reason to suspect a deprecation event would occur. More than a year after completion, around November 2019, the Node.JS 8 Lambda runtime deprecation was announced. It was finally deprecated in March 2020. Up until we tried to redeploy the Lambda in January 2021, it worked perfectly with zero maintenance for about three years straight. If it weren’t for the need to deploy to a new region it would have lasted even more than that.
Let’s analyze this with Eventualism. Under the expectation that an application should run forever, the developer should have acted otherwise. As it was in 2018, today we know what were the technical alternatives back then. It was either running it on a fleet of EC2s or with Docker Containers. It would have been terribly expensive to run it on EC2s, so containers would have been the wiser choice. As in 2017 I’ve set up such infrastructure at Wiser, I could safely say doing the same at RapidAPI would have been about 8 weeks of effort. Let’s presume it was successfully done and the PDF Generator ran well within a container.
Let’s fast forward 3.5 years to the present time of writing this, December 2021. These days AWS is moving fast to ARM based CPUs, offering it to customers for about 20%-30% less than x64 based. Including ARM64 based Containers. What is less known is that a container built for x64 cannot run on ARM64. Containers are processor architecture dependent. It needs to be rebuilt. Thus the potential for the PDF Generator container to become a Legacy, just like the Lambda, would have already existed. As RapidAPI would one day wish to reduce 30% of it’s costs, it is only a matter of time before the container would need to be rebuilt. It would eventually happen.
So, under the misexpectation that an application should run forever, we would have spent 8 weeks of effort instead of 3 days for each [re]deployment (one in 2018 the other in 2021 and another one in a few years a total of about 9 days). That would have been a huge time waste.
Under the awareness of the erosive evolution process towards a Legacy, we would have known not to go down the road towards Docker containers, even without knowing the exact details of when and what technology shift would occur. We would not have tried to prevent the inevitable with Containers, but only postpone it with rebuilding Lambdas.
This insight and principle does not mean to always do Lambdas instead of Containers. Not at all. It comes to say that different choices have different effects on evolutionary processes, they postpone or haste it differently. And this should be one of our technical considerations before performing an action, a consideration that might correctly lead us away from non-beneficial actions.
Under the misexpectation that an application should run forever, it is no wonder that some of our colleagues have reacted with a hint of blame, to say the least. Some expected it would never happen because we have ‘micro services’ and use AWS Lambdas. Misconception, mis-expectations and miss awareness of erosive processes have led to a non-beneficial behavior of blame. It is no one’s fault that these erosive processes happen, there is no need to blame anyone.
Under the expectation that “it will eventually happen” I reacted with ease, trying my best to clear the air. “There is no reason to be annoyed by this, it was bound to happen. That is what eventually happens to each and every application. Let’s not waste our time digging into the past, it is non-beneficial. Let’s just get the work done and move forward”.
I must be honest here Eventualism does have its caveats as it is a different and irregular line of thought. I did a round of exit interviews when I left RapidAPI near the end of 2021. My employees loved it that we were at ease. Specifically when something went wrong because we were indeed focused on fixing it and not on blame. Just as the eastern practice of The Middle Way promises, which is the source of our Beneficial Framework and Eventualism. On the contrary, my superiors’ feedback was that this sense of ease was too much for the company to process. I must also warn that it is known that too much of an ease can hurt Velocity and throughput, much like too much of a stress (The Yerkes-Dodson law about relationship between Stress and Performance).
Off the Edge
When I first joined RapidAPI (late 2020) I was also an interim manager of the Marketplace Squad. They owned an application (Frontend, React.JS) which was the “face of the company”. It was about 6 years old and through the ages about 30 different engineers have worked on it. It was a mess of a Monolith and it was slow as hell. It took 30 to 90 seconds for the site to initially load and a few seconds to load each page. It hurt the business badly. The sales team had a really hard time selling such a slow experience.
I gathered the team and asked them what their plan to fix this was. They detailed a huge refactor and were furious it is constantly being rejected by management. I did my practice of standing still, saying nothing and processing. “What do you need to get it done?”, I’ve asked. “The entire squad”. That’s 6 engineers. “And how long is it going to take?”. They said it would be at least 3 months before maybe we can re-code any product feature. “What are you doing in the meantime?”. They do nothing.
The application at hand was already both at a state of a Monolith, a Bundle and it requires a refactor. I’ve analyzed their suggestions through Eventualism and lectured them the following.
“So.. let me tell you how that sounds to management. You were suggesting a full stop to the business for three whole months during Series A fundraising? For ‘a maybe’ that would just require another refactor some unknown time afterwards? And you fail to understand what is wrong with this plan?”. I’ve then done something wrong and said that it sounds unprofessional to me. No one took it personally but I should have said that it is non-beneficial. To summarize, at that moment in time they were doing nothing, which is a non-beneficial action, because they were mentally on hold to get a green light for a plan that would also be non-beneficial. Both end results are to be avoided.
“Guys, what we need is to buy ourselves more time. What we need is a list of quick tasks done right that will fix the problem at hand, the slowness. A plan that would take the pressure off for enough time so that we’d have enough time to fix the application to the core”. If that sounds like a Continuous Refactor to you – it is. Within two weeks with only part of the team on it, the website was not so slow, enough for the sales team to live with it for a while. It was enough time bought so that a newly recruited manager would hop in and would get the team to fix the slowness problem entirely. Which was then enough time bought to write the client entirely from scratch for a newly designed product.
We’ve done beneficial actions at the right time because we knew to avoid and define the non-beneficial ones to begin with.
In the next chapter, we’ll see how Eventualism helps us with other kinds of decisions we make. Those done while we design.