TL;DR:
- Thinking in a concrete manner on a concrete solution, for an abstract problem would lead to a tightly tailored solution.
- Too concrete of examples would hide the task’s actual focus.
- There is a tradeoff between generalization and effort needed.
- Avoid generalization by postponing decisions and executions.
- Break down tasks to be executed at certain points in time (now, tomorrow, two years) to bridge the gap between the unknown and what you can do now.
- Model abstract as a ridiculous but possible use case
Abstract thinking is hard to do and even harder to communicate. Think of a design meeting with me that never happened, where I’d wish to show you something like this, which is what the system looks like for now / so far / as far as I know:
If someone were to ask me “What’s this mediator?”, the only answer I had at that time was “I don’t know yet”. If someone were to ask me “Okay, what does this Service do?” he would probably rip my head off after me repeating over and over again “I don’t know” or “it doesn’t matter” or “it’s details for now”. I remember I worked with an architect that did that to me many times, but that was his way of showing people the way out. He didn’t want to help anyone, but I do and I care. I really do want to give you an answer, I just don’t have one. It’s too early to answer that anyhow.
It was hard for me as for some of my teammates, those who were not practiced in that way of thought. At first, I utterly failed as the following quite real dialog happened with Dave, more than once, and there wasn’t anything wrong with him:
Me: Your next task is to make sure that a message is sent to the server from the device and another one received from the server
Dave: What is the message?
Me: I don’t know yet and it’s irrelevant. You can do the task without that piece of information.
Dave: Is it a JSON?
Me: it could be
Dave: Is it an XML?
Me: it could be
Dave: Is it “hello world”?!!?!
Me: It. Could. Be.
Dave: CAN YOU GIVE ME SOMETHING TO WORK WITH?!
Me: You already know enough, but if you insist – a message is of type string.
Dave: What is the String’s length?
Me: I don’t know.
And it went on forever, but I did not lie. At that point of time I did not know what the message was and what format it was going to have. Eventually, we did know. It was 4 months afterwards. Obviously we couldn’t wait that long, we need to know now how the System and the Device will communicate with one another. It would only take sending a String back and forth to discover that. It does not no matter what the String is.
Concrete dries fast
If I were to tell Dave at that moment that it is a JSON, even as an example and even if I already knew that it would be a JSON, he would code a JSON parser. For real, around that time it was yet undecided, it wasn’t solid enough. Another engineer on the team started testing Protobuf which was later dropped. Dave could have coded that JSON parser and it could have ended up being a Protobuf. Surely, he would have to change the code but it might not be that easy because he had hard tailored it to a JSON because “you told me four months ago it’s going to be a JSON! Ahhhh!” and then he’d rip my head off. If you’ve noticed me trying to avoid a Velocity Drop, you’re correct! Thinking in a concrete manner on a concrete solution, for an abstract problem would lead to a tightly tailored solution.
This is not even the main issue at hand. If I were to tell him a specific protocol/structure, no matter which one, he wouldn’t be focusing on the task at hand, which is discovering the correct client to server communication method or protocol (HTTP/MQTT/WebSocket/etc).
The other way around, of an abstract solution to an abstract problem, could lead to an over generalized solution, one that would not be needed at all. It would be Dave’s time invested into a solution that we would one day, in hindsight, say was over-engineering. There will be exactly one protocol. No need for him to waste time on an AbstractMessageProtocolWrapperFactory class. There is a tradeoff between generalization and effort needed.
If you think that I could have acted better then you are absolutely correct. This was a gap for me to cross. Something for me to get better at. We came up with several approaches and mental/mind tricks to help my team and I to get through the abstract/discovery phase of the system. We also sometimes mixed between some of them. If you’ve read the article The Road to Know-where, it won’t be the first time you’ll be hearing of these.
Let’s not do this!
The first was to master the artwork of decision postponing, the weird uncle of decision making. To realise that it is okay not to do anything right this instance. To delay coding as much as possible. To give us the time needed to gather as much information as possible, until we get something that is solid enough, when its odds of changing would be fairly low. It took us a while, as it is hard for eager developers to shift it down a little, but we ended up many times saying with a huge smile let’s not do this!
The time traveller
Another method we took was to break down the abstract task into smaller less abstract ones that will be executed over time.
For Dave’s message task the break down would have been:
- For now, a message can be a simple “hello world”
- In four months from now expect that you will need to easily change your code to parse JSONs
- Two years from now someone else might need to change your code to parse Protobuf. Make sure it will not be impossible. It’s okay if it will be hard on him.
The first one is a task that Dave could start code immediately, and he’ll code it knowing what and when the next one is. The second one is the actual one needed for the project. The third one, would most likely never happen, but it’s there just to make sure the first and second tasks’ designs are good ones.
The Ridicule
One day at Silo, Itai, Kiril and I were working on a design for the Metadata Service. It’s sole responsibility was to store metadata on system objects. For example, the Container Metadata and Device Metadata would consist of a UUID, date of manufacture and batch number (not the food stored in the container!). The User Metadata would consist of a UUID, address and email.
At that point in time, we knew almost exactly what the data would look like, and what a change to the data would entail and the odds of it changing. We did not know what it would be stored for. We did not know who for. No one knew and as I keep saying over and over, no one will ever know. Instead of me saying “it doesn’t matter”, and to let them figure it out, I gave them three use cases. A simple one, a normal one, and a most ridiculous use case, but one that does not sound absurd at all.
- Today the COO would like to know how many Devices had been activated to better prepare for the next batch of manufacture
- Tomorrow the product team would like to know how many containers in average each user has, to decide on a change to the mobile’s design.
- One day, the marketing team would like a report. All of our users’ emails, those living in Texas, that have no more than 1 device and at least 4 containers that were purchased during the last 6 months. They would use it and would give each user a Hello Kitty Special Edition container.
(Notice the structure of a time frame, an actor/customer, and a need)
The result was a better understanding that we would need query flexibility, so no matter when in time the data can be queried freely. That was enough to lead us down the path towards relational databases (SQL) and table normalization, and away from non-relational ones (NoSQL).
Cross Validation
Cross validation is not new and is constantly used in machine learning validation. We just took it and traversed it to the world of design. We came up with an extension of The Ridicule above. Instead of just one sample of each kind of example (a simple, a normal and a ridiculous one) we’d write down a bunch of each. We then splitted the list into two. We iterated on the design with the first half of the list. Once we reached a solution that perfectly covered it, we validated the design against the other half of the list. If it failed to cover it, we’d shuffle the list again.
Hopefully, I’ll discover more methods like these and share with you all.