In the previous chapter, we’ve talked about what are the benefits of getting the right Feedback at the right time. It prevents us from going down the wrong rabbit hole, and from wasting time on coding. Alas, there is one more thing that takes us there. And that is misunderstanding Feedback. And we do plenty of that.
A Fixed Mindset
At DealPly (2015) I once had an idea how to completely transform our web scraping mechanism. Instead of an algorithm running on the server, it would run on the client (web browser) where there is much more viable information. The client can make a better-informed decision without actually calling the server’s API.
As it was just a thought, I wanted to properly validate it first myself. It took me 2-3 days to code something to run manually in my browser’s console. Still in my console, I applied it to the company’s top 200 scraped sites.
I ran into the CEO on the stairway and told him the results show I’m about to double the company’s clicks and income. What a showoff. Never do that. Never ever do that. No reason for anyone to believe something so unlikely.
A few weeks later and with some effort, I managed to run the algorithm on 5% of the company’s traffic and on 30,000 sites. The clicks were doubled as predicted. Unfortunately, I also broke one of our main KPIs. It surged from around 5% to 99.9%. It was both an indication of something going horribly right and going horribly wrong.
Although me and the Head of BI had proven it was an absolute positive outcome, we failed to convince others of it. As it is a KPI, people’s minds were just too tightly attached to it.
We will never know whether my algorithm would have made an impact. What we do know for sure is that failure to understand the Feedback, and failure to explain the Feedback has led to discarding it.
Glitch in the Metric
At Wiser (2017) my team and I were working in Israel on a web scraping platform. It was running hundreds of scrapers and emitting about 50 metrics on each. All of them were emitted to Prometheus. It was entirely accessible and visible to the Customer Success (CS) team in the US. Because you know, transparency and such.
One day we in Israel found out that the metrics feature was no longer a PoC. I was assigned a ticket from CS. “metric X is always around -0.2 while regular functioning scrapers are between 0.5 to 0.7. We suspect there is something wrong with it. Can you have a look at it?”. Turns out an entire culture and several guide books were created around these metrics by the CS team.
I replied. “Guys, these metrics are literally meaningless. They are auto generated by an algorithm we wrote and never had the time to look at what they mean. Metric X coincidentally does have a meaning, but you have read it all wrong!”. Not only was it Non-Feedback, it was the wrong Feedback.
Two weeks later I got a call from the CEO asking if the problem with metric Y for customer Z was sorted out. I guess my efforts to change the already written guidelines were almost as meaningless as metric Y. The guideline said to escalate it to me when something is wrong with metric Y.
The Odds Engineer
“Your algorithms and products are amazing, but your code is shit. The problem is you are thinking like an industrial engineer, in the form of processes. And that’s why you produce unmaintainable code”. That is probably the most important Feedback I’ve ever gotten, one from Yuval Gilad (a.k.a Guli) who was my manager at Wiser.
The Feedback made me read and practice Clean Code. And not less important, it made me realize something deep about myself. I’m not different, I’m merely thinking differently.
Computer Science is based on algebra. Industrial Engineering is based on statistics. Probably why to this very day I’m having a hard time thinking of data structures and algorithms (going to work on that). On the contrary, It’s always been much easier for me to solve problems with machine learning than for most software engineers.
Same goes for product/project/process/people management. Not only have I studied this, I’ve practiced it while other engineers were.. well.. coding algorithms. This has given me a hard time. The hardest times are when I’m the only one who sees a decision made based on incorrect understanding of Feedback. Some of which is due to lack of awareness of statistical biases.
At Silo (2018) in order to properly validate some of the physical device’s design, we’ve put out a survey. It had one question with two answers. Answer A was put on the table by the team’s world class designer, and 40% picked it. Answer B was put on the table by the founder, a world class mechanical engineer, and 60% picked it. Only one of them was happy with the results and an argument started. In between the two was me, a working class software/industrial engineer.
“Your solution goes against everything ever taught in design! I don’t care what the people said, it is plainly wrong!”. On the other hand, the founder’s option did have the majority of the vote. I stepped in. “You both are not wrong. There were only 40 participants to the survey and the distribution between the two options is most likely coincidental. The only conclusion possible is no decision should be made based on this”.
Only then the three of us noticed that the question itself was wrong. We reran the survey with 80 participants with a rephrased question. It was a clear 85% vote on the designer’s option. “You see!”, she goes, “the people know a good design when they see one!”.
She was happy that the people, who just the day before knew nothing on design, are now on her side. The founder was not as happy as her. So not only do you have to read the answers correctly, you have to ask the right questions. And to be objective about it. That’s another way to avoid going down the wrong rabbit hole..
But maybe I’m not as good at this as I think I am.
Implications
A few years earlier, while I was working for eWave (2014) I joined a project at Barclays Capital to build a new enterprise search engine. One of the inputs required is how much data in GB on average each IT user has.
My Product Manager had an excel spreadsheet with all the information and she just clicked the AVG function and got a number. Me, with the authority of about 10 courses in statistical algorithms, thought otherwise.
I had a look at the distribution of the GBs and found out it wasn’t uniform nor normally distributed. Plain average could be wrong. I did a weighted average based on the histogram and got a number three times bigger. And we based everything on it.
A year later I told the very same story to DealPly’s Head of BI, who had a M.A. in statistics. “You were wrong”, he said and I gulped. “With a large enough sample size, all distributions converge into a normal’s mean”. “On the other hand”, he continued, “you’ve correctly done the histogram and the weighted average. So I have no idea why there was such a difference between the averages”.
What I do have an idea of is that for sure servers were bought. Three times more due to my calculations. I should be more careful when I provide such Feedback. Someone should definitely properly validate my Feedback.
Irrationality
What’s funny is that I really am no different from any one else. Most of us do think we know better than we do. If you want to know why and have a few hours, I highly recommend reading The Undoing Project. It’s hilarious, well written and a good intro to the field of behavioral economics.
As our reading time is precious, I’ll try and summarize the work of Daniel Kahneman (Nobel prize winner) and Amos Tvertsky, and how it relates to us who read and give Feedback.
The two came up with The Linda Problem:
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is more probable? (write your answer on a piece of paper if you wish to participate)
- Linda is a bank teller.
- Linda is a bank teller and is active in the feminist movement.
The majority always pick option 2, although option 1 is truly more probable. It is called Conjunction Fallacy and it is one of the many erroneous and somewhat delusional ways we human beings perceive reality.
We are all affected by our emotions, past experience and even our spoken languages when it comes to Feedback and decision making. It is good to be aware of such fallacies. One of their findings is that experts, researchers and statisticians are more fallible to it than others. Mind blowing if you ask me.
Speaking of experts and expertise, in the next chapter we’re going to talk about two of them. The one of Software Engineering and the one of Product Management, and the Feedback between the two.