It was arguable how sensitive our customer information was. If someone were to learn the fact you are eating tomatoes instead of carrots, it would not rock anyone’s world. For us, if someone were to steal Silo’s entire data, it may cause some troubles to our business model. Someone else would be able to derive insights and make money out of it.
On the contrary, a breach; a data leak or any other kind of security incident, has the potential to hurt the brand and future sales. For a young startup company depending on future investments, it is a big concern. And also explaining ourselves to our customers. As the CTO, it would mean to do the honorable action of Hara-kiri, or its modern version of getting fired.
Security is not a joke at all. At least not for a small startup company, intending to create a B2C smart kitchen appliance. But the way we handled it started with a joke. Before I joined the company, the founder once went pitching in front of some investors at some convention. He used to go on stage and show a soon to be patented plastic food container. He always said that it was him on stage with a B2C product, and 9 other cyber security companies. I told him “It’s plain plastic, just tell them it’s 100% cyber secured!”. Eventually, the joke was on me. I had to make it so.
What started as a simple stupid plastic container, quickly derailed into including a microwave-safe RFID chip in order to uniquely identify it. You’d think it was there to track each container usage, which might be an interesting thing to know, but no. It was there so when a container is placed on the device, the label given to it would be displayed on screen. It beats me to this very day that we showed a customer something he can see with his own eyes, that there are strawberries inside his own container. Nobody needs help doing that.
The company had one main concern for its plastic containers. That once the product is out, someone would be able to reverse engineer the containers and manufacture them on their own. Kind of what Nespresso are continuously battling against. Every few years their patent runs out, or someone manages to produce coffee pods on their own, and they lose money due to potential lost sales. So, if there already is an RFID chip, no reason to not use it to prevent fraudulent containers from being used with our device.
We had Mike with us, a veteran embedded engineer, and we also met with an expert on the matter. The effort required to make it an air-right solution preventing any kind of unauthorized usage, had involved somehow bridging between an air-gapped physical computer and a cloud based system. The solution quickly turned complicated enough to rethink this whole scenario.
To manufacture the container itself required special equipment. Even if correctly reverse engineered, not just any backyard factory in China can start manufacturing those. Only Tier-1 factories could. Like the one we were in business with, who were years in the industry of food grade plastics. And even they had a hard time with Silo’s containers.
Suppose another Tier-1 factory will try to do the same. They’d have to put in an RFID just like us to get it to work on Silo’s device. Their production costs would be exactly like ours. They’d have no incentive to do so. On the contrary, the device itself is a smart kitchen appliance, full of cutting edge technology such as an RFID reader, speakers, microphones, and an expensive board that runs Amazon Alexa.
What would a smart competitor do? He’d copy the device first and rip out all the electronics from it. That will give him plenty of financial margin as an incentive. Consequently, they’d have no need for an RFID chip. They’d be reducing their own production costs even further. Nothing we can do to prevent that from happening.
It would be a non-beneficial effort to secure our RFID chips. We better leave this scenario to our patent and our lawyers. Or to ourselves, to make a non-branded / white-label version of our own product. There is no software based solution to this.
Security can be an endless effort, so we better invest our efforts to where it is beneficial. And there were only a handful of us, so there was also a limit to how much effort we have to give. Following The Middle Way, we tried to realize what other kinds of effort would be non-beneficial.
Our first thought was we can not prevent everyone from successfully carrying on an attack, if they have enough intent to carry it through. But intent is not enough, it would also require time and resources. An organization like the CIA have almost unlimited resources, same as other less friendly intelligence bureaus from the middle-east. They will eventually penetrate whatever defenses we will come up with. It would be a war we can not win.
The same goes if a competitor of ours would like to bring us down, for defamation. They’ll just withdraw ten million US dollars in unmarked bills, easy money for them. They’ll put it in a suitcase in an agreed upon location, to be later picked up by a notorious hacking group. Those too will win us over. Another war we can not win.
On the other end of the spectrum, there are enthusiasts who will try to breach in for fun. To do so, they’ll have to go through our client-facing Gateway. Fortunately for us, the only way to connect to it is with a physical device that has a pre-burned certificate. So they’ll run to the store and spend a few hundreds of dollars to get their hands on it. We prematurely assured that each certificate can only access its own resources. Whoever this enthusiast is, they can not produce an attack that would affect other customers. They can only hurt themselves. Fine by us.
However, one certificate might be all it takes to publish millions of messages, an attempt at a denial of service attack. Fortunately, our Gateway in the form of AWS IoT knows how to identify attack vectors coming from a single device. Very similar to a web application firewall. Revoking the certificate from our end, can be easily done and easily automated.
Not coincidentally, our messaging platform is built to withstand load bursts. Millions of messages are nothing but a kind of poison pill. It might cause some jitters in experience for a little while, but it would clear out on its own within a few minutes. Sure, it would cost something in scaling up resources, but what attack doesn’t cost something. And because our system architecture is well isolated, an attack would not so quickly bring the entire system down. Only a part of it.
Anyone can easily go to the store and buy one device with one certificate, and it’s something we can handle. An attacker, someone with an intent to do real harm, will go ahead and buy 10 devices with 10 certificates. Our defenses will slowly squat each one down like really annoying mosquitos. Some annoyance to our customers will occur, but our system will recover and go back to normal.
But I’m sure a number exists. If an attacker will go ahead and buy enough devices, he’ll be able to cause some serious damage. Maybe even for a long while. But to buy a device, a certificate, is to spend hundreds of dollars. Buying enough devices would require tens of thousands of dollars, for just a single temporary attack. Resources an enthusiast would not have. Neither would independent hackers, as good as they may be. The CIA does, but again we can’t stand against someone who has a vast amount of resources.
For an attacker or for the CIA, it would be cheaper to get on a plane and break into our factory to steal certificates. Or somehow breach into our dedicated secured network in our factory. Once he has those certificates, all he’d have to do is simulate the devices. And that would be just buying one device, taking the board out and copying the code from it. All that is left to do for an attacker to do is just run thousands of instances of it, with thousands of stolen certificates. It was literally something me and Mike know how to do with our powers combined. Mike also knew how to make sure copying our code would require a 1 million dollar microscope.
Someone will eventually break in. Somehow, in a matter we can not think of. We need to ensure that when he succeeds, a minimal harm would be done. Or at the very least no harm to our customers. We started with the basics, of making sure all of our data is encrypted at rest. All of our communications with our clients were encrypted in transit as well.
But maybe with the help of a disgruntled employee, decryption might be possible. Or maybe one day we’ll just make a mistake, such as leaving an S3 bucket publicly exposed like almost every other company in the world. When that happens, an attacker will be able to derive some interesting insights out of our exposed data. In a very highly unlikely scenario, that would be a shame.
But what an attacker won’t be able to do, is to trace back the data to our customers who produce the data. We made sure all of our data is anonymized to begin with. Our customers’ identifiers, their household ID, device ID, mobile ID, none of which contained any personal identifiable information (PII). And our Events, the messages communicated between all of our applications, do not contain any PII as well. So even if some will eavesdrop into those, good luck with doing something with it.
The only way to transverse the data back to our customers, is to breach into our user pool. Which no application or customer can be granted access to. And no application needs it, because all of our applications, all of our Events and all our databases work solely with uniquely generated IDs. All copied between chained Events by our Event Handler framework.
There is another threat this anonymization protects us from, a threat of unintentional wrongdoing. One day our CEO had a concern. A customer might permanently give away a container to another customer, creating a duplicated ownership. His solution was to immediately delete the previous ownership. Instinctively speaking, it makes sense. But these were exactly the kind of solutions our security measures prevented from being executed. We made sure no one can code a Flow that entangles two customers together. As each is owned by another, we can not allow one customer to access another’s.
“I’m sorry, your solution is not possible. Our system architecture is built to make sure no customer can do anything to mutate another customer’s data. All of our customers are all completely isolated from one another”. Because even if our engineers can not do that, it would be much harder for an attacker to do so.
Obviously I was asked to change the architecture. It was called to have a backbone. “I will not. That’s exactly what our architecture is for, to place Restrictions on ourselves and to prevent us from wrongdoing. And I will definitely not change it for some obscured scenario that hasn’t happened yet, that has zero effect on our customers”. Of course no CEO likes to hear “we can’t”, so we came up with another technical solution that obeys our Restrictions. To be done some day in the future, years from now.
Even if an attacker were to break into our Event Store and download the entire company’s history, nothing would directly hurt our customers. An attacker might be able to say that “people from Middle-earth sure do need to go on a diet!”, but no attacker would be able to say “Jack Donaghy from Manhattan sure does eat a lot of strawberries!”.
But what if that very same Jack Donaghy, would like to stop Silo from knowing so? He’d like to permanently stop using Silo and have all of his personal information and data removed. Supposedly, in order to follow his request, we need to code a process that will go to each and everyone of our databases and physically remove specific data from it. Lest we forget the number of databases and amount of data are both expected to grow.
And here’s the catch. According to GDPR regulations, there’s an alternative. We don’t have to delete the data, but only to make sure it can not be traced back to its owners. For Silo to no longer go back from “Strawberries” to Jack Donaghy. In that case, all we have to do is sever the connection between Jack Donaghy’s internal IDs and his PII. Which can be done manually in exactly one secured spot. No need to automate anything, because we always have 90 days to respond to such a request.
Anonymizing everything in advance was an eventually beneficial effort, as it has saved a lot of future effort. Not only were we protecting ourselves better, we also were protecting our customers better. We were fulfilling the highest requirements and standards of GDPR. And we knew so, because we passed an inspection of a world-class expert on the matter.
But have we done something else wrong? On this, in the next and closing chapter of this series.