You are about to read the horror story behind the creation of Knox. Dropping a few names would make the story more buzzworthy. But legally binding NDA, possible unpleasant consequences for me, yadda, yadda, yadda… So, let’s call the business unit of that Fortune 500 company BU XXX instead of man, this is redacted!.
Once upon a time in one of the IT departments of BU XXX, employees and contractors were working hard on a web application. Nothing fancy, a simple in house app to help the company sales teams. Things started to become tense because of the many consecutive scope evolutions. And tension built up until it was so unhealthy that half of the contractors left the project. New guys had to be hired; including me.
Eventually, we shipped the project with a reasonable delay and most of the final scope. But, just when we were about to celebrate, the shit hit the fan. So fast and so hard that the entire room was covered with an extra thick layer of the smelliest kind.
One of the (ex?) team members decided to hurt the company and the management as badly as he could. Using the external services used in the BU XXX. I don’t know who did that; and I’m not sure if anyone does since the case was not closed the last time I had news. But I don’t understand how someone can get angry enough to go this far. Maybe he just got carried away and could not stop. Nevertheless, I admire the creativity.
How did he obtain the keys and secrets he used? We guessed configuration files carelessly pushed in the git repo and random HipChat messages (the slack of 2014). But maybe just emails or direct access to the accounts. This last one would point to one of the leads and would be super bad.
Obviously, he started by dropping the PostgreSQL databases on AWS RDS. Ok, no big deal. We could always restore from our backups. Except we could not. Because the backups were stored in AWS S3 and he also deleted them.
What we learned a couple of months later is that he went further. Before flushing the system, he made a dump of the databases and sold (or gave away?) the company users data - including credit card data - on a dark web “marketplace”. That was a nasty move.
With the AWS keys, he started the biggest EC2 instances he could to mine crypto. And racked a six figures bill in AWS costs. I don‘t even think this one is still possible today. Of course, if you are a small company, I guess that AWS would lock your account before you get to that point. But given the normal use of BU XXX, it took time.
He used a Mailgun key hardcoded in the Webpack configuration to send millions of spammy emails. The kind talking about Viagra and Nigerian princes. Why? Well, the cost of the campaign itself was not that high. But after this massive campaign, the reputation of the domain name was so bad that almost all transactional emails would end up in the spam folder. That one was really not nice for the rest of the team.
He tweaked the server configuration so every route would be a 301 redirection to a famous adult website if you were visiting as a Google bot. And because human visitors were not affected, it took some time to figure that one out. Google had the time to crawl and index most of the website. As you can guess, a good fat SEO penalty followed.
In the end, the consequences were mitigated as best as possible. All the keys were reset. One dev had a recent dump of the production database that could be restored. And the sysadmin team worked the entire weekend to put back everything online. Because, of course, it happened on a Friday. Eventually, the only guy who was fired was the first one who pushed the .env file in the git repo. This is particularly unfair since the keys could also be found on HipChat history and everybody saw and used that file all the time.
For everyone involved in that mess, it was a wakeup call. I was not a primary suspect, but I have been questioned and it was not a pleasant experience. Since that moment, I noticed how badly the companies are protecting themselves from the damages that individuals with accesses can cause. Intentionally or not. In more than half of the projects I saw as a contractor, there were passwords, keys and configurations in slack messages, in emails or even pushed in code repositories. This is madness.
When you have 50 days to spend on a project, I understand that you cannot afford to spend 5 days making sure that security is top-notch. And that’s why Knox is there. It’s ready in minutes, dirt cheap, and keep the critical secrets out of reach of potential inside or outside threats.
Knox is not a silver bullet and you need more than that. But this is a good starting point. So, give it a try, it’s free for your first project.
By the way, if you have already pushed secrets in your git repo, it’s not too late. You can still remove the file from the history of the repo. We have a small tutorial there.