Earlier this summer, we published a comprehensive Guide to User Data Security detailing steps to harden a server and secure applications. We provisioned a couple Linode servers and hardened them to the guides specifications to stand by our claim. We shared the IP addresses and proposed a challenge.
Github: https://github.com/inversoft/2016-security-scripts
Hack This: https://hackthis.inversoft.com
We dared anyone to hack our database. To add incentive, we offered a fully loaded MacBook Pro as a reward.
Challenge accepted.
Polynome successfully breached our security and have taken the time to publish a thorough post detailing how they did it. They walk you through their thought process, outline false attempts and share the hack that ultimately proved successful.
Breaking into a Hardened Server Via the Back Door
Our account of the events leading up to the hack.
Before promoting the contest we reviewed and verified all necessary security precautions were in place on the target systems. We discussed all possible attack vectors and felt confident that the measures we had employed would keep all attempts to breach our systems at bay.
We moved forward, launched the Security Guide and prompted the contest. Weeks of unsuccessful attempts bolstered our confidence. As Polynome said, “Their hardening guide was and remains correct; there was no way we were getting through the front door of their servers…”
The chink in the armor.
Prior to the successful hack we had been preparing the release of some new cloud offerings. In support of this effort our production server required an API key to request new nodes on our cloud provider.
Admittedly our production system did not have the same rigorous security measures in place as the target systems that we used to taunt “would be” hackers.
The team at Polynome was able to identify our weakest link and exploit it, essentially bypassing all intended security measures on the target system. We had assembled a small army to guard the front door and left the back door unprotected. Once Polynome obtained shell access to our production system through a known ElasticSearch vulnerability and found the Linode API key, it was only a matter of time.
Because the penetration came from our own production server the exploration by the Polynome team went mostly undetected. The final phase of the attack required them to clone the cloud instance and mount it on a newly created node and swap private IP addresses to gain database access through the firewall. While we quickly detected these events it was not immediately clear who or what was triggering them.
If you’re familiar with Linode API keys you know that they act as a proxy for a user. The API key privileges are synonymous with the owning user of that key. The actions taken by the Polynome team with our API key appeared to be coming from our own server by one of our developers.
The Polynome team knew they had to move quickly once they began the sequence of destructive steps. To claim success they needed to dump the Passport database undetected.
They did in fact move quickly. While we were feverishly working to identify who or what was causing these events we received a note from the Polynome team. It was over, they had succeeded.
Lessons learned.
We learned a few things from this exercise.
First. We had not applied the same set of rules we touted in our guide to our own product system. Duh.
Second. Our production system was not a good place for product demo instances. ElasticSearch was running as part of a product demo and was not properly firewalled. “It’s a demo, who cares if it gets hacked, there is no customer data on that search index.” Famous last words.
Third. The principle of limiting privilege was not applied to the API key we kept to create cloud nodes for our cloud offerings.
The irony of course is that we already knew these things. I would wager that in most successful security breaches, the necessary measures were already thought of, discussed and then not implemented for a variety of reasons.
The same reasons companies don’t test their backup restore procedures until they lose critical data is the same reason many wait until they have been hacked to shore up their security.
Job well done.
We want to thank Polynome for the time and effort they put into the contest and the ethical way in which they handled themselves. Congratulations on a job well done. We hope you enjoy your MacBook Pro!
About Polynome
Polynome is a consulting firm providing rapid, high-quality design and implementation of software projects with a focus on scalability and security. Our expertise ranges from mobile and web applications to distributed server systems. Our team has decades of experience building secure applications for Microsoft, Amazon and numerous successful startups.
Join the dicussion on HackerNews: https://news.ycombinator.com/item?id=12361318