Inside the Luxor Casino in Las Vegas, 70 engineers from CircleCI hunch over glowing laptop screens cracking security measures as the ominous digital tones of Aphex Twin filter through the speakers.
Working in pairs, the engineers uncover default system passwords, recreate the Facebook image hack in which hackers imported malicious code through Messenger, and hunt for hidden txt files containing passwords. With each completed task, they earn points and receive a glimpse into the world of hackers, where finding any small mistake in the code is a reward — and a hole to exploit.
This is what CircleCI’s security manager Tad Whitaker wants the engineers to experience. Whitaker organized the event during the company’s July engineering all-hands to reinforce basic security principles. But more importantly, he wanted the engineers to feel the urgency behind them.
“Everybody learns things better by doing rather than being told,” Whitaker said. “Experiencing the aha moment is always better than hearing about from somebody else.”
Here’s how Whitaker and CircleCI built its own DEF CON capture the flag-style competition:
Getting devs excited about cybersecurity and secure code
- CircleCI modeled its security training around the DEF CON capture the flag concept. Working in teams, engineers earned points for every security module they completed.
- Engineers were partnered together based on their experience levels, with senior employees working with junior employees. This evened the teams and created opportunities for engineers to teach each other.
- CircleCI built its training around basic best practices to reinforce the fundamentals and get employees to think about the exercise in the context of their jobs.
- Whitaker selected 10 training modules that could be completed quickly. Breaking it into bite-size chunks made the information more digestible and kept teams engaged.
- In one module, CircleCI engineers recreated a Facebook image hack that had made the news. Experiences like those made the security risks and solutions more tangible.
Security principles require constant honing
Writing secure code requires special insight into how software works and how someone might manipulate it.
While CircleCI deploys a dedicated security team, invests in the latest security software and tests its products using third-party hackers, nothing is more critical than maintaining a secure codebase. It’s like locking your front door or keeping your car windows shut, Whitaker said.
It’s a skill every engineer learns early on, but like locking your door, it can be easy to slip up. Meanwhile, keeping up with the latest best practices can be challenging.
“If you want to go out and learn about security concepts, you can, but it’s disorganized and there’s a steep on-ramp,” Whitaker said. “There’s not a good bridge between learning a new tool and how to use it securely... It could wind up biting you in the rear end.”
That’s what inspired CircleCI to launch its first secure code training event. When Whitaker approached CTO Rob Zuber with the plan, Zuber had only one request — he wanted it to feel urgent.
“Experiencing the aha moment is always better than hearing about it from somebody else.”
Zuber had once participated in a security challenge at Google that uncovered a vulnerability in his own web application that could be used to leak his database. Mortified, he immediately went home to fix it. That was the experience he wanted the engineers to have in the training, Whitaker said.
Breaking security concepts into applicable chunks
With that in mind, Whitaker searched for training modules that broke security concepts into relatable chunks and felt applicable to their work. He settled on a program provided by a Hungarian security firm called Avatao whose security training modules emphasized hands-on exercises.
Whitaker narrowed the list down to 10 modules that centered around basic security concepts like input validation testing, cross-site scripting, cross-site request forgeries, broken access control, Docker build secrets and more.
While he knew some team members might already know these fundamentals, Whitaker wanted the exercises to serve as a foundation for newer employees and a refresher for more experienced ones.
“I wanted everybody to feel comfortable talking about it with their colleagues and have a good time.”
“We wanted to make sure everybody in our engineering department understood the basics,” Whitaker said. “While 20-to-40 percent of our engineers may have seen these things, putting it in front of them again allowed them to put it in the context of their work as CircleCI employees.”
Meanwhile, the modules consisted of topics that would be easy to discuss between partners of all experience levels.
“I didn’t want anybody to feel imposter syndrome around any of this,” Whitaker said. “I wanted everybody to feel comfortable talking about it with their colleagues and have a good time.”
But simply completing modules alone didn’t create the urgency Zuber desired. CircleCI wanted to raise the stakes.
Raising the stakes through competition
Each year, some of the world’s most skilled hackers gather in a Las Vegas casino conference room to compete against each other in DEF CON’s Black Hat’s Capture the Flag competition. Throughout the competition, 10 teams simultaneously attack their opponents’ websites and defend their own in pursuit of digital flags.
That’s the experience Whitaker wanted to simulate. With CircleCI hosting its engineer all-hands in the same city, Whitaker felt it offered the perfect opportunity to run its own version of the event for security training.
“I didn’t want learning about security to feel like a job,” Whitaker said. “I wanted them to think that it was fun and interesting.”
CircleCI partnered with a vendor to incorporate the 10 training modules into a cohesive cybersecurity competition. Instead of attacking websites, they built the competition around who could start and complete the most security modules in three hours.
To even the odds, they decided to pair engineers together based on alternating skill levels. That meant a principal engineer would work with a junior engineer, and a second-level engineer would team up with a fifth level.
“While 20-to-40 percent of our engineers may have seen these things, putting it in front of them again allowed them to put it in the context of their work as CircleCI employees.”
The partnerships had the added benefit of combining employees who might not work together often, and encouraged more experienced employees to share their knowledge.
As a final touch, CircleCI set up a scoreboard in its Luxor conference room.
Change up the routine to spark ‘Aha’ moments
On the day of the event, Whitaker addressed the team of engineers gathered at the conference room to set the tone.
“We build and secure things bad people want,” Whitaker told the engineers.
Through music and setting, Whitaker sought to evoke a scene right out of a hacker movie to get the engineers into a different mindset. With that, they got to work. A team from Avatao and CircleCI’s security engineers roamed the room ready to offer assistance for any teams that got stuck, but mostly, the engineers fell into a hushed fervor attacking the modules.
“I just wanted them to have four or five ‘aha’ moments.”
“I had these nightmare visions of hands firing up all over the room, saying, ‘I can’t log in,’ or ‘I’m not sure what to do,’” Whitaker said. “To my huge relief, no hands flew up and everybody was just fingers and faces down in their keyboards talking quietly and knocking out puzzles.”
In one module, engineers had to fix an input validation code that allowed someone to enter a command into a website that would enable them to drop the company’s database. In another module, they got to recreate the Facebook image hack, giving them hands-on insight into how a hacker might probe a website for weak spots. In a third, they had to crack a web access password: the credentials were “admin,” “admin” — a common combination for newly shipped software that many users don’t change.
Each exercise mixed a series of obvious discoveries with a sly twist that helped engineers think about their work and secure coding differently.
“I just wanted them to have four or five ‘aha’ moments,” Whitaker said.
Finding ways to keep security cOncepts front of mind year-round
While it’s tough to measure the impact the event will have on the team’s code base, Whitaker said the positive feedback suggests it resonated with his team.
“People just genuinely had a good time playing with their colleagues on some coding tutorials and getting to be bad guys for a little bit,” Whitaker said.
“The best thing that you can do is to constantly talk about security and to constantly surface it.”
Next year, CircleCI is looking for ways to continue the training throughout the year. One idea the security team had was to introduce recurring optional modules that teams can complete for additional points or prizes, Whitaker said.
Ultimately, when it comes to security, keeping it front of mind at all times is key.
“The best thing that you can do is to constantly talk about security and to constantly surface it,” Whitaker said. “The next big step is to get people doing security things, and that’s what the capture the flag accomplished.”