By most definitions, generating large amounts of traffic to your website is a sign of success. However, it also presents a unique set of business challenges. Tools and techniques for handling high volumes are prone to break down as variables like user activity, ad traffic and e-commerce dollars grow beyond the capabilities of existing systems. And as a business grows, so do its vulnerabilities to cyber attack.
That’s the challenge facing Internet Brands, an El Segundo-based survivor of the dot-com era. The company launched in 1998 as online auto marketplace CarsDirect, rebranding as Internet Brands in 2006 as its portfolio of websites grew. Since then, it has made almost 200 acquisitions and sprawled into new verticals. The company now runs some 80-odd websites, from coupon cutting hubs to hunting forums to legal directories. In addition to CarsDirect, its properties include brands like Lawyers.com, Wikitravel and, with about 80 million monthly visitors alone, consumer health resource WebMD.
In total, Internet Brands serves an audience of more than 250 million each month. As CTO and 14-year Internet Brands veteran Joe Rosenblum put it: “We do a lot of different things for a lot of different people in a lot of different ways.”
DIY Scalability
Internet Brands came of age before AWS, Azure and Google Cloud Platform turned server scalability into a pay-as-you-go service. To this day, the company runs its sites on a proprietary grid computing system — a precursor to cloud technology wherein a decentralized group of computers break large tasks into smaller chunks — which it calls the Toaster Environment.
The system offers what Rosenblum called “the ilities”: scalability, operability and maintainability, along with the security that comes from common technology underlying each application. Simply put, the Toaster Environment performs tasks many developers now take for granted.
“I actually got that idea from... Facebook, which was one of the only other companies dealing with similar scaling issues at the time.”
“Ten years ago, we’d be getting 25,000 database queries per second on some of our sites,” Rosenblum said. “We’d go to Oracle, the vendor for [database management system] MySQL, and they couldn’t help us. We’d have to solve those kinds of problems on our own.”
Rosenblum’s solution for scaling a MySQL database involved ultra-fast, high-bandwidth PCIe SSD cards for storage, separating read traffic from write traffic and imposing limits on the amount of data pulled in by any given query.
“I actually got that idea from the team at Facebook, which was one of the only other companies dealing with similar scaling issues at the time,” Rosenblum said. “They put in a hard limit that said no query could return more than a thousand rows, and it solved all these weird issues you might run into in your coding.”
Many of the solutions Rosenblum’s team once built from scratch are now easily accessible via a cloud vendor near you. But Rosenblum said he continues to learn from Internet Brands’ acquisitions, highlighting his team’s ongoing migration from older Toaster architecture to an environment centered on containerized orchestration systems like Kubernetes, as well as their adoption — when appropriate — of cloud technology.
SECURING THE SYSTEM
A company that draws as much traffic as Internet Brands is bound to gain the attention of the wrong kind of visitor. While the general advance of digital tech has dealt with many of the more serious scalability issues Internet Brands has faced in the past, cybersecurity remains an ongoing challenge. Rosenblum recalled a statistic he once presented to his board of directors: At one point, Internet Brands’ network was receiving at least 64,000 automated attacks every single day.
“64,000 was all I could keep in my buffer — so it was a lot more,” Rosenblum said. “And that was years ago. It’s much worse now.”
Over the years, the company has experienced intense distributed denial of service attacks, including one in 2010 that Rosenblum said took all of Internet Brands’ sites and applications down for a full 36 hours. The team has worked with the FBI to help identify the origins of ransomware attacks and even partnered with Linux kernel creator Linus Torvalds in 2016 to patch the Dirty COW exploit, a privilege escalation vulnerability that Internet Brands’ tech team first discovered using its proprietary security technology.
“Having our own infrastructure-as-a-service layer gives us a lot of fine-grain control over the security of the system.”
“Having our own infrastructure-as-a-service layer gives us a lot of fine-grain control over the security of the system, and we have high visibility because there’s a lot of predictability in the way we build it,” Rosenblum said.
That predictability is predicated on a degree of consistency in the system, which Internet Brands achieves by migrating newly acquired websites, communities and applications into its Toaster Environment. Even with the best firewalls, internal controls and bug bounty programs, Rosenblum said a system comprised of an administrator’s ad-hoc fixes and tweaks made over time is impossible to properly monitor.
Part of his acquisition game plan, he said, involves exploring a company’s security posture pre-acquisition, learning about past breaches and surveying potential security risks.
During this process, Rosenblum said he’s skeptical of any company that says it’s never had a security breach.
“They’ve probably been hacked and they don’t know it,” Rosenblum said. “It’s not if, it’s when. And in my paranoid worldview, I’m always thinking about what I can do to minimize and mitigate when something happens. What can I be doing proactively?”
