In the decade between 1990, when Tim Berners-Lee invented the worldwide web, and 2000, the number of websites on the internet ballooned past 17 million.
During the internet’s first few years, digital ads were bought and sold much like ads in print publications. Potential advertisers contacted websites — known as “publishers” in the digital advertising world — directly to negotiate the terms of purchasing available ad space. But the stratospheric growth of the internet quickly outpaced human teams’ abilities to keep up, with millions of websites and their tantalizing legions of visitors still out of reach.
Ad agencies and adtech startups quickly stepped in to close the gap. Starting around 1995, advertisers could purchase from “ad networks” that aggregated ad space from websites into groups thought to appeal to similar visitor demographics.
By 2005, another type of programmatic service offered by “ad exchanges” was using a system of automated auctions to sell available ad space. Auctions are triggered each time a user loads a page on a website, sending information about the site, the ad space and the user to the ad exchange, which disseminates it to advertisers and announces the auction. The winning bidder’s ad is displayed on the user’s browser by the time the page finishes loading.
The Pros and Cons of Programmatic Advertising
- PRO: Allows companies to sell ads at a greater volume.
- PRO Small sites without sales teams can still make money on ad exchanges.
- PRO: Advertisers can direct ads at users who fit their consumer demographics and pay for performance.
- CON: Automation increases the risk of fraud within digital advertising.
- CON: Complexity within the industry leads to lack of transparency.
- CON: Privacy concerns have led to pushback against third-party cookies.
This type of ad buying and selling — called “real-time bidding,” a method still widely used today — can theoretically place an advertiser’s ad on any website that participates in the exchange, greatly expanding the range of an advertiser’s audience. Because the auctions are automated, running based on preset agreements about the type of ad space advertisers want and how much they’re willing to pay, each individual purchase can be independently negotiated even while the volume of sales dramatically increases.
The Rise of Targeted Ads
Companies operating ad exchanges tout these qualities as advantages over traditional ad sales. And the increased volume of sales is undeniable: real-time bidding is often compared to the stock market, but an average day of trading on the New York Stock Exchange sees 4.5 billion transactions, while a single ad-tech company, Fiksu, reportedly sold 32 billion ad spaces daily.
Programmatic ad sales also had a democratizing effect. Unknown websites with no sales teams and little chance of attracting advertisers before can now easily sell their ad spaces on exchanges alongside well-known publishers and make some money. Advertisers liked it too, because they were reaching more potential customers.
Proponents say programmatic advertising also means good prices for both publishers and advertisers. By participating in auctions, publishers know they’re getting the optimal price for their ad spaces, at least among the ad exchange’s advertisers. And advertisers can make purchasing decisions on a case-by-case basis and not waste money showing ads to users who are unlikely to buy their products.
For instance, an advertiser selling women’s shoes may tell its ad exchange to submit bids only if the user is a woman, and to bid more if the user has previously visited the advertiser’s website. This type of ad transaction, where advertisers decide whether to purchase ad space — and how much to purchase it for — based on information about the user is known as “targeted advertising.” It’s what makes it possible for users to see “relevant” ads based on their personal characteristics, location and browsing history.
Another selling point of programmatic ads is the idea that advertisers can pay based on how well an ad performs. Traditionally, advertisers could only be sure ad campaigns were successful if the company saw a significant increase in sales shortly after. Programmatic advertising introduced the idea of paying for ads only if they are successful. With those types of ads — called “pay per click” (PPC) or, more broadly, “pay for performance” (P4P) ads — the advertiser only has to pay if the user performs a desired action, such as clicking on the ad or making a purchase from the advertiser.
Attention Is Difficult to Quantify
But the transition to automated ad sales wasn’t entirely straightforward. Traditionally, advertisers had people assess potential ad space to determine whether to buy it and how much it’s worth. Going from that to real-time bidding required a way to quantify the quality of ad spaces so the time-intensive assessment step could be removed.
In his book Subprime Attention Crisis, technology policy researcher Tim Hwang reports that this required a method of measuring and quantifying users’ attention, which he calls the “commodification of attention.” Real-time bidding required standardizations so that “the amorphous, shapeless concept of attention [was] transformed into discrete, comparable pieces that can be captured, priced, and sold,” Hwang writes.
The question is whether it’s possible to do it well, or at all.
In 2004, Hwang writes, the Interactive Advertising Bureau, an organization made up of digital advertising industry leaders, published a set of standards defining how to tell whether an ad was consumed by a user. There were a lot of different definitions to decide between. One way of measuring attention, Hwang explains, is just based on page load: if a particular ad is on the user’s page when it finishes loading, then the user could be considered to have consumed the ad. But what if the ad is at the bottom of the page and the user never scrolls down?
How can one be sure that the user is actually looking at the ad, or registering what they are seeing?
A better definition is based on the length of time an ad is on the user’s screen. For instance, an ad is considered consumed by a user if more than 50 percent of its pixels are viewable on screen for more than one continuous second — which is the current IAB standard, something the organization calls a “viewable impression.” But even for this definition, how can one be sure that the user is actually looking at the ad, or registering what they are seeing?
Hwang argues that, although the industry benefits from having an easily measurable definition of ad “impressions,” it’s forcing standardization on something that resists easy quantification. He predicts that the industry’s shaky foundations has created a bubble that is due to burst at any time.
Whether that comes to pass is yet to be seen, but the speed and scale of real-time bidding did give rise to an unwanted phenomenon: ad fraud.
How Can Companies Weed Out Ad Fraud?
Ad fraud can be prosecuted as a crime — such as a case in 2015, when a man was extradited from Estonia to the United States to face charges of wire fraud — although more often it’s the subject of civil lawsuits.
Because it’s easy for websites to sell ad space on ad exchanges, some people realized they didn’t actually need to attract audiences to earn ad revenue — instead, they could use bots or click farms. Augustine Fou, an ad-fraud consultant, said perpetrators of ad fraud can list thousands of sites on each ad exchange. Botnets are used to repeatedly load webpages and earn money from ad impressions, or click on ads to get pay-per-click revenue.
“Now we have millions and millions of sites that I don’t know that any humans go to,” Fou said. “And because of this automation, it’s allowed bad guys to also scale the fraud.”
Juniper Research, a market research company, estimated that ad fraud cost the industry $42 billion in 2019. Since the global revenue from programmatic ads in the same year was $106 billion, that means an estimated 40 percent of programmatic ad money spent was spent on fraudulent sites.
That may seem unbelievably high, but consider this: Ads on Google Ads, a large real-time bidding ad exchange, have an average click-through rate of 0.46 percent in 2018, which is about one click for every 200 ads displayed. How many webpages do you load every day, how many ads are on each one — and when was the last time you clicked on an ad?
Although ad fraud has been a known problem within the industry for a long time, it isn’t easy to address. The director of fraud prevention at marketing analytics company Adjust, Andreas Naumann, said in a previous interview with Built In that many in the industry were resistant to the idea because there are a lot of jobs on the line.
“If the marketer chose to look more carefully at their own analytics, they can see so many things that are just out of whack and don’t make any sense.”
Fou said that is because ad exchanges actually stand to benefit from fraudulent publishers operating on their exchanges, due to the way in which deals between ad exchanges and advertisers are usually structured.
“When a buyer just says, ‘OK, I’m going to buy from the exchange, here is $10 million, go spend it for me,’ the exchange will find every possible means of spending it all,” Fou said. “So [exchanges] don’t mind.... It’ll help them make money faster, because their profits are tied to the volumes that run through their exchange.”
There are also tools available for companies to measure how much fraud is occurring within an exchange, such as DoubleVerify, but Fou said existing tools are unreliable and are often used to appease customers.
“Sometimes the marketers don’t want to know there’s fraud,” Fou said. “They tend to buy the fraud detection technology services, but unfortunately those services are a black box. They tell you a number — ‘It’s 1 percent bots’ — but they don’t explain how they measured it, so you can’t determine if they measured it correctly or not.”
He said that finding and stopping at least some ad fraud isn’t actually that difficult and doesn’t require fancy technology — a lot of the time, it’s pretty obvious. He’s helped clients look at their own analytics and found instances where 100 percent of a certain website’s traffic originates from Android 10, instead of the usual mix of devices from real users. Other indications of bot traffic that he’s seen are constant levels of traffic during all hours of the day.
“If the marketer chose to look more carefully at their own analytics, they can see so many things that are just out of whack and don’t make any sense,” Fou said. “Then all you have to do is say, ‘OK, here’s this site that’s cheating. Let’s turn it off.’ ... You don’t need anything special, you just need to look.”
Real-Time Bidding Isn’t the Only Game in Town
In May of 2020, the Incorporated Society of British Advertisers released a study examining the programmatic advertising supply chain that expressed concerns about the industry’s operations.
The study found that, although programmatic advertising is premised on the idea of programmatic’s advantage over traditional advertising due to being more data driven and targeted, it was almost impossible for advertisers to get the data to understand how their money was being spent within the exchanges — let alone to measure their ads’ success or failure.
Another complaint of the study was that publishers only received 51 percent of advertiser spending, with the rest going to various fees — including 15 percent of total advertiser spending that researchers were unable to account for. In ISBA’s announcement of the study, the organization expressed concerns over “the depth of the supply chain’s lack of organisation and complexity,” and the report recommended the industry work “urgently” to reduce its amount of complexity and improve its lack of transparency and standardization.
Perhaps publishers and advertisers’ frustrations have contributed to the recent decline of real-time bidding compared to other types of programmatic advertising. Aside from real-time bidding, there are two other types of programmatic ad sales: “preferred deals” and “automated guaranteed” deals. These methods are “programmatic” because deals are still automated, but they give publishers more control over who they are selling ad space to, and advertisers more control over which publishers they are buying from.
“The method of buying is not as important as who you’re buying it from.”
Publishers often sell ads using a combination of all these methods, but there has been a recent shift away from real-time bidding toward other programmatic methods. In 2019, market-research company eMarketer reported that advertisers have been spending less on real-time bidding compared to other types of programmatic advertising year over year, with real-time bidding down to 38 percent of all programmatic advertising spending.
Within real-time bidding, publishers have also been moving away from “open exchanges” where buyers can be any advertiser on the exchange and to “private marketplaces” that only have advertisers the publisher has vetted. Private marketplaces have also earned larger percentages of sales over open exchanges year over year.
For advertisers, Fou said knowing their publishers is more important than how they buy ads.
“The method of buying is not as important as who you’re buying it from,” he said. “Because if you buy it direct but you end up buying it from fake sites, you’re still going to get exposed to fraud.”
Instead, he recommended advertisers use publisher allowlists, so they can be sure all websites their ads appear on are known and vetted, regardless of purchasing method.
How Will the Industry Adapt to a Post-Third-Party Cookie World?
Another problem facing programmatic advertising of late is the backlash against the practice of using cookies to gather information about users’ internet habits. Information about user behavior is valuable in programmatic advertising because advertisers rely heavily on user data to decide whether to purchase ad space being shown to the user. Data management companies that provide this information often use cookies — often referred to as “third-party cookies” — to track users’ behavior across the internet.
In response to concerns over third-party cookies, the European Union enacted General Data Protection Regulation in 2016 and the state of California enacted the California Consumer Privacy Act in 2018, both of which regulate and constrain how companies are allowed to collect user information. Browsers, which have a lot of power in facilitating how cookies operate, have also weighed in. Firefox and Safari browsers currently block third-party cookies by default, and Chrome is planning on phasing out third-party cookies by 2022. Within the industry, companies are looking for ways to still track user behavior without using third-party cookies, per se.
User information may be important to programmatic advertising, but Fou argued that the data collected by third-party cookies isn’t great to begin with. He said a lot of information third-party cookies claim to know about users is derived from what websites they’ve visited — but the accuracy of these assumptions are questionable.
“A lot of times a certain cookie is marked as both male and female, because they literally don’t know — they can’t derive it properly,” Fou said. “If the data that you’re buying is not going to get you better than the 50 percent [chance], then you might as well not pay the extra money for that. And what do you derive about a person when they go to CNN or Walmart? What do you derive that could be accurate?”
Still, Vice President Brent Carter at OpenX, an ad exchange, cited restrictions around third-party cookies as one of the biggest challenges facing the industry. Under GDPR, users must be allowed to opt out of having their data collected — in the United States, you may already be familiar with websites asking whether you accept cookies, often with only one choice saying you accept. But in the European Union, websites are required to allow users to decline and continue to use the site.
In 2018, Ster, the exclusive sales house for the public broadcasting company in the Netherlands, saw 90 percent of its users opt out of third-party cookies within one week of implementing changes to be compliant with GDPR. Ster was using Google Ad Manager, so the sharp loss of user data meant advertisers on Google’s ad exchange were suddenly a lot less interested in purchasing ad space from the publisher.
“We’re probably the most expensive publisher in Holland, without using data.”
“We had a revenue drop of roughly 90 percent within one week,” said Tom van Bentheim, Ster’s former head of programmatic advertising.
In response, van Bentheim’s team decided to build a way to sell ad space themselves, without going through any ad exchange and without collecting user information. A developer created a working prototype over a weekend, and soon, several advertisers had signed on to test it.
“We had, of course, no idea what the cost-per-impression price should be, so we just started with a 50 percent discount of our regular fare,” van Bentheim said. “And within one or two months, we made 100,000 euros.”
The prototype was a success, and Ster soon partnered with a tech company to build a fully functional ad server that didn’t rely on user data. Van Bentheim, who is now Ster’s manager of digital strategy, said his experience contradicted the industry’s conventional wisdom that cost-per-impression earnings are higher for publishers that have access to user data.
“We’re probably the most expensive publisher in Holland, without using data,” he said.
Two years after Ster began selling ad space using its new programmatic method, van Bentheim doesn’t miss the ad exchanges — instead, Ster is evangelizing the system to others.
He doesn’t see the current predominant way of selling ad space — with more data on users, in real time — as better. In fact, the system’s frequent mistakes often results in laughable mistakes.
“I booked a holiday two weeks ago, and that holiday is still haunting me all over the web,” van Bentheim said. “I’m already at home for seven days, but I still get airline offers every single minute on every goddamn website.”
