The coronavirus pandemic has created a major business opportunity for online retailers like Amazon. As of mid-2019, online shopping only accounted for about 12 percent of retail revenues in the United States — but now, Amazon is seeing a “significant increase in demand,” the company reports. Amid widespread layoffs, Amazon is hiring 100,000 new order fulfillment and delivery workers in the U.S. alone.
That’s because we’re all more reliant on Amazon’s quick, minimal-contact deliveries as we attempt social distancing — which means we’re also more reliant on Amazon product reviews. But they’re not all trustworthy, as ReviewMeta demonstrates.
You may have heard of ReviewMeta on Planet Money, or stumbled on it after buying a shoddy yet highly rated dongle. If you haven’t, the company’s software — available through a website, an app and a browser plug-in — scrapes Amazon product pages, scanning the reviews for suspicious patterns.
Not that the online retailer has a major problem with fake reviews.
“I shop on Amazon and for the most part trust the reviews,” Tommy Noonan, chief technology officer at ReviewMeta, told BuiltIn.
He doesn’t even like the term “fake reviews,” which suggests a strict binary between real and fake reviews — instead, he likes to place reviews on a spectrum, from high-quality and trustworthy to low-quality and likely promotional.
Noonan estimates that only about 7 to 11 percent of Amazon products have a pattern of low-quality reviews. That estimate might skew high, too, since people use ReviewMeta on the site’s sketchiest-looking wares.
More than it’s a dig at Amazon, ReviewMeta is a way of highlighting trustworthy product reviews — which have always been hard to ensure.
The Other Wild West
Reviews are, of course, bigger than Amazon. Before we had online reviews, we had celebrity endorsements, consumer testimonials and word of mouth. Yelp and TripAdvisor have built entire digital companies on reviews. Like Amazon, Walmart hosts a vast ecosystem of reviews. And that’s not to mention platforms like Influenster, which send people free products in exchange for product reviews, or publications like the Strategist, which specialize in longer-form product recommendations.
This world of reviews is complicated though; Noonan compares it to the Wild West. The warring factions, on this frontier, are consumers and marketers. Consumers want authentic reviews; marketers, meanwhile, want positive reviews.
The two aren’t mutually exclusive, but most review outlets skew one way or the other. Celebrity endorsements, obviously, skew positive; they’re paid recommendations, though celebrities won’t endorse just anything. Yelp and TripAdvisor, meanwhile, skew authentic, Noonan believes. Yelp, for instance, often filters out reviews from new users, reviews it deems too short to be useful, and hyper-positive or -negative takes.
“The main part of their business is their reviews,” Noonan said, “and so they’re focused on authenticity.”
Why Amazon?
The core of Amazon’s business isn’t reviews — it’s selling products, though Amazon obviously doesn’t ignore authenticity. Once upon a time, for instance, the company allowed sellers to send out free products in exchange for reviews, so long as reviewers disclosed that they’d gotten the product for free.
“Everyone was like, ‘That’s bullshit,’” Noonan said. “‘You got a free product, that’s not unbiased!’”
So recently, Amazon banned that practice.
It continues covertly, though. Sellers often strike deals with consumers through private Facebook groups, wherein a consumer buys a product with their own money, and then, once they positively review it, the seller refunds the price, sometimes adding a little extra on top for the good review.
One woman has gotten tens of thousands of dollars worth of free stuff off Amazon this way. “It’s like Christmas every day,” she told BuzzFeed.
Noonan has noticed other black hat marketing schemes on Amazon’s platform too. Sometimes, people will hijack a well-reviewed colored pencil listing and use it to sell, say, headphones. The headphones look like they have a great star rating, unless you actually read the reviews and realize they’re not for the headphones at all.
Other times, people simply buy reviews in bulk — Noonan estimates that 200 reviews cost 100 dollars. Marketers sometimes use passwords leaked in a data breach to break into Amazon accounts and leave five-star reviews.
Ultimately, these sketchy reviewing tactics are a testament to Amazon’s success, Noonan said — they show it’s a retailer worth defrauding. Amazon’s platform is so vast, and well-reviewed products make so much money on it, that scammers won’t stay away, no matter how much the site cracks down.
So far, though, it has cracked down only cautiously. Amazon mainly regulates reviews by deleting egregiously bad ones, but the company doesn’t want to delete real users’ reviews — that would be hurting authenticity in pursuit of authenticity. So the site errs on the side of caution. You could call it a policy of cautious authenticity. Amazon makes a lot of reviewing data public, like the reviewer’s unique handle and their total body of reviews — then it lets the user decide what’s real and what’s not.
But Amazon reviews have gotten unwieldy — popular products have more reviews than a customer could plausibly read in a day. Or a week.
“There’s products with 27,000 reviews on Amazon,” Noonan noted. “It’s like, ‘Who’s reading this?’”
How ReviewMeta Works
ReviewMeta can process reviews by the thousands, and quickly. At this point, the software has crawled and cached nearly 350 million reviews spanning more than 30 million different product pages, Noonan reported. As it scans, it looks for 12 core tells of low-quality product reviews.
Noonan learned about these tells years before he started ReviewMeta, back when he ran a site called SupplementReviews.com. It was a small, close-knit community of athletes who reviewed workout supplements, and Noonan felt like it was an important antidote to the supplement industry’s ambitious marketing.
“[Manufacturers] just throw around these terms like ‘revolutionary breakthrough’ and ‘muscle building technology,’” Noonan explained. “It's all just the same ingredients.”
Administrating the site, though, Noonan encountered his fair share of sketchy reviews. They stood out clearly from his regular’s reviews. For one, the suspect reviews often came in from brand new users who never returned to the site. For another, they often came in floods on specific days, rather than gradually over time; this pattern suggested they were somehow bought or otherwise incentivized.
Today, ReviewMeta looks for those same tells in Amazon product reviews. The software also flags “brand monogamists” — reviewers that only review products from one brand. It also looks for reviews from unverified purchasers, an Amazon-specific metric that means the user reviewed the product but didn’t purchase it through the site.
The site ultimately churns out a report on the product’s reviews. For each tell, the report answers two different questions: What percentage of this product’s reviews have this tell, and how do those reviews differ from the natural-looking reviews? In other words — without those reviews, would this product look like a total dud?
Finally, ReviewMeta recalculates the average star rating for every product it scans, giving suspicious reviews less weight than the authentic-looking ones. It also gives the reviews a one-word overall rating: “pass,” “warn” or “fail.”
Conceptualizing all these features, Noonan said, came relatively easily after working in product reviews for a decade. The backend was harder, he said, especially database architecture. ReviewMeta caches a product’s reviews on its server, then runs analysis on them, which sounds simple enough. If 100 people use the site at the same time, though, that’s a lot of activity.
“The database is getting huge,” he said. “Setting up the fundamentals to the site was the most challenging.”
It’s a challenge by pre-pandemic standards, at least. Today, even basic things have become temporarily challenging — tracking down toilet paper (don’t hoard!), going to restaurants (you shouldn’t, but you can order in), and figuring out the most ethical ways to shop. Some argue we should shop as much as we comfortably can, to stimulate the economy; others argue that we should only buy essentials, to protect delivery workers’ health.
Whichever way you lean, ReviewMeta can shield consumers from black hat marketers. Though Noonan noted that it’s not perfect — “nothing is ever perfect” — the software makes online shopping a little less like the Wild West.
