A recommendation engine, or recommender system, is a data filtering tool that provides personalized suggestions to users based on their past behavior and preferences. Using machine learning algorithms and statistical analysis, it can predict a person’s wants and needs based on the data they generate, as well as suggest products, content or information they’re likely to find interesting or relevant.
“The goal,” according to Patrick Thompson, director of product at recommendation engine provider Amplitude, “is to get to the point where you’re recommending the right content to the right person at the right time, based off of their previous journey.”
What Is a Recommendation Engine?
A recommendation engine is a tool that uses machine learning to detect patterns in a person’s behavioral data (such as browsing history and past purchases) to suggest specific content, products or information they’re likely to find interesting or relevant.
Recommendation engines are just about everywhere, from video streaming services to e-commerce sites. Some familiar examples include Netflix, which suggests shows and movies a user might like based on their watch history, and Google, which uses a person’s browsing history to rank information and predict what they may search for next.
In a world of information overload, recommendation engines make it easy for consumers to discover products and content they want — and for companies to create personalized experiences that keep those consumers coming back.
How Do Recommendation Engines Work?
Put simply, recommendation engines bring together lots of data and then use machine learning to recommend the “next best action,” Thompson said, and that could be anything from buying a product to clicking on a video.
There are two main categories at play in a recommendation engine — users and items, according to Eugene Medved, an AI developer at recommendation engine provider InData Labs. “The task itself,” he explained, “is all about ranking the items for a specific user by probability of the interaction.”
This is accomplished by a standard order of operations, starting with data gathering.
1. Data Gathering
Data is crucial to how recommendation engines work. Information about a person’s browsing habits, purchase history — and even more personal details like their gender and age — form the building blocks from which patterns are extracted. The more data a recommendation engine has access to, the more effective it will be in making relevant suggestions.
This data typically comes in two forms. One is implicit data, which refers to information about a user’s search history, clicks, purchases and other activities; it’s gathered by a company every time a person uses their site. The other is explicit data, which covers the user’s inputs, such as previous ratings, reviews or comments. (Recommendation engines also use data regarding a person’s age, gender and general interests to identify similar customers.)
Gathering all of this customer data is essential to building a recommendation engine.
2. Data Storage
Once that customer data is gathered, it has to be stored. How and where it’s stored depends on the kind of data that’s been gathered.
In addition to data about the users, companies also store data about the items they provide, whether that be shoes or television shows. This can be anything from price to genre to item type, all of which is used to help determine product similarities and user preferences.
3. Data Analysis
Then, a machine learning system is placed on top of that data, drilling down into it and analyzing it.
Recommendation engines use all kinds of algorithms to analyze data, but the most common one is singular value decomposition, or SVD. This is a mathematical technique that breaks down a matrix into three smaller matrices in an effort to detect patterns and relationships in the data, as well as determine the strength of those patterns and relationships. The goal is to better understand the underlying structure of a large data set so that meaningful information can be extracted.
4. Data Filtering
The final step is filtering the data. This involves rearranging and sectioning off data that will be most relevant to show users based on data analysis. Different mathematical rules and formulas are applied to the data to filter it, and will vary depending on the type of recommendation engine used.
5. Presenting The recommendation
Suggestions made by a recommendation engine can be presented to a user in a variety of ways. They can be served as a message directly on the site (“products similar to this,” or “users also liked,” for example), as a targeted advertisement that comes later on social media or another website, or as part of a personalized marketing message, like an email.
Recommendation engines often sync their recommendation data across all different devices, helping to ensure that users receive consistent and personalized suggestions, regardless of whether they’re using a TV, mobile app or personal computer. Recommendation engines also continuously learn from user interactions and feedback through adaptive learning, refining their recommendations to better align with individual preferences as they evolve.
Types of Recommendation Engines
There are three types of recommendation engines: collaborative filtering, content-based filtering and hybrid filtering. Each type determines how data filtering will be carried out in the recommendation process.
Collaborative Filtering
Collaborative filtering collects and analyzes data on user activities, behavior and preferences in order to predict what a person will like based on their similarity to other users. An advantage to this approach is that it doesn’t require the system to understand the content or products at hand, only the users. But collaborative filtering only works well if it is supported by lots of data on lots of different users.
Content-Based Filtering
Content-based filtering is based on the metadata collected from a single person’s actions and preferences. To make recommendations this way, algorithms create a profile of an individual user, cross reference that with a description of the item or content at hand (genre, product type, and so on) and figure out whether that item or content should be recommended to that individual. While good at creating personalized suggestions, this kind of recommendation engine is limited to whatever information a person has provided in the past.
Hybrid Filtering
Hybrid filtering is a combination of collaborative filtering and content-based filtering, and is designed to improve the accuracy and relevance of their recommendations.
How Recommendation Engines Are Used
Recommendation engines are used across a variety of industries, and have become a popular means of improving both customer experience and a company’s bottom line.
E-Commerce
In e-commerce, recommendation engines play a crucial role in driving sales. About 35 percent of purchases on Amazon come from product recommendations, according to a McKinsey & Company report. These days, messages like “you may also like this” and “buy this product again” are a familiar site on just about every online retail site.
Recommendation engines are also used to identify products that are frequently bought together by customers and present them as bundled or related items. For example, if a shopper is searching for dumbbells, the recommendation engine may suggest compatible accessories like yoga mats and resistance bands.
Recommendations based on things like location, season, price point and similar users are also common tactics in e-commerce, and are used as a way to incentivize customers to keep shopping.
Social Media
Social media platforms like Facebook and Instagram use recommendation engines to suggest friends or groups based on a user’s existing network, interests and location. They also use them to show relevant posts and advertisements, depending on a user’s preferences.
For example, YouTube considers a viewer’s watch history and ratings to suggest new videos. And TikTok considers videos the user has interacted with in the past, accounts and hashtags they’ve followed, the type of content they create, and their location and language preferences to determine what videos to show on their For You page.
Media Streaming
When a user browses movies and TV shows on a streaming platform like Netflix, Hulu or Max, the recommendation engine analyzes their viewing history, searches and previous ratings to suggest content they’re likely to watch and enjoy. Once a user finishes watching that content, the recommendation engine suggests the next title to watch. All of this is a useful way of keeping users engaged and reducing the time they spend searching for content.
Gaming platforms, like Steam and Playstation Store, and music streaming services, like Spotify and SoundCloud, also use recommendation engines to suggest relevant content based on a user’s preferences and historical data.
Benefits of Recommendation Engines
Recommendation engines can be beneficial both to the companies that deploy them and the users that encounter them.
Improves Customer Experience
A more personalized experience can lead to more satisfied, engaged and loyal customers, mainly because they are being fed the content or products they want without having to put in the effort of finding it themselves.
After all, a lack of a recommendation engine creates a “pretty subpar experience” for customers, as Amplitude’s Thompson put it. Without it, our social media feeds would be full of content we don’t care about. And we’d have to search for every product, movie, show and song ourselves, which would be a pretty time-consuming undertaking
Increases Time Spent on Platform
Social media platforms, media streaming services and even news outlets all want people to spend as much time as possible on their sites. Consistently providing relevant recommendations of more videos to watch, songs to listen to and articles to read keeps users hooked.
This translates to more click-through rates, conversions and — as is often the case with websites — more dollars.
Boosts Revenue
Perhaps the biggest benefit of recommendation engines — on the business side, at least — is that they can help platforms make more money. Not only do recommendation engines incentivize people to make more purchases (a technique known as cross-selling), but they can also suggest product alternatives and draw attention to items that have been abandoned in a customer’s online shopping cart.
Even if a company isn’t in the business of selling physical products, per se, recommendation engines can still do wonders for their bottom line. For example, if Netflix’s recommendation engine consistently feeds viewers content they enjoy watching, they’re less likely to cancel their subscription or choose another streaming service, saving Netflix about $1 billion a year, according to the company.
“If you’re an organization that’s looking to increase revenue, being able to provide tailored experiences for your customers based off their likelihood to purchase or likelihood to complete a particular action, drives growth for your business,” Thompson said.
Challenges of Recommendation Engines
Recommendation engines do come with some challenges, though.
Limited to What They Already Know
A recommendation engine is only as good as the data it’s fed. If it doesn’t have accurate or abundant information about users or items, it likely won’t work correctly.
“They’re limited in their knowledge,” Alexander Marmuzevich, founder and CTO of InData Labs, told Built In. “They can’t propose something which doesn’t exist, they can’t generate completely new ideas.”
A common example of this is what Alexei Tishurov, a lead data scientist at InData Labs, calls a “cold start problem.” This is when a recommendation engine struggles to deal with new users who have not yet provided enough data for the engine to make accurate recommendations. New items with little or no historical data tied to them can be challenging for the engine as well.
“You need to have users interacting with items to do collaborative filtering,” Tishurov explained. “But if you have a completely new service you do not have such history.”
Can Be Biased
Like any machine learning system, recommendation engines can produce biased results if they are based on biased data. This can result in inaccurate or even discriminatory recommendations, posing both functional and ethical problems.
By extension, recommendation engines may fall victim to popularity bias, where popular items tend to be suggested more frequently than lesser-known items. This can lead to a lack of diversity in the recommendations, and prevent users from discovering niche or less popular items.
Gathering Customer Data Can Be Tricky
Data is the backbone of recommendation engines. But as regulations and policies regarding the collection and storage of data continue to evolve, acquiring enough accurate customer data to generate decent recommendations will be an ongoing challenge.
Companies have to be sure they’re compliant with whatever security and privacy regulations exist within the jurisdictions they’re operating out of. And even then, customers can often opt out of providing the data recommendation engines need.
“If a customer is not giving you permission to track them or track their behavior while they’re browsing your website, it’s a lot harder for you to provide those tailored experiences,” Thompson said. Sites like Netflix and Amazon “can’t operate without being able to use the models to provide tailored recommendations,” he continued. “It’s a core, business critical system when it comes to providing their service.”
Frequently Asked Questions
What is an example of a recommendation engine?
Amazon, Netflix and YouTube are well-known examples of recommendation engines. These sites gather data about users’ search history, behavior and reviews to suggest things they might want to buy (or watch) next.
What is the most popular recommendation algorithm?
Singular value decomposition (SVD) is the most common algorithm used in recommendation engines. SVD is a mathematical technique that detects patterns and relationships in the data, and determines their strength, in order to extract meaningful information.
