Facial Recognition, Explained

Facial recognition is a technology used to verify a person’s identity by analyzing a digital image of their face and then comparing it to a database of known faces. It works by measuring their facial features, such as nose shape, jawline and the distance between their eyes.

Facial recognition is used to improve airport security, unlock smartphones, assist law enforcement with surveillance and much more. Despite its ubiquity, the use of facial recognition remains controversial, particularly when it comes to its potential for misuse, bias and privacy infringement. Governments and companies alike are grappling with the legal, ethical and social implications of this technology.

What Is Facial Recognition?

Facial recognition systems are capable of verifying or identifying a person’s identity by analyzing their unique facial features from an image. A form of biometrics, facial recognition relies on computer vision and artificial intelligence algorithms to work. 

Face recognition software leverages artificial intelligence, image recognition and other advanced technology to map, analyze and confirm the identity of a face. This can then be used to identify specific people in photos and videos, determine if a face in different images belongs to the same person, and search for a particular face among a large collection of images.

More on Artificial IntelligenceExplore Built In’s AI Coverage

Benefits of Facial Recognition

Greater Convenience

In many ways, facial recognition is a quick and easy verification system. It is faster and more convenient than comparing facial data by eye, and it requires fewer touchpoints compared to entering passwords or PINs.

“All of those friction points are kind of taken away by just saying, ‘Hey, take a photo,’” Suvrat Joshi, the chief product officer at identity verification company Veriff, told Built In. “It is fairly unobtrusive. The individual doesn’t have to do much.”

Efficient Security

Because there are fewer friction points, a facial recognition system can help expedite otherwise time-consuming processes — whether that be going through security checks at the airport or logging into a bank account.  

Easy Integration

Many facial recognition systems integrate easily with most security software. This limits the amount of money a company or organization has to invest upfront in order to implement this technology, making it fairly accessible.

How Does Facial Recognition Work?

In general, face recognition works in three basic steps: detection, analysis and recognition. But it’s important to note that all software operates a little differently because they are built on proprietary algorithms.

1. Detection

First, a camera is used to locate a specific face in an image, either alone or in a crowd. You see this technology in action every time your phone camera highlights a face and draws a box around it to auto-focus.

Facial recognition systems often use computer vision to detect faces with more accuracy and speed than the human eye is capable of. Using artificial intelligence, this technology can automatically extract, analyze and classify specific information from image data, including single images, video footage and views from multiple camera angles. 

2. Analysis

Once a face is detected, it is digitally mapped out — often by measuring distinguishing facial landmarks like the distance between the eyes, depth of the eye sockets, shape of the cheekbones and contour of the chin, lips and ears. The system then converts all that information into a string of numbers or points called a faceprint. Similar to a fingerprint, everyone has their own unique faceprint.

For example, if you’ve ever unlocked an iPhone using facial recognition, the system works by projecting thousands of invisible dots to create a depth map and infrared image of your face. These are both transformed into a mathematical representation, and compared to previously collected facial data to make sure it matches. 

3. Recognition

The final step is when a system attempts to confirm the identity of the person by cross-referencing their face to a database of other faces. 

These faces can be anything from passport photos to selfies posted on social media, depending on the facial recognition tool. Many facial recognition systems calculate a confidence score, which gauges how similar two images are to each other. A confidence score measures the probability of a prediction. A greater confidence score means a greater probability that two images match.

For example, if a bank robbery is recorded on a security camera, police can use a facial recognition system to compare the face captured at the scene with mugshots of previously convicted criminals, which could help them identify or rule out suspects.

Meet the Top Industry Players5 Top Facial Recognition Companies

Applications of Facial Recognition

Here are some of the most common ways that face recognition is used:  

Unlocking Phones and Computers

Facial recognition is a typical user authentication method in technology like smartphones and computers. This allows users to unlock their personal devices, make banking transactions and access secure information using their face instead of a password.

• Example: Apple has offered a facial recognition feature called Face ID since 2017, allowing users to quickly unlock their phones, log into apps and make purchases.  

Law Enforcement

Police and other law enforcement agencies use facial recognition to identify crime suspects. They accomplish this by comparing a person’s face captured in surveillance footage to other photos across various local, state and federal databases.

• Example: The FBI has a facial recognition program called the Next Generation Identification - Interstate Photo System (NGI-IPS). According to the U.S. Government Accountability Office, the system has access to millions of criminal mugshot photos, as well as passport photos, biometric data collected by the Defense Department and state driver’s licenses — all of which are used to help law enforcement agencies all over the world identify suspects and persons of interest. The FBI says it prohibits any law enforcement agencies from making arrests based solely on the results of an NGI-IPS search, though. 

Airport Security

Many airports allow travelers to use their face data as a sort of passport, allowing them to skip long security lines, check bags and board planes without having to show a physical ID or boarding pass. 

• Example: With Delta Digital ID, travelers on TSA’s PreCheck program can use their faces in lieu of boarding passes and ID at both bag drops and security checkpoints at participating airports. 

Fraud Detection in Banking

People can authenticate their banking transactions by simply looking at a phone or computer instead of using passwords or two-step verification. Banks may also use facial recognition systems at their ATMs or branch locations to verify customers’ identities.

• Example: The Chase banking app supports Apple’s Touch ID and Face ID, as well as Android’s Fingerprint Login, to allow customers to access their accounts using their biometric data. The company also allows customers to withdraw cash from its ATMs with their mobile wallet, which often requires users to scan their face before using. 

Smart Cars

Carmakers implement face recognition to enable keyless entry, monitor fatigue or lack of attention at the wheel, provide customized settings according to who is driving, and much more.

• Example: Hyundai uses facial biometrics in its electric vehicles to automatically adjust seating, mirrors and infotainment displays according to specific drivers’ preferences. And the Genesis GV60, a luxury car owned by Hyundai, became the first vehicle ever to offer keyless entry with facial recognition.

Patient Identification in Healthcare Settings

Healthcare facilities have adopted face recognition to serve a variety of roles, including identifying patients. A facial recognition system can simplify the registration process for patients, make it easier to locate and access patient data and even detect if a patient has a temperature when outfitted with thermal imaging technology.

• Example: Acadian Ambulance has partnered with Duality Health to adopt the company’s facial recognition systems. The goal is to better coordinate patient care across ambulance services, hospitals and other healthcare facilities, so healthcare personnel can more quickly identify patients and deliver personalized treatment.

Want More Examples?5 Ways Facial Recognition Is Making Waves Across Industries

Challenges and Controversies of Facial Recognition

Loss of Privacy

One of the biggest concerns about facial recognition is its potential to violate people’s privacy. In many cases, individuals may not even be aware that their biometric information is being collected or used for facial recognition purposes. 

“It can be used without your consent,” cybersecurity expert Sean Grimaldi told Built In. “That makes it really hard to get control of what is essentially your data.”

For instance, if footage from a street camera is being used to determine whether a crime suspect walked through a certain area, a police department could scrutinize potentially hundreds of innocent people’s faces before they find who they’re looking for. 

“Suddenly everyone, whether they know it or not, is part of this lineup in a police investigation,” Patrick K. Lin, a technology law researcher focusing on AI privacy and surveillance, told Built In.

Even if you opt-in to face recognition, it can be difficult to know all the ways your face will be used, where images of your face will be stored, and who will ultimately be able to get their hands on them.

That said, some regulations have been put in place to better protect people’s privacy. Illinois, Texas, New York, Vermont and Washington state have laws regulating the collection, storage and disclosure of biometric data like face scans and fingerprints. And several corporate giants have had to pay large sums of money for violating them. 

People can better protect their own privacy by choosing to opt out of facial recognition services like Apple’s Face ID or face scans at security checkpoints. But beyond that, there’s not much an individual can do to completely safeguard their facial data. 

“If you are engaged in civil society, it’s kind of impossible to cut out facial recognition,” Lin said. “Especially for people who live in cities. The moment you step outside, you cross the street, you’re captured.” 

Potential Misuse By Authorities

Because law enforcement agencies use facial recognition so regularly, another concern is that governments misuse this technology for mass surveillance, potentially infringing on people’s civil liberties. 

The Chinese government has used facial recognition to monitor and control ethnic minorities, and protesters in Hong Kong famously wore masks to thwart police cameras. Meanwhile, in the United States, facial recognition systems were used to monitor people during the Black Lives Matter protests in 2020. And Immigration and Customs Enforcement uses it to track people who enter the country. 

A handful of U.S. cities, including San Francisco and Boston have banned the use of face recognition by government entities like police departments. But plenty of private companies continue to sell their software to law enforcement agencies big and small — usually with very little oversight or public scrutiny into where their databases of images come from or how their algorithms work. 

Bias and Inaccuracies

Facial recognition algorithms have been found to exhibit biases against certain demographics, which can cause systems to make mistakes. In fact, researchers at the National Institute of Standards and Technology (NIST) found that the majority of the 189 facial recognition algorithms they studied exhibited bias, misidentifying Black and Asian faces up to 100 times more often than white faces. Women were also falsely identified more often than men.

These kinds of errors are largely attributable to a facial recognition system’s training process, which involves feeding the system tons of examples of faces so that it can learn to identify one. If you train an algorithm on enough examples, eventually it will learn the difference between a wall outlet and a human face. But if the corpus of training data largely comprises white, male faces, then a system will struggle to accurately identify anyone outside of that demographic.

At best, this can lead to a frustrating user experience with face recognition, where photos are automatically tagged with the wrong person’s name, for example. At worst, it can lead to false arrests and wrongful deportations. It can also prevent certain people from being able to use the tech entirely — as we’ve seen with African and Haitian asylum seekers attempting to use the U.S. Customs and Border Protection’s face scanning app.

Security Risks

Many companies go to great lengths to make sure their databases are secure. Still, no technology is totally tamper-proof. Companies get hacked all the time, Grimaldi said, and databases of face data can wind up for sale on the dark web. There, these images can be used to commit fraud. And once your data is stolen, you can’t solve it in the same way you could solve a stolen password. After all, you can’t exactly change your face.

“If you have a passphrase or a password and it’s compromised and for sale on the dark web, you can just get a new one,” Grimaldi said. “But if that happens with your facial recognition data — like, for example, the points mapping your face — you can’t change it, ever.”

Related ReadingPassword Alternatives: Will Biometrics and Cryptography Become the Successors to Passwords?

Is Facial Recognition Accurate?

It is impossible to assess the accuracy of all facial recognition systems, as there is no single metric to offer a complete picture. But many systems have accomplished near-perfect precision in conditions with consistent lighting, a controlled background and clear and unobstructed features. Of course, this isn’t always possible.

Several factors can affect a facial recognition system’s accuracy. For one, while all of us have the same face our whole lives, it can change quite dramatically — we age, facial hair comes and goes, sometimes we wear glasses or makeup. Plus, camera quality can vary quite a bit, which can then affect image quality and a system’s ability to accurately identify people. Not to mention if the room is dark, or the camera is angled strangely.

Frequently Asked Questions