AI Bias: What It Is and How to Prevent It

AI bias — also called machine learning bias or algorithmic bias — refers to the unfair decisions made by artificial intelligence systems due to skewed data, flawed algorithms and inherent human biases. It is one of the greatest dangers of AI because it not only mirrors real-world prejudices but also amplifies them, disproportionately favoring or discriminating against specific groups in ways that can perpetuate systemic inequality.

The harms of AI bias can be significant, especially in areas where fairness matters. A biased hiring algorithm may overly favor male applicants, inadvertently reducing women’s chances of landing a job. Or an automated lending tool may overcharge Black customers, hindering their chances of buying a home. And as artificial intelligence becomes more embedded in consequential industries like recruitment, finance, healthcare and law enforcement, the risks of AI bias continue to escalate.

“Whether we like it or not, all of our lives are being impacted by AI today, and there’s going to be more of it tomorrow,” Senthil Kumar, chief technology officer at data analytics company Slate Technologies, told Built In. “Decision systems are being handed off to machines — and those machines are biased inherently, which impacts all our lives.”

What Causes AI Bias?

Bias primarily creeps into AI systems through training data. AI models learn how to make decisions and predictions based on the data they are trained on — and if that data is full of societal inequalities and stereotypes, those biases will inevitably be absorbed by the model and reflected in its outputs. Plus, if the data is incomplete or not representative of the broader population, the AI may struggle to produce fair and accurate results in scenarios it hasn’t encountered, further perpetuating discrimination.

“AI bias is born from data bias,” Adnan Masood, chief AI architect at IT services company UST, told Built In. “It is a mirror and magnifier of existing inequalities. It amplifies the prejudices we already have.”

Indeed, all artificial intelligence is the product of human beings, who are inherently biased, making it nearly impossible to avoid bias in AI systems. Developers may inadvertently introduce their own prejudices, overlooking important information while collecting data or teaching an algorithm to favor certain patterns during the machine learning process. In the end, the model will reflect those prejudices in its own decisions.

But unlike human decision-makers — whose biases can be more readily identified and challenged — AI systems operate in the background, often making decisions that are difficult (if not impossible) to fully understand or trust. This not only upholds existing inequalities but also hinders adoption of the technology itself, as the public grows increasingly wary of systems they can’t fully count on or hold accountable.

Related ReadingWhat Is AI Safety?

Real-Life Examples of AI Bias

There have been a number of documented instances of AI contributing to already existing biases in societies. Below are some notable examples of algorithms fueling biases in healthcare settings, workplaces and beyond. 

AI Algorithms Reinforce Biases in Healthcare 

The healthcare industry is riddled with AI bias, according to a 2024 study. A lack of diversity among AI developers and physicians is a key reason those building AI tools often fail to account for current racial health disparities when training algorithms. As a result, developers feed a patient’s medical records, biomarkers and other health data to an algorithm, instead of considering factors like a patient’s access to public transit and healthy food options.  

Shifting what types of data healthcare professionals pay attention to is the focus of another study led by Yale researchers. The team notes that race and ethnicity may not be effective for training predictive clinical algorithms, suggesting that algorithms alternatively be trained on variables like genetic variation and social class. Changing these practices can help root out long-standing biases in healthcare that unfair algorithms continue to uphold.

AI Hiring Algorithm Worsens Gender Biases 

In 2014, a team of Amazon AI researchers created a machine learning recruiting engine to automate the process of assessing job applications. To train the recruiting engine, the team exposed it to parameters from resumes previously submitted to the company over the span of 10 years. The only problem with this method is that there is a major gender gap in the tech industry, and Amazon is no exception to this rule. 

The resulting model proved to be biased against women, favoring male-dominant keywords in resumes. Although researchers attempted to counter biases present in the model, this wasn’t enough to prevent it from following gender-biased logic. Researchers eventually retired the algorithm from use in 2018. 

Image Generator Promotes Race and Gender Biases Among Professions

A 2023 study conducted by Bloomberg showed just how ingrained societal biases are in generative AI tools. Upon creating more than 5,000 images with Stable Diffusion, the company found the generator reproduced glaring racial stereotypes. For example, the tool generated more images of lighter-skinned faces for high-paying professions like CEO, lawyer and politician. On the other hand, it produced more images of darker-skinned faces for lower-paying professions like social worker, fast food worker and dishwasher. 

Stable Diffusion also depicted disparities along gender lines. While CEOs, doctors and engineers were mostly portrayed as men, cashiers, teachers and social workers were largely presented as women. As more online content is AI-generated, studies like Bloomberg’s continue to raise concerns about AI technologies further grounding society in damaging stereotypes. 

Examples of Biased AI

Unfortunately, AI bias isn’t limited to a few instances. The issue spans various industries and technologies: 

Racial Bias in Generative AI

Generative AI tools — particularly image generators — have developed a reputation for reinforcing racial biases. The datasets used to train these systems often lack diversity, skewing toward images that depicted certain races in stereotypical ways or excluding marginalized groups altogether. As a result, these biases are reflected in AI-generated content, often portraying white people in roles of authority and affluence, and people of color as low-wage workers and criminals. Some experts predict that as much as 90 percent of online content will be artificially generated in the next few years, raising concerns that these biases will further perpetuate prejudice and hinder progress toward more equal representation.

Hiring Biases in Generative AI

A UNESCO study found that large language models (LLMs) carry gender-based biases, often associating women with roles like “prostitute,” “cook” and “domestic servant.” This has a direct impact on the hiring process. According to a University of Washington study, LLMs preferred female-associated names over male-associated names only 11 percent of the time. In addition, LLMs selected white-associated names 85 percent of the time, leading to clear hiring biases along gendered and racial lines.   

Inappropriate Content in Generative AI

A Stanford University study discovered more than 3,200 images of possible child sex abuse in the AI database LAION, which has been used to train tools like Stable Diffusion. Perhaps not surprisingly, an earlier study led by the University of Washington found Stable Diffusion sexualizes women of color. While there will always be bad actors looking to exploit AI technologies, these flaws in AI image generators reveal how easy it is to produce and spread harmful content, even if unintentional.   

Gender Bias in AI Voice Assistants

From Apple’s Siri to Amazon’s Alexa, many of today’s most popular assistants have feminine voices and embody traditionally feminine personas that inadvertently position them as subservient, submissive and overly obliging. This gendered design choice not only reflects harmful stereotypes, but it also influences how users interact with this technology — often leading to more dismissive, condescending and even harassing behavior toward the voice assistants. The ongoing feminization of voice assistants reinforces and upholds outdated notions of gender roles, carrying overly simplistic generalizations into modern times.

Sex and Racial Disparities in Healthcare Algorithms

AI is increasingly being applied in healthcare, from AI-powered clinical research to algorithms for image analysis and disease prediction. But these systems are often trained on incomplete or disproportional data, compounding existing inequalities in care and medical outcomes among specific races and sexes. For example, an algorithm for classifying images of skin lesions was about half as accurate in diagnosing Black patients as it was white patients because it was trained on significantly fewer images of lesions on Black skin. Another algorithm developed to predict liver disease from blood tests was found to miss the disease in women twice as often as in men because it failed to account for the differences in how the disease appears between the sexes.

Ableism in AI Recruiting Tools

Hiring algorithms used to screen applications have a demonstrated bias against individuals with disabilities — often because these systems are trained on data that only reflects able-bodied norms and assumptions. For example, programs that assess candidates’ communication skills, attention span, physical abilities and even facial expressions tend to unfairly disadvantage those with disabilities by failing to account for different modes of communication or movement. And resume scanners are apt to reject applicants with large gaps in their work history, without considering that those gaps may be due to health-related reasons. By reinforcing ableist hiring practices, AI recruiting tools limit job opportunities for individuals with disabilities and perpetuate discrimination in the job market at scale.

Class Bias in Automated Credit-Scoring

AI models for predicting credit scores have been shown to be less accurate for low-income individuals. This bias arises not necessarily from the algorithms themselves, but from the underlying data, which fails to accurately depict creditworthiness for borrowers with limited credit histories. A thin or short credit history can lower a person’s score because lenders prefer more data. It also means that just one or two small dings (a delinquent payment or a new credit inquiry) can cause outsized damage to a person’s score. As a result, these individuals may be wrongfully denied loans, credit cards and housing, which can hinder their ability to build wealth and improve their economic situation down the road — perpetuating a cycle of financial disadvantage that further entrenches socioeconomic inequalities.

Related ReadingWhat Is Responsible AI?

What Are the Consequences of AI Bias?

Biased models are likely to spread their prejudices to the AI products they support, eventually impacting society. For example:

• Consider an LLM used to review candidate applications as part of the hiring process. If the LLM is taught to believe that all doctors are men and all nurses are women, its flawed logic will transfer over to the hiring tool it is powering, leading it to prioritize gender over qualifications in its decision-making.
• Facial recognition systems often don’t work as well for people with darker skin tones, which can lead to issues that range from something as minor as struggling to unlock a smartphone to more serious consequences like false arrests and wrongful deportations.
• According to UNESCO, the top AI chatbots have only been trained on about 1 percent of the world’s 7,000 natural languages (with English being the primary source), rendering them useless for large swaths of people.
• Even something as innocuous as recommendation systems in streaming services can exhibit bias, disproportionately promoting popular content in ways that prevent users from finding more diverse narratives they might also enjoy.

“Once a model has learned incorrect information, it is going to spread this knowledge,” Kumar said. “So the models aren’t just learning through biased datasets and biased processes, but they’re also propagating biases.”

Sources of AI Bias

AI models can exhibit many types of AI bias, including:

• Algorithmic bias occurs when the AI algorithm itself introduces or amplifies bias, often as a result of poor training data and the biases of the humans who compiled the data and trained the algorithm.
• Selection bias (also known as sample bias) arises when a training dataset is so small, disproportionate or incomplete that it fails to represent a real-world population, making it insufficient to properly train an AI model.
• Cognitive bias is introduced by the humans developing the model, stemming from their ingrained thought patterns and unconscious mental shortcuts that inadvertently shape how the data is interpreted and the AI system is designed. 
• Confirmation bias is closely related to cognitive bias; it involves humans favoring information that validates a pre-existing belief or trend. Even with accurate AI predictions, developers may ignore results that don’t align with their expectations.
• Historical bias occurs when outdated stereotypes and patterns of discrimination are embedded in the algorithm’s training dataset, causing the AI to replicate and reinforce these biases in new contexts. 
• Out-group homogeneity is the tendency to perceive members of a different group as more similar to each other than they actually are, often resulting in harmful stereotypes and generalizations. This can lead AI systems to misclassify individuals within that group, resulting in racial bias and inaccurate results.
• Exclusion bias is when important data is left out of a dataset, often because the developer has failed to understand its significance. 
• Recall bias happens during the labeling process, where data is inconsistently categorized based on the subjective observations of humans. 

How to Prevent AI Bias

Mitigating AI bias is a complex challenge that requires a multifaceted approach: 

Maintain a Diverse Development Team

Different perspectives can help identify potential biases early in the development stage. A more varied AI team — considering factors like race, gender, job role, economic background and education level — is better equipped to recognize and address biases effectively. 

“Homogenous teams are more likely to overlook biases,” Shomron Jacob, head of applied machine learning at AI development company Iterate.ai, told Built In. “Different people from different backgrounds bring in multiple perspectives. And those inputs end up mitigating biases by default, because they bring in a perspective other people wouldn’t consider.”

Employ the Human in the Loop (HITL) Approach

Human in the loop (HITL) involves humans in training, testing, deploying and monitoring AI and machine learning models. While models still learn on their own, humans can step in to solve problems the models struggle with and correct any mistakes they make. This way, teams can quickly identify biases and remove them from the learning process.   

The HITL method also aids reinforcement learning, where a model learns how to accomplish a task through trial and error. By guiding models with human feedback, HITL ensures AI models make the correct decisions and follow logic that is free of biases and errors.   

Train With the Right Data

Algorithms are only as good as the data they have been trained on, and those trained on biased or incomplete information will yield unfair and inaccurate results. To ensure this doesn’t happen, the training data must be comprehensive and representative of the population and problem in question.

“We should have diverse training data from a wide range of demographics and a wide range of scenarios. This is how you will reduce the likelihood of the system learning biased patterns,” Jacob said. “If you keep collecting bigger sets from different places, it will not learn any biased patterns directly.”

Equip Models With Fairness

Ensuring models are inherently fair can be accomplished through a number of techniques. One approach is known as fairness-aware machine learning, which involves embedding the idea of fairness into each stage of model development. For example, researchers can reweight instances in training data to remove biases, adjust the optimization algorithm and alter predictions as needed to prioritize fairness. 

Developers may also ingrain fairness into an AI model through adversarial debiasing. This method pits two neural networks against each other, where one network tries to identify a protected attribute like gender or race in a data set and the second network tries to make it as difficult as possible for the first network to locate the attribute. Models then learn not to put too much weight on a protected attribute, leading to more objective decision-making.

When it comes to testing whether a model is fair, a good method to use is counterfactual fairness. The idea is that a model should make the same prediction for two instances, given that these two instances are identical with the exception of a sensitive attribute. For example, if a hiring algorithm is presented with two candidates who have identical experiences and only differ in gender, the algorithm should theoretically either approve or reject both. 

Implement Constitutional AI

Constitutional AI is a training method that teaches a model to obey ethical principles. The process starts by feeding a model a list of rules to adhere to — its “constitution” — and examples that follow and do not follow these rules. A second model then monitors and determines whether the first model is acting according to its constitution, adjusting any of the first model’s responses that break from the rules. 

Anthropic has been a champion of this approach and uses it as a safeguard for its chatbot Claude. With constitutional AI, developers not only limit a generative tool’s ability to deliver harmful responses but also make it easier for users to understand and fine-tune the outputs.     

Implement Explainability Wherever Possible

The inner workings of AI models are often unclear, which makes it difficult to pinpoint the exact origins of their bias. Developers can opt for more explainable algorithms like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHaply Additive exPlanations) to process the data, helping them to better scrutinize and evaluate their decision-making processes.

“If we have that traceability and explainability, that at least helps us to be able to further detect that this particular prediction might be biased, and we can take actions to prevent it,” Jim Olsen, chief technology officer at AI governance company ModelOp, told Built In. “Sometimes it’s not about not having any AI bias in the model itself, but it’s about the ability to detect it and stop it.”

Continually Monitor Models

AI models should be regularly monitored and tested for bias, even after they’ve been deployed. Models constantly take in new data with use and their performance can change over time, which may lead to new biases. Routine audits allow developers to identify and correct the issues they see before they cause harm.

“No model is ever going to be perfect, and that’s not the expectation. The expectation is that the due diligence is done,” Olsen said. “If you monitor your models constantly to make sure they aren’t drifting in their nature, their predictions, et cetera — and can prove when you are audited that you did all these steps — that’s all anybody can ask for.”

To further avoid bias, these assessments should be carried out by independent teams within the organization or a trusted third party.

Establish More Regulations

Regulation can play an important role in addressing and mitigating AI bias by establishing guidelines and standards that ensure fairness and accountability. There are already many laws on the books protecting people from wrongful discrimination in areas like banking, housing and hiring (and several companies have been punished for violating those laws with AI). But for less obvious forms of AI bias, there are fewer legal safeguards in place.

Governments around the world have started taking steps to change that though, including the European Union, the United States and China. And various industry groups are implementing best practices in responsible AI development, promoting things like diverse data collection, transparency, inclusivity and accountability.

Frequently Asked Questions