Algorithmic Bias and the Tools Working to Prevent It

Like people, algorithms can have significant blind spots. Algorithmic bias refers to algorithms making systematic errors due to low-quality training data and the biases of the humans training the algorithms. These errors can help or harm specific groups of people, leaving a significant impact on society.

And as algorithms become increasingly ubiquitous, the choices they make for us will have far-reaching implications, according to Aron Culotta, associate professor of computer science at the Illinois Institute of Technology.

“In applications like criminal sentencing, loan applications and self-driving cars, we need algorithms that are not only accurate, but also error-free,” Culotta told Built In. “When algorithms make errors that are somehow unfair or are systematically biased against certain groups of people, they reinforce and worsen any existing inequities.” 

What Is Algorithmic Bias?

Algorithmic bias is when an algorithm commits systematic errors that result in unequal outcomes, unfairly favoring or disadvantaging groups of users. While algorithms ultimately produce the mistakes, the cause of this can be traced back to a combination of inadequate training data and the biases possessed by the humans training the algorithms.

For example, skewed data and small data samples can lead to inaccurate results. However, humans decide what types of data to collect, how much data is needed and whether results meet their standards. If teams don’t test their results, they may move forward with faulty or incomplete data and impede algorithms’ ability to make accurate decisions.         

Biased Training Data Leads to Biased Algorithms 

Algorithmic bias often stems from the data that is used to train the algorithm. And because bias runs deep in humans on many levels, training algorithms to be completely free of those biases is a nearly impossible task.

Algorithmic biases often stem from the text and images that data scientists use to train their models. For example, if you search for images of a “police officer” or “boss” on the internet, most of the pictures that show up will likely be of white men. If you fed this data into your algorithm, your model would likely conclude that bosses and cops are usually white and male, perpetuating stereotypes against women, minorities and other groups.

Everything from the tools and equipment used to collect data to the factors data scientists select to analyze and how they train their models, can cause biases to creep into algorithms. As a result, many industries are grappling with the issue of biased algorithms. 

Examples of Algorithmic Bias

As technology has been integrated into more aspects of daily life, algorithmic bias has become a major problem for a range of industries:  

• HR and Recruiting: To practice skills-based hiring, teams may train hiring algorithms to pick out specific words. But if algorithms are trained with skewed data, they may favor the speech patterns of candidates based on their gender, race and other factors.
   
• Online Search: Algorithms may pick up on social biases and meanings surrounding certain words. Search engines may then pull up biased or inappropriate results when users look up terms or phrases.
   
• Facial Recognition: Facial recognition technology sometimes struggles to see darker skin. This impacts everything from automatic sinks to self-driving cars and their ability to sense various people.
   
• Digital Advertising: Digital ads may target users based on traits like race and gender, and algorithms can reinforce these patterns. For example, a Facebook algorithm prevented users from viewing insurance ads based on their age and gender.
   
• Law Enforcement: Police have begun using AI algorithms to predict where crimes occur, but this technology relies on historical data. Overpoliced areas may then receive even more attention, furthering unfair police practices.

How to Prevent Algorithmic Bias

1. Use Auditing Tools From the Beginning

One way to build better algorithms is to use auditing tools to detect biases in the training model before deploying it in the real world. Aequitas is one such open-source toolkit developed at the University of Chicago. 

To understand how Aequitas works, however, it’s important to understand how data scientists decide if their models are accurate. In data science, there are four kinds of findings:

• True positives: when an algorithm spots a real-world pattern.
• False positives: when an algorithm identifies a pattern but there isn’t one.
• True negatives: when there is no pattern, and the algorithm doesn’t identify one, either.
• False negatives: when the algorithm fails to spot a pattern that exists in the real world.

A model is biased if false positive or false negative rates are significantly higher or lower for a subgroup of people than for the population as a whole, or when compared to another sub-group. Aequitas compares the false positive and negative rates between the “overall reference” group against the “protected or selected” group. If the disparity for a “protected or selected” group is within 80 and 125 percent of the value of the reference group, the audit passes — otherwise, it fails.

The tool assesses different kinds of metrics, such as false negative rate parity, false positive rate parity and false positive discovery rate parity — a criteria that considers whether your rate or errors is the same across all subgroups. Then, it creates a report indicating which metrics are biased.  

2. Define “Fairness” Within the Context of What You Want to Achieve

The idea of fairness is different for each user or application because every government, society or organization will have its own definition of fairness, said Rayid Ghani, one of the primary creators of the Aequitas software and a distinguished career professor at the Heinz College of Information Systems and Public Policy at Carnegie Mellon University. 

For example, one policymaker may define fairness as no one being left behind, while another stakeholder may want algorithms to proactively reduce inequity across all sub-groups over time. In the end, the definition might depend on the consequences of getting things wrong. 

Still, clarifying what fairness means can help teams determine what they want to accomplish with their algorithms and what biases they want to root out.

3. Employ a Fairness Tree 

To help define fairness in past situations, Ghani’s team has designed a Fairness Tree: a systematic way for data scientists and stakeholders to navigate their way directly to the errors that are most impactful to the outcome they were trying to achieve. One of the key considerations in the fairness tree is whether a proposed intervention is punitive or assistive. 

For example, if the algorithm is charged with punitive intervention, such as deciding whether someone should go to jail, then a high false positive rate for any subgroup (sending too many people to jail) becomes much more important than a false negative rate (sending too few), said Ghani, because an incorrect prediction can have a massive impact on an individual’s life and perpetuate societal inequities. 

Alternatively, if you are working on a model whose function is assistive — an algorithm that helps to find the best health insurance option, for instance — recommending an insurance plan one time too many is far less bad.

It’s these types of considerations that the Aequitas toolkit and fairness tree help data scientists and policymakers parse through.

4. Optimize for Both Accuracy and Fairness

Most data scientists are focused on building models that predict correctly for the largest possible number of scenarios — this concept is called “accuracy.” But for some, it can be easy to get so swept up in optimizing for accuracy that they forget about the notion of fairness. 

Part of the problem in the industry today, Ghani said, is that accuracy and fairness are viewed as being mutually exclusive. Ghani and his team started thinking about the fairness and equity of the entire system. What they found was that, based on the desired outcomes, some error variables were more important than others. 

“You shouldn’t care about all types of disparities equally,” Ghani said. “In machine learning models, there is a lot in data. Ultimately, what’s most important is that the overall system is fair.”

The most accurate model for predicting the behavior of large groups of people might lead to unfair outcomes for members of smaller subsets. To avoid that, data scientists can use auditing tools to build models that are both accurate and fair. 

5. Be Transparent About Test Results

The tricky part about avoiding bias, Ghani said, is that predictive models inevitably rely on some level of generalization.

“To get the greatest possible number of people correct, the algorithm is going to be biased or incorrect about some smaller sub-group of people,” Ghani added.

Algorithms may then fail to pass audits or tests in many situations. This may cause some setbacks, but teams must take these results in stride and communicate their findings with employees, customers and other stakeholders.  

“If the algorithm doesn’t pass the audit, a stakeholder can either decide not to use it or use it but let the public know we’ve vetted the problem and it’s the best we can do,” Ghani said.

6. Convene a Comprehensive Group of Stakeholders

Ghani said it should not just be the policymakers or lead data scientists at a company who define the ideal outcome. Outside stakeholders should have a say in algorithms that impact entire communities. Assembling a group of diverse perspectives can better inform projects and illuminate other issues that data science teams may have overlooked.   

Having a collective conversation about bias metrics upfront will help data scientists encode systems more effectively. It also offers an opportunity to talk frankly about what kinds of biases are most important to avoid.

7. Keep Testing Algorithms After They’re Released

Even if data science teams repeatedly test algorithms before releasing a product, it’s impossible to know the actual behavior of these algorithms until they’re placed in real-world situations. Testing algorithms and AI technologies after their release is then essential for creating an accurate picture of algorithms’ performance.  

“You should be constantly monitoring the system to see if you are having the impact you thought you were going to have,” Ghani said.  “It’s important to track AI tools and make sure we have guard rails in place. The alternative is too risky.”

Frequently Asked Questions