With AI systems becoming increasingly complex and leading AI entrepreneurs predicting that AGI may be realized before 2030, it might appear that now, more than ever, we need to have some guardrails in place to control the mass implementation of AI. After all, AI is creeping into all corners of our lives, from a simple call center chatbot to sophisticated scientific applications, like Google Deepmind’s AlphaFold for protein folding, to much more frightening cases, such as Palantir’s and Anduril’s solutions that are modeled using classified military data.
As AI systems evolve, a critical question emerges: Can we trust these technologies if we can’t fully understand them? With an increasing number of businesses and ordinary people adopting AI systems daily, the need to understand their inner workings has never been more pressing. But can we expect black box AI to be transparent? And how can we safeguard ourselves from increasing reliance on a technology that is becoming too opaque in its computations to keep its advancements in check?
Black Boxes or Glass Boxes?
This is where the concept of explainable AI (XAI) comes in. XAI comprises a range of processes, methods and technologies that help make AI systems more transparent and understandable. The aim of XAI is to shed light on how a particular AI system formulates predictions, passes judgement and implements its actions. Thus, we can consider XAI models as “glass boxes” rather than black boxes.
According to IBM, XAI is key to the large-scale implementation of AI technologies in real-life organizations, as it enables fairness and accountability. In certain business sectors, like manufacturing, XAI is a non-negotiable step to broader adoption.
XAI addresses much broader questions than those specific to one industry, however. As scientists fear that large AI systems are beginning to develop consciousness and businesses are moving towards agentic AI architectures, where multiple models interact among themselves, the necessity of opening the black box is a matter of safety and security, not just effectiveness and trust.
A 2020 paper called “Four Principles of Explainable Artificial Intelligence,” published by the National Metrology Institute for the United States, laid out the cornerstones of XAI fairly simply:
4 Principles of Explainable AI
- Explanation: A system provides reasons or evidence for its outputs and processes.
- Meaningfulness: Explanations given by the system are understandable to the intended users.
- Explanation accuracy: The explanation correctly shows why the system generated a particular output or accurately describes how the system works.
- Knowledge limits: The system only operates within its designed parameters and when it has enough confidence in its output.
Although AI today is nothing like it was in 2020, these simple principles still offer a response to one of society’s main objections to embracing AI: our deep-seated reluctance to trust things we don’t understand.
Most of you will probably agree that a cancer diagnosis given by an AI system is way more convincing when it’s supported by the specific imaging patterns that led it to that conclusion. Likewise, in criminology, a recidivism risk score becomes actionable when explained by the factors that contributed to that high risk assessment. So, when an AI system makes an important decision, it’s completely reasonable for those affected (and society as a whole) to ask how that decision was made.
The XAI approach offers a number of advantages. It can help developers improve their models more easily, as increased transparency is beneficial for debugging and fine-tuning. Explainability is important for legal compliance as well, as regulations like GDPR and the EU AI Act demand that decisions made by certain automated systems be explainable, especially (but not only) when personal data is involved.
Theoretically speaking, correctly executed XAI could provide many real-world benefits to businesses. Take, for example, an AI-powered attrition prediction system that might be implemented to help an HR department. Such a system becomes significantly more valuable if it identifies who might leave the company, at the same time explaining the factors driving that risk, thus empowering targeted interventions. Is such a model practically possible, however?
XAI: A Workable Model or Wishful Thinking?
The problem is that as AI models have grown in scale and complexity, traditional approaches to explainability have begun to strain under their weight. Several fundamental challenges have emerged.
First, modern large language models (LLMs) and multimodal systems contain billions or even trillions of parameters, with architectures that make tracing the exact reasoning path virtually impossible. The computational graphs representing the decision-making in these systems are so gigantic that even the most sophisticated XAI techniques can provide only approximations of their internal workings.
Second, many of the popular XAI techniques are used post hoc, where they analyze a model’s outputs to explain its decisions. Unfortunately, these methods can sometimes churn out explanations that sound good on the surface but don’t truly represent the model’s internal workings. As AI models grow in complexity, the disconnect between these explanations and the model’s actual reasoning only gets larger.
To compound matters even further, there is the fact that different stakeholders require different kinds of explanations:
- Developers troubleshooting a model need technical details about activation patterns and gradient flows.
- End-users need intuitive, non-technical justifications.
- Regulators require traceable documentation of data lineage and algorithmic processes.
Taking the diverse needs of stakeholders into consideration, it’s difficult to construct a unified explanation that caters to all parties.
And finally, there’s the disconnect that can appear between the need for optimal performance and complete transparency. Striking the right balance between a model’s performance and transparency remains one of the biggest challenges facing XAI today. Generally, the most accurate and powerful AI systems are also the least transparent, while models that are easier to interpret often compromise on their predictive capacity or generative abilities. This creates a tough dilemma: Is it better to prioritize performance or trust?
Is There a Future for XAI?
These challenges indicate that conventional XAI might be hitting a wall as AI is rapidly advancing towards AGI. Most static explanations fall short when it comes to capturing the fluid nature of complex AI reasoning. What might work better are interactive methods that let users ask questions about the models’ thought processes and explore their decision boundaries. For instance, to gain a deeper understanding of how AI systems reason, some researchers use counterfactuals — small tweaks to inputs — to find out which input changes lead to the most noticeable differences in output.
Another promising future direction involves shifting AI systems away from purely statistical pattern recognition toward causal reasoning. Models that understand causality rather than mere correlation can provide “why” answers that align more naturally with human reasoning. Recent work on causal representation learning and causal discovery in deep neural networks offers promising early steps in this direction.
The most radical solution would be to move away from trusting systems because we know how they tick and instead trust them based on the good results they deliver that resonate with our values. This idea recognizes that while we might never fully understand every detail of complex AI systems — e.g., quantum AI — we can still find ways to trust them based on their reliability in delivering the goods.
Therefore, rather than doubling down on increasingly strained explanatory techniques, we might soon need a significant shift in our approach — one that goes beyond just explaining AI and focuses on rethinking how humans and AI can work together.
So, instead of striving to clarify the workings of increasingly complicated AI systems, we might be better off ensuring that AI systems are truly in sync with human values, objectives and thought processes. This approach, centered on alignment, would emphasize developing AI that acts in ways we deem reasonable and ethical, even if we don't have a complete grasp of the underlying mechanisms.
Rethinking AI
As we edge closer to AGI, the main question we should be considering is not whether to give up on making AI transparent and understandable, but rather how we can adapt to this new reality. The XAI approach needs to shift from merely providing dry, mechanical explanations to meeting AI systems halfway.
This change demands a team effort that spans various fields, including machine learning, cognitive science, philosophy and social sciences. It also involves embracing innovations like causal reasoning, interactive explanation systems, and new ways to evaluate whether our explanations truly achieve their intended outcomes.
Above all, we must honestly recognize that our interactions with increasingly sophisticated AI will be fundamentally different from our experiences with traditional software. Just as human relationships thrive not on complete transparency but on trust built through shared values and consistent, mutually beneficial actions, our connection with advanced AI might evolve in a similar way.
