When Brandon Cole was six years old, his older brother Justin played a cruel trick on him. He invited Cole, who is blind, into the living room of their Minnesota home to play Super Mario Bros. on the original Nintendo Entertainment System.
For hours, Cole broke blocks and stomped on turtle shells, believing he was on his way to restoring Princess Peach to her rightful throne in the Mushroom Kingdom. But when he defeated Bowser, or so he thought, his brother came clean with his double-dealing.
“I believe that I made a vow that day that I would one day beat a game without his assistance.”
“The punch line was he had handed me the completely unplugged second player controller and told me to press buttons while he played the entire game. So literally zero of my inputs actually mattered,” Cole said. “I believe that I made a vow that day that I would one day beat a game without his assistance.”
He has done that, of course, finishing Killer Instinct for the Super Nintendo shortly after the console’s release and becoming something of an expert at bidirectional 2D fighting games like Mortal Kombat, which are “kind of naturally accessible” because “the plane of movement is very small.”
“So we don’t worry about navigation because we always know we’re going after our opponent, no matter which way we’re going. As long as we learn the strings of attacks, we can work out which side we’re on,” Cole said.
But Cole’s experience with games goes far beyond playing them. As a consultant and advocate for the video game industry, Cole maintains a blog, speaks regularly at game conferences and events — such as Summer Game Fest 2021, which he narrated live on his superblindman Twitch channel — and helps designers and developers make their games more accessible, especially for people who are blind.
Accessibility Is Key to Bringing Games to the Mainstream
One of his most recent and notable efforts was a consulting project for The Last of Us Part II, a post-apocalyptic saga starring Ellie, the survivor of a fictional pandemic who leaves a community in Jackson, Wyoming and sets out on a rogue mission of vigilante justice.
Directed by Neil Druckmann of Naughty Dog, a first-party studio for Sony PlayStation, the game, recognized by more than 300 Game of the Year awards, brings together audio narration of all screen text, sound and controller vibration cues, traversal assistance, and slow-motion combat and invisible combat modes to improve playability for blind and low-vision gamers.
“I like to say that when The Last of Us Part II came out, we launched the accessibility bar into the stratosphere. No epic game has ever done this, especially on this scale,” Cole said. “Just because it is finally a game made by a developer who cares about totally blind people enough that they did everything necessary for us to play the game to completion.”
That developer was Sam Thompson, who, with guidance from Cole, oversaw a team that created pre-recorded screen-reader narration for every choice players make in the game, from selecting menu items to retrieving and reading notes from the ground. This was a mammoth undertaking. While some studios offer self-voicing, text-to-speech screen readers built on proprietary game engines, this wasn’t an option at Naughty Dog due to technical limitations. So they created the audio descriptions themselves and hired voice actors to read them, embedding the recordings — sampled in 14 languages — directly into the action of the game.
“I like to say that when The Last of Us Part II came out, we launched the accessibility bar into the stratosphere. No epic game has ever done this, especially on this scale.”
But they didn’t stop there. At their discretion, players can activate an enhanced listen mode, “basically, a sonar ping for enemies and items that will navigate the player toward either the nearest item or the nearest enemy they’re looking for,” Cole said.
In truth, the navigational assistance tool is a bit more complex than that, he told me. A technique called ray casting lets players manually cast an invisible spherical “net” that scans a range of 30 meters in any direction. By holding down the R1 button on the controller to scan the environment, players can determine their distance from objects and enemies. The duration of the sound they hear upon scanning indicates how far away things are — or if there’s anything there at all. After a player knows what’s in front of them — or behind them — they can push the L3 analysis button while holding down R1 to navigate to the nearest waypoint, be it a door, wall or wild-eyed marauder.
The game’s crowning achievement, though, may be an adaptation of a feature Cole stumbled on in the popular survival horror game Resident Evil 6, which was not originally intended as an accessibility tool.
“So in Resident Evil 6, as long as you’re holding the map button that brings up your map on screen, [then] it also points the camera in the direction of your next objective,” Cole said. “They decided to make a very particular design choice in that game. They didn’t do it for blind people. But they did what I like to call ‘accidental accessibility,’ or something that they chose to do that works out for us.”
When Cole plays Resident Evil 6, he holds down the button that brings up the map the entire time. By pressing forward while the map is displayed, his character is automatically led to the next objective. The workaround led to an epiphany he had while testing a prototype of The Last of Us Part II in Naughty Dog’s studio.
“I broke a window and then spent probably five minutes trying to get through it,” he explained. “Because jumping through a window is a precise thing. It basically means you have to line your character up precisely with the window and only then will your jump button actually bring you through. Otherwise, you just jump in place.”
To alleviate the problem, the developers incorporated something called traversal assistance. At certain trouble spots, hard-coded triggers predict users’ intentions and automatically perform actions — such as jumping through a window or over a hole. The feature, available as a custom configuration option or the default setting for users who select the game’s accessibility mode, is one of the reasons Cole said he’s been able to play the game to completion, without human assistance, six times.
The First Blind-Accessible Side Scroller Uses Directional Sound Panning
Liam Erven is a 36-year-old self-employed audio game designer and accessibility advocate who has been blind from birth with a rare condition called Norrie disease. When he was four years old, his parents bought him an NES console as a birthday present, and he quickly became one of the best Mortal Kombat and Track & Field players on his block in Des Plaines, Illinois. He’d rent games from Blockbuster and invite his friends over to play them, mastering technical moves or developing workarounds like aiming the Duck Hunt gun at a lightbulb to fool the sensor into thinking it was shooting ducks.
“When Mortal Kombat came out, I was all over that game,” he said. “And people in the neighborhood didn’t want to play me because I got so good at it. I could beat everybody. Because I memorized all the moves and really got good at it. And I just wasn’t fun to play with.”
As reported in Scientific American, there is mounting evidence that, in the absence of vision, people who are blind from a young age — for instance, Stevie Wonder and Ray Charles — often develop heightened auditory and tactile abilities. Theories differ as to why that is, but one emerging explanation is that the brain rewires itself. As writer Mary Bates noted in the article, brain imaging studies on cross-modal neuroplasticity show “the visual cortex in [people who are] blind is taken over by other senses, such as hearing or touch, and contributes to language processing.”
This may be partially why Erven (and Cole) excelled at Mortal Kombat from an early age. But despite Erven’s success with games involving relatively fixed areas of action, there were other games, like Super Mario Bros., he struggled to navigate and experienced largely vicariously by listening to his friends.
“I think I invented the playthrough,” he said, referencing games played by professional gamers with live-running commentary. “I really do. Because people were playing games for me. It was like the precursor to just going on YouTube and typing in Mass Effect and looking up a playthrough of it.”
Though Erven relished this social aspect of gaming, he wanted to make side scrollers like Super Mario Bros. more accessible to people who were visually impaired. As a teenage student at Maine West High School, he took a Visual Basic 6 programming class and developed a vertically-oriented shooting game somewhat analogous to Space Invaders, but with audio cues to distinguish falling objects. When he showed the finished design — a blank screen — to his programming teacher, the instructor wasn’t amused and pointedly asked Erven why there weren’t any graphics.
“So why can’t we put all of our audio to the side? Why can’t things in front of me be in my right ear? And things behind me be in my left ear?”
“I had to explain to him that my whole life I’ve never been able to partake in a game that way,” Erven said.
If anything, the teacher’s chilly reception only intensified his desire to develop games on his own. Some years later, he designed and coded a game called Super Liam, an homage to his love-hate affair with Super Mario Bros. The key insight for how he could create an accessible knock-off arose one night while he was lying in bed.
“So my idea was, well, if the game is a side scroller, obviously, you’re going from left to right on your television. That’s the way Mario works. So why can’t we put all of our audio to the side? Why can’t things in front of me be in my right ear? And things behind me be in my left ear?”
The game took a year and a half to code and was completely lost at one point when Erven’s hard drive crashed. But the effort was worth it. The final version included multiple levels, character voices Erven self-recorded, and enemies whose actions were created using object-oriented programming; a coding technique that groups items and avatars by inherited characteristics, using recycled command strings to expedite the development process.
But the stereo-panned, bi-directional audio cues were the real breakthrough: the first of their kind for a horizontally scrolling platform game.
“I wasn’t the first one to make an audio game by a longshot, there were people making audio-based games for DOS, there were all the text adventures, Infocom stuff, Sierra Entertainment, Zork, all those things. But I hold the distinction of making the first-ever blind-accessible side scroller,” Erven said.
The Evolution of Audio Games
Since Erven’s accessible reinterpretation of Super Mario Bros., audio games — and audio cues — have come a long way.
Modern games like Mortal Kombat 11, Cole told me, use sound panning to indicate not only from which side of the screen an enemy is coming, but how far they are from the player. Screen-reading apps like Microsoft Narrator for Xbox let users play visual novels like Walkerman, which don’t require navigation, by activating a self-voicing mode that reads the text aloud.
As accessibility features have improved, large publishers like Ubisoft and Electronic Arts, as well as smaller indie studios, have grown more daring with their designs. There are point-and-click adventure games with audio-only mechanics and shooting games that improve the user experience for players who are blind or have low vision by limiting the targets to a horizontal axis. Most gaming consoles now support eight-channel surround sound, and the PlayStation 5 goes even further, simulating sounds in a 360-degree radius to alert gamers to dangers in all directions.
“Interest in making audio games is growing,” Cole said. “And I think that’s because of games like The Last of Us Part II or maybe just because the conversation about accessibility itself is growing.”
In the audio game Blind Drive for iOS and Android, for instance, which Cole describes as “an over-the-top, silly, drive-against-traffic experiment,” the protagonist puts on a blindfold and attempts to drive toward oncoming cars. The game is incredibly immersive; though haptic feedback, the controller rumbles when the car encounters bumpy roads, vibrates when a player’s cell phone rings and shakes violently during accidents. It even pulses in time to the beat of songs being played on the driver’s radio.
Cole is equally excited about The Vale: Shadow of the Crown, an audio game developed by Falling Squirrel and Creative Bytes Studios and scheduled for release on PC and Xbox One later this year. The medieval combat game stars a sword-wielding hero who is blind and has been exiled from the kingdom by his older brother who has assumed the throne. Narration by acclaimed voice actors and three-dimensional acoustic cues are aspects of the game Cole, who has seen early demos, is particularly eager to see fully fleshed out in the release.
Accessibility Moves to Triple-A Games
Meanwhile, Erven has been working behind the scenes on several ideas to improve audio games recently released or now in development. A former software engineer at Horizons for the Blind, where he wrote code to translate phone bills into condensed, printable Braille formats, Erven designs audio games and features, and records himself playing video games on a Patreon-supported YouTube channel.
“It was a relatively unknown thing, you know, a guy who is blind is going to play video games. And so I started demonstrating various games for people,” he said. “I was never approached by Rockstar Games. I mean, if they’d like to, that would be fantastic.”
But other game studios have taken notice of the channel. The indie startup myTrueSound, based in Finland, even offered Erven a job as a lead programmer, but he turned it down because their game engine, Unity, wasn’t accessible.
“The problem is Unity is inherently inaccessible for users who are blind because it’s a lot of drag and drop; it’s a lot of layering. On the editor, they have claimed that they’re working on it, and that they’ve hired somebody. But who knows when that’ll bear fruit,” Erven said.
It hasn’t stopped him from working on games on his own.
“Forza, who Microsoft now owns, created this series of racing games. And so my first project was: Can I take telemetry data from the car’s dashboard in Forza Motorsport and create something that tells a driver what is actually going on? Where is the car traveling? Where is it in the race? Stuff like that.”
By mining data from the dashboard, he built a console application that detected and narrated a car’s speed, current lap and position in the race. Then he added a chirping signal to create an auditory feed that worked as a sort of compass, providing sound-based directional information.
“So the higher the [pitch of the] beep was, the closer to 360 degrees you were. That way, you could figure out, ‘Okay, I’m traveling west, I’m traveling northwest.’ It was far from perfect, but I wanted to see if it would work,” he said.
“My first project was: Can I take telemetry data from the car’s dashboard in Forza Motorsport and create something that tells a driver what is actually going on? Where is the car traveling?”
Though the idea was never picked up by Microsoft Game Studios, it drew interest from Alan Hartman, vice president of Forza, and led Erven to his next guerrilla programming project: Grand Theft Auto V for PC. Using a script hook, which makes high-level functions accessible in multiple programming languages, Erven developed a downloadable accessibility mod for 64-bit versions of Windows. According to an online description, the mod allows players to access location and direction information, teleport to different areas of the game, “spawn” vehicles and train with “god mode” and vehicle “god mode.”
Erven’s goal was not to have the features formally implemented by Rockstar Games, the game’s publisher, but to make the game more approachable to players who are blind and leverage his YouTube platform to raise awareness among developers about the potential of audio as a semantic tool to provide users information about their surroundings.
“Firstly, can I have the game tell me where I actually am? And then I did. And then, can I have the game tell me what direction I’m walking in? And then I did. And then, I kind of went, well, what else can I do with the game?”
For his coup de grâce, Erven developed a set of unique tones to describe the height of a player’s target.
“When you go into targeting mode, it will actually raise and lower the targeting icon and make a sound to give you an idea of how high or low the target is. There’s another tone that tells you if you’re targeting a vehicle or a human, or maybe an object,” he said. “So you can actually use this to shoot down helicopters, which is kind of fun if you’re into that sort of thing.”
Though excited by recent progress in audio game development, Cole believes all games — even big-budget, triple-A role-playing games, sports games and first-person shooters — should be accessible to users who are blind.
Moving toward that vision won’t be easy. Studio budgets for accessibility tend to be small, Erven said, and few development teams include programmers who are visually impaired. Moreover, many industry game engines aren’t well adapted to screen readers.
Yet advocacy from individuals like Erven and Karen Stevens, director of accessibility at Electronic Arts, who is deaf and has worked on visual and auditory accessibility features of games like Madden NFL, appears to be moving the needle.
Stevens helped steer the course for recent iterations of the game, such as Madden NFL 18, which, according to an online accessibility guide, includes “assistive rumbles,” “enlarged on-field icons,” configurable gameplay options, and volume controls for commentary, field noises and menu selections. In fact, at a UX Summit of the 2018 Game Developers Conference in San Francisco, Stevens gave a talk called “AAA Gaming While Blind,” describing how gameplay without sight is possible on Electronic Arts titles like UFC, Need for Speed, Madden NFL and NHL.
Given the potential of these games to reach a large, untapped market of marginalized gamers, other studios may soon be following suit. The National Federation of the Blind reported in 2016 that nearly 7.7 million people between the ages of 16 to 75 had a visual impairment. And data from Statista shows that, in 2018, 66 percent of the 327 million people living in the United States played video games.
“We still consume content,” Cole said. “We listen to games we can’t play. We watch anime. We watch TV shows. So to have games that are accessible and also contain the things everyone else is doing, to join the gaming community without having to do workarounds and things like that. That’s important to me.”
Could Audio Description Services Enlarge the Storytelling Power of Games?
The next stride forward could come in the realm of audio description services. Rhys Lloyd is the studio head of Descriptive Video Works, a Vancouver-based affiliate of Keywords Studios, a company that creates audio descriptions for television, film, corporate advertisements, and most recently, video games.
Audio descriptions could play a major role in the evolution of accessible gaming, Lloyd told me, beginning with voice-over clips for promotional trailers, such as one Descriptive Video Works created for Ubisoft’s Assassin’s Creed Valhalla.
The next step — and something that’s also already happening — is the creation of audio descriptions for broadcasts of live gaming events and ceremonies, such as the Game Awards and Summer Game Fest. Cole’s Twitch feed during the latter, for which the studio provided an audio description, is an example of what could eventually become common in esports.
Lloyd envisions the format as something similar to what Keywords Studios currently provides for televised broadcasts of NCAA college football.
“Intuitively, you might think, well, college football, the announcers describe everything,” Lloyd said. “They tell you who’s got the football, who’s throwing it to whom, who tackles, etcetera. But there’s a lot of information that is missing, maybe because the announcers let the graphics on the screen tell you a story they don’t necessarily call out, whereas audio description providers fill in the gaps.”
They could perform a similar service in esports tournaments, he said, where the action is often fast-paced and not always clear from the broadcaster’s narration alone.
The real shift, though, could be felt in actual gameplay. Many games, Lloyd said, have locked cinematic moments that link together game sequences or signal setting changes, like when a character opens a door to enter a new room. These set pieces could open the door to evocative descriptions of a world and its features — similar, he said, to acoustic setting descriptions audio providers insert in dialogue breaks in television and film.
“Could you actually record everything you want to say about the physical environment of the game, or the characters that you, as a player, are interacting with?” Lloyd proposed.
Even further afield, these descriptions could influence the interactive moments of adventure games and RPGs. There is a concept in the field of audio description, Lloyd told me, called “extended ad.” Instead of splicing an audio clip to fit a video’s length, the process works in reverse: The video is looped to match the length of an extended narrative description. The technique is used, for example, in cruise ship ads: “You’re going through the Norwegian fjords, please describe the fjords so people know how beautiful it is,” Lloyd said, referring to a client’s hypothetical request.
But imagine if players could access such moments in a game? And at their discretion?
“It feels like there was a lot of pushing in the door. And finally the doors started to open, and open into different areas. And, honestly, I think The Last of Us Part II was a really [defining] moment.”
“This is very much the same conversations I’m having with people at the development level of studios,” Lloyd said. “You’re giving people a little control. It’s not so much toggle on and off, but choice at almost the event level of whether you hear a description or not.”
There are still plenty of barriers to getting there, not least the ever-present challenge of squeezing time into already tight development schedules. And that’s to say little of the technical challenges of embedding human recordings into modern games with near-limitless machinations.
“Right now something like the service we provide wouldn't work for the Fortnites of the world. The game is too chaotic. It’s too dynamic for something like what we do,” Lloyd said.
But fast forward a few years and that might not be the case.
“If you combine services like ours — where we can take the visual and provide a contextual and evocative description of it — with the AI text-to-speech engines that are powering the games, you could potentially have a truly dynamic experience that allows a person who is blind to be in the same room as a sighted person and be on the exact same Fortnite adventure. To have virtually the same experience of it,” he said.
Lloyd, by his own admission, is a bit of a hypebeast for audio description services. A big part of his job is to talk up the technology’s potential to game studios. Still, over the past 18 months, the tenor of those conversations has changed.
“It feels like there was a lot of pushing in the door,” he said. “And finally the doors started to open, and open into different areas. And, honestly, I think The Last of Us Part II was a really [defining] moment.”
8 Lessons for Designing Games for People Who Are Blind
To continue to move the industry forward, Cole and Erven shared eight key lessons for designing games for people who are blind or visually impaired.
8 Lessons for Designing Games for People Who Are Blind
- Move away from optical character recognition.
- Improve navigation and traversal assistance.
- Use auditory cues to define a player’s position and relative distance from nearby objects, enemies and hazards.
- Provide full text-to-speech narration.
- Increase studio budgets for accessibility.
- Make game engines more accessible to developers who are blind or visually impaired.
- Create options to play games with auxiliary devices.
- Include adjustable difficulty modes.
1. Move Away From Optical Character Recognition
Optical character recognition does not pull text from a game’s source code, it scans a video image to figure out what the text says. In some cases, it works well. But elegant fonts, richly detailed graphics, bright colors and animation — the features many modern gamers admire — cause it to perform poorly.
2. Improve Navigation and Traversal Assistance
Users who are blind or have low vision often rely on sound cues for pathfinding. Unless all items, enemies and objects can be tracked and pursued with computer-assisted navigation options, certain aspects of a game can be frustrating or unplayable. This is particularly important for areas of a game, such as windows or terrain gaps, that can require highly precise positioning to perform successfully.
3. Use Auditory Cues to Define A Player’s Position and Relative Distance from Nearby Objects, Enemies and Hazards.
Many modern games use sound panning to indicate not only from which side of the screen an enemy is coming, but how far they are from the player. Both sound duration and pitch can orient players to their surroundings. Providing these options in accessibility presets at the start of a game can introduce players to setting configurations important to their enjoyment of a game.
4. Provide Full Text-to-Speech Narration
All aspects of the game that are critical to completing it should be legible to users who are blind or have low vision. Some game systems, like the PlayStation 5, offer self-voicing, text-to-speech options built on proprietary game engines. Pre-recorded audio descriptions can also be created to describe menus, items and key objectives.
5. Increase Studio Budgets for Accessibility
Several companies are taking strides to make gaming more accessible. For instance, Microsoft built Narrator, a screen reader that’s built into Windows 10 and provides voice assistance on Xbox One. And Electronic Arts’ Madden NFL 17 was released with low-vision options and tested by in-house developers who have macular degeneration. But budget constraints for accessibility at large software companies and game studios remain obstacles to making triple-A games more accessible to users who are blind or have low vision.
6. Make Game Engines More Accessible to Developers Who Are Blind or Visually Impaired
Many games are developed on engines with drag-and-drop interfaces that are inaccessible to developers who are blind. This prevents skilled programmers from entering jobs in the field. Plug-in tools are beginning to address this issue, but the industry awaits a fully accessible development solution. Audio game engines like Ren’Py and Frostbite, an engine for large-scale multiplayer games created by DICE and used by Electronic Arts developers, suggest that workable solutions are available and could be leveraged by more studios.
7. Create Options to Play Games With Auxiliary Devices
Offering users the ability to play a game from a second device can improve gameplay. Using a keyboard to navigate a game can make it easier for users to input information quickly and provide access to semantic cues like ARIA (Accessible Rich Internet Applications) landmarks that a screen reader can detect and read to let a user know what they are encountering on a page — like a menu or search field, for instance. Selection options on a browser window on a phone or computer can also be useful.
8. Include Adjustable Difficulty Modes
Many studios are releasing games with adjustable difficulty modes. Perhaps the enemies move slower or a player is able to momentarily turn invisible or store more ammunition in a weapon. These settings give more control to users and make games more playable for individuals who require more time to interpret their surroundings.
Correction: An earlier version of this story incorrectly used Brandon Cole’s last name in a passage about Liam Erven. It also incorrectly described the game Phoenix Wright: Ace Attorney as employing optical character recognition (OCR). OCR is a feature of a user’s screen reader, not the game itself. We regret the errors.
