For the last four years, Daniel Belquer has been working on a device that translates live music for people who can't hear.
While most people consider music an auditory experience, Belquer, who completed his Master’s thesis on listening, had already been toying with the physical aspects of sound, even building glass speakers to incorporate vibration into his music compositions.
Not Impossible Labs, an LA-based social innovation company, wanted him to take that experience to the next level. They wanted him to help translate the joy of rocking out to a live band into a physical experience for the Deaf community. To do so, Belquer has delved deep into the world of sound and vibrations, worked closely with members of the Deaf community and conducted hundreds upon hundreds of user tests.
Last year, Belquer and his team unveiled the latest prototype of its Music: Not Impossible vibratory wearable. The system connects to the soundboard through a long-range radio signal, and then translates the music into a series of vibrating pulses sent to 24 points across the body, creating a “surround-body-experience.”
Belquer detailed how the device transformed from a proverbial napkin sketch into a working prototype, and how Not Impossible Labs plans to scale it.
- A DIY approach built out cheap objects is often the best way to go for early prototypes. This gives you the freedom to experiment and break things quickly.
- Seek feedback and advice from people in the community for whom you are building the technology.
- Test. Test. Test. Trying out prototypes on yourself and other people creates opportunities to see flaws in the design and find avenues to improve.
- Embrace simplicity. No matter how clever your technology, it won’t be embraced if people can’t figure out how to use it.
- Innovation doesn’t have to mean creating everything from scratch. Compatibility with more commonly used software or devices can allow your tool to reach a wider audience.
Understanding the science behind music
Every project at Los Angeles-based Not Impossible Labs begins with what Ebeling calls an absurdity. The term is Ebeling’s shorthand to describe challenges a person experiences that underscore a larger societal need.
This can be a skateboarder robbed of his eyesight and thus his ability to skate, a once-adventurous graffiti artist paralyzed and bed-bound due to ALS, or the societal challenge of people going hungry. In each case, Ebeling and his team take it upon themselves to find a solution with technology.
The company has since created an eyewriter — a device that relays the movement pattern of a person’s eye to a projector system — to allow the graffiti artist to write on walls again, a text messaging platform that connects those lacking food sources and sonic technology that makes it possible to skateboard without eyesight.
When it comes to live music, a person who is deaf often has limited means to join in on the experience. They can either stand close to the speakers to feel the music vibrations or hold a balloon or water glass to feel it. However, in a crowded venue, speakers can be blocked by large crowds and equipment and holding a glass of water isn’t always easy.
To better understand this experience and translate it into technology, the team brought in neuroscientist Dr. David Putrino, who runs a rehabilitation innovation lab at Mount Sinai.
Putrino explained the science behind how the brain engages with music. Since music comes down to vibrations, the auditory cortex isn’t the only part of the brain responsible for interpreting sounds. The hippocampus, cerebellum and amygdala all work together to translate music vibrations into the emotional, physical and memory responses you experience while dancing to Beyonce or grooving to The Grateful Dead.
That insight provided the foundation Not Impossible Labs needed to begin developing prototypes.
“Dr. Putrino gave us valuable input on how the skin works and what are the most sensitive areas, and why you can’t just blast different vibrations into a person’s back,” Belquer said. “He had this vision from the start, and understood what boxes needed to be checked to serve the deaf, while also being a cool experience for people.”
Developing a wearable device, starting with old-school pagers
At this point, Not Impossible Labs had a theory that music could be translated into vibrations on the skin, and a sketch for how to make that happen. It was time to start experimenting.
Without much research available on sound vibration technology, Belquer and his team relied heavily on testing DIY prototypes.
“The DIY approach comes in handy because you can try things out without worrying that you will damage a $3,000 piece of equipment,” Belquer said. “Things burn and stuff explodes in the beginning.”
For the first prototype, Belquer turned to a device that last had its buzzworthy moment in a 1990s music video featuring Notorious B.I.G. — pagers. Armed with pager buzzers, wires, wire cutters and hot glue, Belquer patched a device together. He then plugged it into the headphone jack of the computer and connected it to the sound files on the music platform Ableton Live to test it.
“The DIY approach comes in handy because you can try things out without worrying that you will damage a $3,000 piece of equipment.”
The device buzzed, but it came nowhere near replicating the music experience. Still, it gave Belquer a starting point, and the knowledge that he needed to purchase actuators that could distinguish finer vibrations.
“The motors weren’t good enough, they weren’t fast enough, they were noisy and they consumed a lot of power,” Belquer said. “But you learn a lot from the first prototype, and then you move on to the next version, where you’ll find other issues.”
Other prototypes included:
- A complex eight-by-eight matrix of motors hand sewn into pouches and worn across the chest.
- A pair of pants that he turned into a strap with pouches containing motors.
- A baseball catcher’s vest with motors attached to it.
But the team also needed to find a way for the motors to connect to the music wirelessly to give concertgoers the freedom to move around and dance. That meant the motors needed to have zero-latency between sound and vibration. Belquer looked into connecting the device via Bluetooth and XDS before settling on Wi-Fi, which had the benefit of being cheap and easy to set up.
The devices then communicate with the venue’s soundboard through Open Sound Control, Belquer said. While the process worked in a vacuum, Not Impossible’s team struggled with Wi-Fi connectivity when they tried hooking it up in a crowded room.
Still, the trial and error process of those early prototypes allowed the team to home in on what worked and didn’t.
Also critical to that process was the feedback they received from the Deaf community and other users in testing.
lessons learned from 600-plus product tests
From the beginning, Not Impossible Labs set out to create a device that would bridge the divide between the hearing abled and the Deaf community. The device needed to be easy to use, and its output needed to be immediately understood as music.
Since the device’s inception, Belquer estimates that they’ve completed more than 600 tests of it to fine-tune its capabilities and address the needs of the Deaf community.
To better understand that perspective, Not Impossible Labs brought in Mandy Harvey, a deaf singer/songwriter, early on to test the device and offer feedback. They also tested it with deaf composer Jay Zimmerman, and at a variety of small shows, including SXSW in 2015.
Most people can’t perceive tiny variations in vibration without proper training, but the deaf and Mandy can. She helped the system get to a more refined level.”
Harvey helped Not Impossible better understand where to focus the vibrations for the best user experience. They learned that if they went too high, it felt uncomfortable for women. If they went too low, it resonated too much with the stomach. Early testers also helped them fine-tune the sounds and vibrations to create a range of expressions.
“Mandy was very adamant on making the vibrations more subtle, precise and delicate,” Belquer said. “Most people can’t perceive tiny variations in vibration without proper training, but the deaf and Mandy can. She helped the system get to a more refined level.”
Meanwhile, ASL music interpreter Amber Galloway held a workshop to help the team better understand the Deaf community. Her insights about that community helped Belquer connect to the project on a deeper level, and informed the features he designed, he said.
“It became more of a crusade for me personally,” Belquer said. “I didn’t want to just include the Deaf community as users, I wanted to empower them as creators.”
Explore other use cases to improve adoption
After four years of trial and error, Not Impossible Labs unveiled its ninth prototype at a Greta Van Fleet show in the Las Vegas music festival, Life Is Beautiful.
The latest prototype involves 24 different points of vibration that are divided among the torso, back, wrists and ankles. Each actuator has a corresponding LED that glows when in use. The motors are built with a long-range radio signal that allows hundreds of devices to communicate with the software and interpret the music at once.
The motors have then been calibrated to interpret notes across five octaves, with 64 steps between each half-step of a note, totaling 3,900 frequencies — in essence replicating the range of sound most of us experience with our ears.
At the Life Is Beautiful show, Not Impossible Labs tested the devices on a half deaf, half hearing-abled crowd. Everyone wore the vest and rocked out to the same experience for the first time.
“Half the people in the room were deaf and half could hear, and you couldn’t tell the difference,” Belquer said. “This was a beautiful moment for the project where the vision was finally realized.”
But this is only part of Not Impossible Labs’ mission. The company set out to break down the barriers the Deaf community experiences in music, which means the technology needs to be adopted and scaled.
That accessibility has continued to influence both the final design and software that powers the device. The engineering team has created a modular platform on Ableton Live (a software most musicians have access to), in which users can easily make changes to the vibration patterns, LED colors and more.
“My dream is to have this being used by artists and creators so that people can have experiences through this technology without any concerns about hearing level — or sight for that matter”
They plan to launch a Creator Network in December to train and encourage artists across multiple disciplines to play with this software and design new experiences. Already, some artists are working to incorporate it into a movie experience and for a show at the Museum of Moving Images in Queens, New York.
For concerts, the team is working with venues to hone the charging process, storage and delivery of the devices. Their goal is to iron out any difficulties through additional testing before they officially roll it out.
“My dream is to have this being used by artists and creators so that people can have experiences through this technology without any concerns about hearing level — or sight for that matter,” Belquer said. “I would like this to be a powerful artistic tool for everyone.”
Belquer sees the device more like a new instrument than a musical add on. To that end, he’s already putting together ideas for ways to artistically express himself through it. But more importantly, working on this project has helped Belquer tap into a different mindset of creativity.
“I have been tainted by this virus of accessibility,” Belquer said. “Even stuff I’m thinking about developing, I can’t help but think about it from an inclusive perspective. Especially the deaf, I cannot leave them out anymore.”