Making Hearing Selective Again: Semantic Hearing Could Transform Noise-Canceling Headphones

Friday - November 10 2023, 09:46 UTC - 1 year ago

Researchers from the University of Washington have developed a Semantic Hearing system which, when applied to noise canceling headphones, would allow wearers to select which sounds they would like to hear and blocks out others. The system has potential application to those with hearing loss, as well as those who require focused listening for their career, such as military or engineering professionals.

Future noise-canceling headphones could let users opt back in to certain sounds they’d like to hear, such as babies crying, birds tweeting, or alarms ringing. The technology that makes it possible, called semantic hearing, could pave the way for smarter hearing aids and earphones, allowing the wearer to filter out some sounds while boosting others. The system, which is still in prototype, works by connecting off-the-shelf noise-canceling headphones to a smartphone app .

The microphones embedded in these headphones, which are used to cancel out noise, are repurposed to also detect the sounds in the world around the wearer. These sounds are then played back to a neural network, which is running on the smartphone; then certain sounds are boosted or suppressed in real time, depending on the user’s preferences. It was developed by researchers from the University of Washington, who presented the research at the ACM Symposium on User Interface Software and Technology (UIST) last week .

The team trained the network on thousands of audio samples from online data sets and sounds collected from various noisy environments. Then they taught it to recognize 20 everyday sounds, such as a thunderstorm, a toilet flushing, or glass breaking. It was tested on nine participants, who wandered around offices, parks, and streets. The researchers found that their system performed well at muffling and boosting sounds, even in situations it hadn’t been trained for .

However, it struggled slightly at separating human speech from background music, especially rap music. Mimicking human abilityResearchers have long tried to solve the "cocktail party problem"—that is, to get a computer to focus on a single voice in a crowded room, as humans are able to do. This new method represents a significant step forward and demonstrates the technology’s potential, says Marc Delcroix, a senior research scientist at NTT Communication Science Laboratories, Kyoto, who studies speech enhancement and recognition and was not involved in the project .

"This kind of achievement is very helpful for the field," he says. "Similar ideas have been around, especially in the field of speech separation, but they are the first to propose a complete real-time binaural target sound extraction system." "Noise-canceling headsets today have this capability where you can still play music even when the noise canceling is turned on," says Shyam Gollakota, an assistant professor at the University of Washington, who worked on the project .

"Instead of playing music, we are playing back the actual sounds of interest from the environment, which we extracted from our machine-learning algorithms." Gollakota is excited by the technology’s potential for helping people with hearing loss, as hearing aids can be of limited use in noisy environments. "It’s a unique opportunity to create the future of intelligent hearables through enhanced hearing," he says .

The ability to be more selective about what we can and can’t hear could also benefit people who require focused listening for their job, such as health-care, military, and engineering professionals, or for factory or construction workers.