Decoding Language from Brain Activity Non-Invasively with fMRI

Friday - May 5 2023, 16:59 UTC - 1 year ago

Researchers have successfully decoded language from brain activity non-invasively using fMRI scans, a feat that could facilitate communication in paralyzed individuals. The 'language decoder' was able to accurately capture and interpret auditory and imagined speech. The technology opens up new avenues for researchers to explore the underlying pathways between brain activity and language.

Language and speech are how we express our inner thoughts. But neuroscientists just bypassed the need for audible speech, at least in the lab. Instead, they directly tapped into the biological machine that generates language and ideas: the brain.

Using brain scans and a hefty dose of machine learning, a team from the University of Texas at Austin developed a "language decoder" that captures the gist of what a person hears based on their brain activation patterns alone. Far from a one-trick pony, the decoder can also translate imagined speech, and even generate descriptive subtitles for silent movies using neural activity.

Here’s the kicker: the method doesn’t require surgery. Rather than relying on implanted electrodes, which listen in on electrical bursts directly from neurons, the neurotechnology uses functional magnetic resonance imaging (fMRI), a completely non-invasive procedure, to generate brain maps that correspond to language.

To be clear, the technology isn’t mind reading. In each case, the decoder produces paraphrases that capture the general idea of a sentence or paragraph. It does not reproduce every single word. Yet that’s also the decoder’s power.

"We think that the decoder represents something deeper than languages," said lead study author Dr. Alexander Huth in a press briefing. "We can recover the overall idea…and see how the idea evolves, even if the exact words get lost." .

The study, published this week in Nature Neuroscience, represents a powerful first push into non-invasive brain-machine interfaces for decoding language—a notoriously difficult problem. With further development, the technology could help those who have lost the ability to speak to regain their ability to communicate with the outside world.

The work also opens new avenues for learning about how language is encoded in the brain, and for AI scientists to dig into the "black box" of machine learning models that process speech and language.

"It was a long time coming…we were kinda shocked that this worked as well as it does," said Huth.

--- Decoding Language --- .

Translating brain activity to speech isn’t new. One previous study used electrodes placed directly in the brains of patients with paralysis. By listening in on the neurons’ electrical chattering, the team was able to reconstruct full words from the patient.

Huth decided to take an alternative, if daring, route. Instead of relying on neurosurgery, he opted for a non-invasive approach: fMRI.

"The expectation among neuroscientists in general that you can do this kind of thing with fMRI is pretty low," said Huth.

There are plenty of reasons. Unlike implants that tap directly into neural activity, fMRI measures how oxygen levels in the blood change. This is called the BOLD signal. Because more active brain regions require more oxygen, BOLD responses act as a reliable proxy for neural activity. But it comes with problems. The signals are sluggish compared to measuring electrical bursts, and the signals can be noisy.

Yet fMRI has a massive perk compared to brain implants: it can monitor the entire brain at high resolution. Compared to gathering data from a nugget in one region, it provides a birds-eye view of higher-level cognitive functions—including language.

With decoding language, most previous studies tapped into the BOLD signals from single words. But Huth thought that was too restrictive.