A brain implant has decoded speech and hand gestures at the same time in people with paralysis, and used both to run a virtual avatar. The study, published Sept. 14 in Nature Neuroscience, tested three participants and reports real-time accuracy of 66% to 85% for the one participant who did the full task.
The paper, “Simultaneous speech and gesture decoding for multimodal communication in paralysis”, comes from the lab of Edward Chang, a neurosurgeon at the University of California, San Francisco. Samantha C. Brosler is the first author. The National Institutes of Health announced the work the same day.
How the Speech and Gesture Brain Implant Works
Earlier brain-computer interfaces mostly decoded speech or movement, not both. This team used a 253-channel electrode array that sits on the surface of the brain’s motor cortex. The contacts are 1 millimeter wide and spaced 3 millimeters apart. A connector that passes through the skin links the array to a computer, so the system in the study is wired. The team plans to test a fully implantable wireless version, the NIH said.
The participants are enrolled in a trial called BRAVO, short for BCI Restoration of Arm and Voice, registered as NCT03698149. The paper says it is a single-center early-feasibility study that began in November 2018. Data for this paper were collected from September 2022 to November 2025, a span of 38 months. The FDA approved the investigational use of the implant, and the UCSF institutional review board and the National Institute on Deafness and Other Communication Disorders oversee it.
Each participant tried to say phrases and make gestures, such as a wave or a thumbs-up, while the array recorded activity. Machine-learning decoders turned the signals into text for speech and into movements for a personalized full-body avatar. Chang said in the NIH release that “Conversation is about much more than the words being spoken”.
Who Took Part in the Brain-Computer Interface Study
Three adults took part. The paper identifies them as Bravo-1r, Bravo-3 and Bravo-6. Bravo-1r was 36 at enrollment and has paralysis from a brainstem stroke at age 20. Bravo-3 was 47, with paralysis from a brainstem stroke at age 30. Bravo-6 was 60 and has ALS, diagnosed at 56. All three kept intact cognition and cannot produce intelligible speech. Their native languages are Spanish, English and German.
The three do not contribute equally to the results. Bravo-3 withdrew before the avatar experiments, so every avatar and simultaneous result comes from Bravo-1r and Bravo-6. The authors list that as a limit. They also say the group is too small to tell how the cause of paralysis or leftover movement affected results.
The participants received $50 per session, with an annual cap of $5,000. At that rate, the cap equals 100 sessions a year for each person. That arithmetic comes from the paper’s compensation terms, and the paper does not report how many sessions each participant completed.
What the Accuracy Numbers Show for Speech and Gesture Decoding
The vocabulary was small. Bravo-1r worked with four gestures and five phrases. Bravo-6 worked with 10 gestures and 10 phrases, which can combine into 100 pairs. Chance performance was 9.1% for Bravo-6, because each task offered 10 options plus a rest state.
In the real-time copy task, where Bravo-6 attempted a gesture and a phrase together, the decoder got gestures right 66.0% of the time and speech right 70.0% of the time. Against the 9.1% chance rate, that is about 7.3 times chance for gestures and 7.7 times for speech. Onsite! calculated those ratios from the paper’s figures.
In a conversation task, Bravo-6 reached 85.0% for gestures and 75.0% for speech across five blocks. The paper’s confidence intervals are wide for these results: 70.0% to 100.0% for gestures and 55.0% to 90.0% for speech. Bravo-1r reached a median of 100% on both across three blocks. The paper says Bravo-1r’s simultaneous accuracy was assessed only offline, so that result did not come from live use.
Offline, with the full data set, Bravo-1r’s decoder scored 88.3% on gestures and 84.0% on speech. Bravo-6’s scored 68.8% on gestures and 77.5% on speech. The offline numbers use recorded data and do not show how the system behaves in live conversation.
Why Training on Both Together Mattered
The paper’s most practical finding concerns training. A decoder trained only on isolated gestures or isolated speech did not carry over cleanly to moments when the person tried both. Models trained on isolated attempts and simultaneous attempts together performed best across every setting.
The error tables show the gap. A decoder trained without simultaneous data wrongly fired on the opposite channel 30.6% of the time for Bravo-6 and 76.0% for Bravo-1r, meaning a speech attempt could trigger a gesture that the person did not intend. With training on both, that false-positive rate fell to 0.0% for both participants. Missed attempts, called false negatives, fell from 14.5% to 1.0% for Bravo-6 when tested on simultaneous trials, and from 34.9% to 4.5% for Bravo-1r.
The authors also tested whether the decoder could handle speech and gesture pairs it never saw in training. For Bravo-6, accuracy on unseen pairs matched seen pairs: 66.7% against 65.8% for gestures and 80.0% against 79.3% for speech. The differences were not statistically significant.
Limits and Next Steps for Multimodal Communication
The authors list several limits. The vocabularies are restricted, and the system needs to scale up to many more phrases and gestures. Simultaneous decoding was still imperfect. Movements of the limb on the same side of the body as the implant decoded less accurately than speech or movements on the opposite side. Bravo-6 also had brain shift after surgery, which complicated the exact placement of electrodes. The paper does not report latency, and the authors name reducing it as future work.
Debara Tucci, director of the National Institute on Deafness and Other Communication Disorders, said in the NIH release that the results give her hope that people with severe paralysis may recapture the full nature of human communication. The decoded speech appears as text on a screen. The authors describe one demonstration in which the avatar of a participant held a fully virtual conversation with an experimenter’s avatar. The authors deposited the dataset on Zenodo.
