Neurophysiologists and engineers at the University of California have improved a system that converts human brain activity into coherent sentences. Even paralyzed people who have completely lost their speech can now use it. The research was published in the journal Nature Communications (https://www.nature.com/articles/s41467-022-33611-3) and also at the website of the University of California (https://www.universityofcalifornia.edu/news/clinical-trial-aims-restore-speech-those-who-lost-it-due-brain-injuries).
Scientists have perfected their own system with a brain implant. The implant reads the activity of neurons in the motor areas and then uses deep learning and a language model to translate the information into full sentences.
The development and testing of the new system involved a man with anarthria, i.e. absence of speech related to neuromuscular damage. However, the patient retained the ability to produce some sounds, including moaning. To optimize the model, the researchers asked the patient to try to pronounce (either mentally or aloud) a word or letter displayed on a screen. In another task, he had to try to pronounce a phrase displayed on the screen or an arbitrary phrase.
The resulting system was able to decode a vocabulary of 1152 words with an error rate of 6.13% for letters and 10.53% for words. Out of 150 sentences, the system recognized 70% without errors, the average decoding rate was 29.41 letters or 6.84 sentences per minute. This was faster than the assistive device the patient would normally use.
In the previous version of the system, the vocabulary was only 50 words and the participant always had to try to pronounce the correct words out loud. Now, the model not only increased the vocabulary by 23 times, but also learned to work when silently saying the words to himself. In addition, data simulations showed that the system’s vocabulary can be expanded to 9170 words, allowing patients to communicate fluently on various topics.
However, not all speech impaired people will be able to use the system. For example, it will not suit patients with Broca’s aphasia, in which the part of the cerebral cortex responsible for the motor organization of speech is affected.
