Recently, the top academic journal Nature published a landmark study titled "High-performance brain-to-text communication via handwriting" in the form of a cover article by researchers from Stanford University, Brown University, Harvard Medical School and other teams. The study returned to the ancient way of expression - handwriting, and for the first time decoded the neural signals related to writing and displayed them in real time.
The research team combined artificial intelligence software with a brain-computer interface device and worked with a paralyzed patient who had the device implanted in his brain to let the patient imagine that he was holding a pen and "trying" to write on a piece of lined paper, as if his hand was not paralyzed. And quickly converted the man's handwriting intention into text on the computer screen. "This brain-computer interface is designed for people with 'Locked-in Syndrome', who have paralyzed almost all of their voluntary muscles and cannot speak or communicate." Dr. Francis R. Willett, the corresponding author and first author of the study, said, "Imagine if you can only move your eyes up and down but can't move. Such a device can allow you to enter your thoughts at a speed comparable to normal handwriting or typing on a smartphone."
Willett is a research scientist at the Stanford University Neurorestoration Translational Laboratory, appointed by the Howard Hughes Medical Institute. His research focuses on improving brain-computer interfaces and understanding how the brain expresses and controls movement. In addition to Willett, the research is mainly directed by Stanford neurosurgeon Henderson and Krishna Shenoy, a researcher at the Howard Hughes Medical Institute at Stanford University. Willett is a member of Shenoy's team. Shenoy and Henderson have been collaborating on brain-computer interface research since 2005.
Nature also published a perspective article on the study by two researchers, Pavithra Rajeswaran and Amy Orsborn, from the University of Washington. "Although there is still a lot of work to be done, the study by Willett and his colleagues is a milestone that broadens the horizons of invasive brain-computer interface applications." They commented that the method in this study "brings neural interfaces that allow rapid communication closer to reality."
▍Find new ways for people to communicate faster
The brain-computer interface used in this study is for research purposes only and has not been approved for commercial use. Stanford’s Office of Technology Licensing has applied for patents on intellectual property related to the work of Willett, Henderson and Shenoy.
For the first time, researchers have decoded the neural signals that underlie the writing of letters, allowing them to be displayed in real time. (01:40) "Finding new ways for people to communicate faster".
How important is the brain-computer interface developed in this study for people who cannot speak normally? "If it is a brain-computer interface, Jean-Dominique Bauby may be able to write 18 words per minute." Willett told reporters.
In fact, people who lose the ability to move or speak for different reasons have different needs for brain-computer interfaces. People who have lost the use of their hands can still use devices with speech recognition and other software, while for those who have difficulty speaking, scientists have been developing other ways to help people communicate.
Current commercial assistive typing devices rely on the user being able to make eye movements or give voice commands. For example, an eye-tracking keyboard allows a paralyzed person to type about 47.5 characters per minute, which is slower than the 115 characters per minute for a person without impairment. However, these technologies are not suitable for people whose paralysis also impairs eye movement or speech.
So far, brain-computer interfaces for typing have not been able to compete with simpler assistive technologies such as eye trackers. One reason, Rajeswaran et al. point out in their perspective article, is that typing is a complex task. In English, we need to choose from 26 letters. It is also challenging to build a classification algorithm based on the user's neural activity to predict which letter they want to choose.
The most successful invasive BCI so far is also a study by Shenoy's team published in the journal eLife in 2017. In that study, three participants with limb paralysis, including T5 (a participant in this latest study), had a BCI implanted in their motor cortex and were asked to concentrate on moving a cursor from one key on a computer screen to another, and then focus on clicking that key.
In that study, the T5 set the record to date: transcribing displayed sentences at a rate of 40 characters per minute. But these invasive brain-computer interfaces, like noninvasive eye trackers, occupy the user's visual attention and cannot provide significantly faster input speeds.
If the 2017 study was similar to typing, the new study is similar to handwriting, which no one had thought of before. Willett wondered if it was possible to use the brain signals triggered by writing. "We want to find new ways for people to communicate faster." So the research team continued to work with T5, who was 65 years old at the time. He was paralyzed by a spinal cord injury in 2007 and had lost almost all movement below the neck.
▍“Brain-to-text” brain-computer interface
Willett et al.'s new approach requires a classification algorithm that can predict the 26 letters or five punctuation marks that a paralyzed user is trying to write, which is challenging because scientists cannot observe these intentions.
To overcome this challenge, Willett et al. redesigned a machine learning algorithm based on a machine learning algorithm originally developed for speech recognition. This allowed them to estimate when participants began to attempt to write a character based solely on neural activity. Based on this information, the research team generated a labeled dataset containing the neural activity patterns corresponding to each character. They used this dataset to train a classification algorithm.
“When we first began exploring the concept of handwriting BCIs, we had no idea whether attempted handwriting movements could still evoke strong and repeatable patterns of neural activity after years of paralysis.”
"What's exciting is that when we asked the participant to write different letters by hand, even though his hand had been paralyzed for more than a decade, we could still see clear patterns of neural activity, enough to even recreate the movements of the pen he imagined and the letters he wanted to write," said Willett.
When a study participant imagines writing a letter or symbol, sensors in his brain can sense the pattern of electrical activity and convert it into a handwriting trace. (00:12) To achieve accurate classification, Willett et al.'s classification algorithm also uses existing machine learning methods and an artificial neural network called a recurrent neural network (RNN), which is particularly good at predicting sequential data. Rajeswaran et al. mentioned in the opinion article that taking advantage of the power of RNN requires a large amount of training data, but this data is limited in neural interfaces because few users are willing to imagine writing for hours.
The research team solved this problem using a method called data augmentation, in which the neural activity patterns previously produced by the participants were used to generate sentences on which the RNN was trained. They also expanded their training data by introducing artificial variations in the neural activity patterns to mimic variations that occur naturally in the human brain.
In the study, T5 was also asked to concentrate on trying to write individual letters in an imaginary notebook with an imaginary pen. He repeated each letter 10 times, allowing the software to "learn" to recognize the neural signals associated with his attempt to write that specific letter. Over the next few hours of testing, T5 was shown several sets of sentences and asked to mentally try to "handwrite" each sentence without using capital letters. These sentences included, "i interrupted, unable to keep silent," and "within thirty seconds the army hadlanded."
Over time, the algorithms improved their ability to distinguish between the patterns of neural firing that represented different letters or symbols. The algorithm’s interpretation of whatever letter T5 intended to write appeared on a computer screen after a delay of about half a second.
T5 was also asked to replicate sentences that the algorithm had never been exposed to. He was eventually able to produce 90 characters per minute, about 18 words. Afterward, he was asked to answer open-ended questions (which required some pauses to think), and he wrote 73.8 characters per minute (an average of nearly 15 words), three times the speed of the free-writing records in the 2017 study.
Willett et al.'s algorithm provided impressively accurate classifications. The error rate for copying was about one error in every 18 or 19 characters; the error rate for freewriting was about one error in every 11 or 12 characters. When the researchers included a predictive language model (similar to the auto-correct feature on smartphones), the error rates were significantly lower: less than 1% for copying and just over 2% for freewriting.
"Compared to other brain-computer interfaces, these error rates are quite low," Shenoy said.
Two tiny implanted electrode arrays convert information from brain regions that control the hand and arm into algorithms that translate into letters on the screen. (00:10)
▍When will it be transformed into a real product?
It is worth noting that in this study, Willett and others came to another important conclusion. Willett told reporters, "This brain-computer interface is faster than before. This is because the neural activity patterns evoked by complex movements such as writing different letters are easier to distinguish. We found that asking participants to write different letters by hand evoked very unique neural activity patterns in their brains."
Willett et al. believe that this allows them to achieve greater accuracy than before while doing so at much faster speeds.
"When you can only record from a small number of neurons picked up by sensors (compared to the millions of neurons in motor brain areas), it helps to have very different neural patterns, and the chances of accidentally confusing them are low. This is why complex movements, such as writing different letters, may be easier to decode; the complexity makes them more unique and different from each other." Willett further explained that, in contrast, the previous state-of-the-art typing method, "moving along a straight line to different keys would evoke very similar patterns of neural activity, because all that is involved is a straight-line movement with different angles or different distances."
This also means that, perhaps contrary to our intuition, decoding complex behaviors is more advantageous than simple behaviors, especially in classification tasks. This information will be of great reference significance for future brain-computer interface research.
Of course, this study is not perfect. Before it can be put into large-scale clinical applications, the lifespan, safety and effectiveness of this technology still need to be further verified. Rajeswaran et al. wrote in a perspective article that this technology "needs to have excellent effects and benefits to prove that the cost and risk of implanting electrodes in patients' brains is worthwhile."
Input speed is not the only factor that determines whether to adopt this technology. Rajeswaran et al. believe that further research may be needed to ensure that the performance of the device is maintained throughout its life cycle, such as how it performs when it encounters changes in neural activity patterns. Continued research to test whether the method can be generalized to other users and settings outside the laboratory will also be crucial.
Another concern for Rajeswaran et al. is how the method will scale and translate to other languages? Willett et al.'s research also showed that several characters are written similarly, such as r, v and u, and are therefore more difficult to classify than other characters. "One of us (Rajeswaran) speaks Tamil, which has 247 letters that are often closely related, so they may be difficult to classify," they wrote in the article.
Willett told reporters that to turn this technology into a real product, it needs to be simplified, and users should not need to spend too much time training it to use this brain-computer interface. In addition, it should be smart enough to automatically track real-time changes in neural activity so that users do not have to stop and retrain the system every day. Finally, the microelectrode device should be wireless and fully implanted.
"This is the work that a company has to do to create a real-world product." He said that in this study, they only conducted a proof-of-concept demonstration. "Handwritten brain-computer interfaces are an exciting and potentially feasible way for us to restore communication with severely paralyzed people."