How many times do we wish we could simply look inside another person’s mind and figure out what they’re thinking? We could see their future plans, their daydreams, or maybe see hidden intentions. This might sound like something straight out of sci-fi stories like Minority Report or Inception, but recent advances in neuroimaging methodology have allowed the idea of “brain reading” to leave its confines of fiction. In this article, we’ll dive into the current state of brain reading, its mechanisms, and several applications of brain reading techniques, from playing games to tackling psychological and physical impairments.

How do we look inside the brain?

There are several ways of acquiring brain data. In the past, scientists primarily used electrodes implanted during brain surgery to measure neuronal activity directly or via electroencephalography (EEG), which picks up that activity indirectly through the skull in the form of electrical signals. These methods have already led to some very impressive results over the last decades via brain-computer interfaces (BCIs), on which we have written this article previously. Some applications are very useful for patients suffering from strong body paralysis, especially locked-in patients (who are fully conscious but can’t move their bodies). For instance, using an EEG-based BCI, patients learned to control a robotic arm and regain some physical autonomy in their everyday lives.

How is all possible? Imagine you want to compare the artworks of two painters. Painter A loves using bright colours, soft brushstrokes, and round shapes. Painter B on the other hand is a fan of monochrome paintings filled with rough brushstrokes and angled objects. This means the two artists’ styles can be distinguished along these three dimensions (colours, strokes, shapes). You now ask an art expert, who knows these two painters very well, to look at a painting he’s never seen before. They will make an educated guess if it was painted by Painter A or Painter B based on how well the three features of this new painting match with either painter’s style. This is exactly how we do it with brain data as well!

(Source: freepik)

So, let’s apply this analogy to the brain: As a simple example, suppose you want to control a robotic arm to the left and to the right using brain activity. You choose to measure the activity of 9 neurons from the arm region of the supplementary motor area (SMA), since it represents the intended direction of arm movement. Each of these 9 neurons is a feature (e.g., colours, shapes, etc.), also called data point, with its own neuronal activation level. The neuron’s activation level varies on a spectrum of low to high activation. Using the painting analogy, this is like how, for example, the colours of a painting can vary on a spectrum between washed out and vibrant.

Together, the combinations of features and their intensities (i.e., combination of neurons and their activation levels) characterise the whole painting (i.e., the brain state). For instance, if the brain now commands an arm movement to the left, the 9 neurons each activate with a certain intensity, culminating in the characteristic “painting” of features associated with the brain state “left” (see figure, upper left). In the same way, a unique combination of features paints a unique painting associated with “right” (see figure, upper right).

(Figure by author)

What’s still missing is an expert — something to group these brain states. The machine learning equivalent of the art expert is called a classifier. Once the classifier has been trained to distinguish between the “left” and “right” activation patterns of our 9 neurons, it can make an educated guess about brain states it has never seen before and judge if they are “left” or “right” states. (Looking at the figure, take the role of the art expert: would you say the new brain state looks more like “left” or “right”?)

The decision of the classifier can then, at last, be fed into the robotic arm, which executes the intended movement. Of course, like any human (even an art expert), the classifier won’t always be correct. However, classifiers can reach some very good accuracy rates of 90% or more after enough training (ca. 20-300 trials in 10-90 minutes, depending on your BCI method).

Note that for simplicity this example used 9 neurons, meaning that two brain states can differ in exactly 9 ways (i.e. in the activation level of each neuron). While a classifier can work with 9 data points, more data points are generally preferable. That’s because each extra data point adds another feature through which two brain states could differ, making it easier for the classifier to detect that difference.

The added value of fMRI

All of this is very exciting, but the above techniques are both pretty inconvenient: Electrode recordings require surgery, which makes them very impractical and invasive to apply, whereas EEG suffers from poor spatial resolution, as they only cover the surface level of the brain and less the deeper layers of the brain. Both methods are limited in regards to which areas are mapped, with electrodes usually being applied at one particular spot on the brain, and EEG signals mainly being caused by electrical signals originating in the cortex of the brain. This is where fMRI comes in, with its ability to image the entire brain noninvasively with an astounding number of data points (more than one million ways for the paintings to differ) using modern ultra-high field scanners. Now, we can reach any area in the brain with formidable resolution and interpret its activation patterns. While in the early days of fMRI all the data had to be analysed in bulk after the experiment ended, advances in computer processing power allowed researchers to analyse the fMRI data in real time, that is, while the subject was still in the scanner. That way, they could interpret every new signal as it came in. In fact, as a proof of concept, Maastricht University researchers playfully implemented a “Brain Pong” feature into their real-time fMRI software: They found that it was possible to let two participants, each in a different scanner, play Pong with each other, based only on their momentary brain activation that was being decoded on the fly by a classifier in order to decide whether the player wanted their bar (see picture) to move up or down.

(Source: Wikimedia Commons)

Literal Brain Reading?

Now we know how we can read out an fMRI signal in order to let people interact with their environment with their mind. Now, to move closer to the literal meaning of the term brain reading: what if we wanted to know the thoughts that are on someone’s mind? Put differently, what if someone wanted to communicate their thoughts wordlessly? With some clever workarounds in past studies, participants could spell words with their minds using letter-speller BCIs.

For instance, one team of researchers instructed participants in the fMRI-scanner to express letters by performing different mental tasks. That way, there was one unique brain state for each of the 26 letters of the alphabet, like a dictionary. A classifier could then look up the brain state in that dictionary and decode the corresponding letter. If you look at the below lookup table, [1] [2] [3]  you’ll see that if, for instance, a participant performed motor imagery (e.g., imagining running/climbing/etc.) for 30 seconds, the classifier would decode the letter “B”. Using this method, subjects could slowly but surely spell words in real time, accurately answering personal questions and even unscripted follow-up questions!

Figure from Sorger et al. (2012)

However, this approach is based on an arbitrary letter-coding scheme set by the researchers. After all, it’s not like the letter “B” is typically represented in the brain by imagining running for 30 seconds. This begs the question: do approaches like these even qualify as brain reading?

There have certainly been more direct attempts at decoding brain activity. For instance, one study aimed to decode letters by exploiting the retinotopic properties of the brain, that is, the way our visual cortex maps out spatial relationships in a way that mirrors the real image in the visual world that we see. First, the researchers determined which parts of the visual cortex encode which part of visual space. In other words, they determined for each part of the visual cortex to which point in the visual field it is connected. That way, using visual cortex activation data, the researchers could reconstruct outlines of viewed shapes, such as the letters “H”, “T”, “S”, and “C”. Even more stunningly, the same applied to the mere imagining of the letters! Although the outlines were weaker, the letters were still identifiable.

Figure from Sorger & Goebel (2020)

This proof of concept begs the question of how much further we can take this idea of literal brain reading. In fact, a new study using the same methods is already underway, but this time they want to map all 26 letters of the alphabet. It’s safe to say that exciting times are ahead. Especially with the construction of the 14 Tesla MRI scanner (twice the strength of common ultra-high field scanners) in Nijmegen, higher resolutions with even better brain reading performance will likely become possible.

Neurofeedback

Aside from reading out and drawing conclusions from the neural signal, it can also be used to help people regulate their own brain activity in real-time. This approach is called neurofeedback, where (some transformed version of) the real-time fMRI signal is fed back to the subject lying in the scanner, as soon as it’s acquired. How does this work exactly? It turns out subjects swiftly acquire a regulation strategy for how to control the activity level of a selected brain region (e.g., thinking of faces to upregulate the face area). The current activation level of that area is then displayed on a screen (usually with a thermometer-like scale). That way subjects get feedback on whether their brain activity is changing as intended or not.

This opens up a whole new realm of possibilities, especially for clinical applications. In one pioneering neurofeedback study, participants suffering from chronic pain were encouraged to control the activity level of their own anterior cingulate cortex (ACC), a brain region linked to pain perception. Using the activity level represented on the screen in front of them, participants successfully regulated ACC activity using several strategies, for instance shifting attention towards/away from painful sensations. The cool finding here is that the higher activation in the ACC was correlated with lower subjective pain ratings. This opens up the possibility of helping chronic pain patients by letting them develop regulation strategies to upregulate ACC activity. Similar studies have been conducted for other psychopathologies such as depression, also with promising results.

Some studies have also reported what’s known as transfer effects. This means that, once participants have successfully learned how to regulate their own brain activity using neurofeedback, they can apply the same strategy even outside the scanner and don’t have to rely on neurofeedback anymore. Interestingly, when neurofeedback-trained subjects shared their personal regulation strategies with control subjects that didn’t have access to neurofeedback, it turned out that these strategies were less effective for the control subjects. This can be interpreted as follows: to reach their full effect, regulation strategies need a lot of individualised fine-tuning. Neurofeedback allows for such fine-tuning by letting participants develop their very own regulation strategy in response to their individual neural dynamics.

Where does brain reading go next?

These are just some of the promising avenues that real-time fMRI has explored. This leaves us with some final problems, which are cost and portability. As you may know, fMRI scanners are both very expensive and very large, making brain reading on a regular basis very difficult for many patients. Luckily, we’re currently seeing the rise of functional near-infrared spectroscopy (fNIRS), which measures the same kind of activity as fMRI, but instead it detects it using relatively cheap and portable electrodes that send infrared light through the cortex. This promising tool still hasn’t garnered too much popularity so far, but as more BCI research is conducted in the future, real-time fNIRS might turn into a common alternative to real-time fMRI, facilitating patients’ everyday tasks like movement and communication in many environments.

Author: Emil Stroecker

References:

Alexander GE, Crutcher MD. Preparation for movement: neural representations of intended direction in three motor areas of the monkey. J Neurophysiol. 1990 Jul;64(1):133-50. doi: 10.1152/jn.1990.64.1.133. PMID: 2388061.

deCharms, R. C., Maeda, F., Glover, G. H., Ludlow, D., Pauly, J. M., Soneji, D., Gabrieli, J. D. E., & Mackey, S. C. (2005). Control over brain activation and pain learned by using real-time functional MRI. Proceedings of the National Academy of Sciences, 102(51), 18626–18631. https://doi.org/10.1073/pnas.0505210102

Looned, R., Webb, J., Xiao, Z., & Menon, C. (2014). Assisting drinking with an affordable BCI-controlled wearable robot and electrical stimulation: a preliminary investigation. Journal of NeuroEngineering and Rehabilitation, 11(1), 1–13. https://doi.org/10.1186/1743-0003-11-51

Mur, M., Bandettini, P. A., & Kriegeskorte, N. (2009). Revealing representational content with pattern-information fMRI—an introductory guide. Social Cognitive and Affective Neuroscience, 4(1), 101–109. https://doi.org/10.1093/scan/nsn044

Senden, M., Emmerling, T. C., Van Hoof, R., Frost, M. A., & Goebel, R. (2019). Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. Brain Structure and Function, 224(3), 1167–1183. https://doi.org/10.1007/s00429-019-01828-6

Sorger, B., & Goebel, R. (2020). Real-time fMRI for brain-computer interfacing. Handbook of Clinical Neurology, 289–302. https://doi.org/10.1016/b978-0-444-63934-9.00021-4

Sorger, B., Reithler, J., Dahmen, B., & Goebel, R. (2012). A Real-Time FMRI-Based spelling device immediately enabling robust Motor-Independent communication. Current Biology, 22(14), 1333–1338. https://doi.org/10.1016/j.cub.2012.05.022

Quiles, E., Dadone, J., Chio, N., & García, E. (2022). Cross-Platform Implementation of an SSVEP-Based BCI for the Control of a 6-DOF Robotic Arm. Sensors (Basel, Switzerland), 22(13). https://doi.org/10.3390/s22135000

Weiskopf N. Real-time fMRI and its application to neurofeedback. Neuroimage. 2012 Aug 15;62(2):682-92. doi: 10.1016/j.neuroimage.2011.10.009. Epub 2011 Oct 14. PMID: 22019880.