Image Reconstruction from Electroencephalography Using Latent Diffusion

Abstract

In this work, we have adopted the diffusion-based image reconstruction pipeline previously used for fMRI image reconstruction and applied it to Electroencephalography (EEG). The EEG encoding method is very simple, and forms a baseline from which more sophisticated EEG encoding methods can be compared. We have also evaluated the fidelity of the generated image using the same metrics used in the previous functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) works. Our results show that while the reconstruction from EEG recorded to rapidly presented images is not as good as reconstructions from fMRI to slower presented images, it holds a surprising amount of information that could be applied in specific use cases. Also, EEG-based image reconstruction works better in some categories–such as land animals and food–than others, shedding new light on previous findings of EEG’s sensitivity to those categories and revealing potential for these methods to further understand EEG responses to human visual coding. More investigation should use longer-duration image stimulations to elucidate the later components that might be salient to the different image categories.

Keywords EEG  $\cdot$ visual-evoked potential  $\cdot$ latent diffusion  $\cdot$ rapid-serial visual presentation  $\cdot$ visual perception

Code Availability

Code can be accessed on the GitHub repository:

https://github.com/desa-lab/EEG-Image-Reconstruction

1 Introduction

Visual perception is an important aspect of human cognition and a gateway to understanding more complex cognitive processes such as visual imagery and dream visuals. However, it has been a challenge to come up with an objective and reliable metric for measuring visual perception that is also commensurate with its complexity. The advent of functional neuroimaging allows for putting theories into practice, such as decoding visuals using the receptive field model (Kay et al., 2008). The introduction of the latent diffusion model (Xu et al., 2022) into the decoding scene transformed the previously blurry reconstructions into something vivid and humanly interpretable (Takagi and Nishimoto, 2023). As specific semantic contents are now rendered onto the images, it is also possible to compare the high-level semantic similarity of those images with their original images using deep neural network models trained to perform these tasks.

Compared to fMRI, which has well-defined source localization in space and fine-grained spatial resolution, EEG not only has an under-determined source space but is also constrained by volume conduction across different types of tissue between the neurons and the electrodes, which limits its functional spatial resolution to a few centimeters. Under such constraints, it is unlikely that EEG would contain remotely sufficient retinotopic information to reconstruct the images. On the other hand, it is common knowledge that the early components in visual-evoked potentials (VEP) reflect low-level visual features such as visual field (Halliday and Michael, 1970; Jeffreys and Axford, 1972), color (Paulus et al., 1988) and contrast (Schechter et al., 2005), and certain ERPs such as N170 are sensitive to certain visual categories such as faces (Taylor, 2002). This combined with the wide availability and low cost of EEG makes it appealing to test the image reconstruction despite its shortcomings.

It is worth noting that even though the latent diffusion models are generally complex and cost-prohibitive to train, the mapping from the brain signals (fMRI, MEG, EEG, etc) to the embedding space in the diffusion models is generally simple and can be achieved using linear models. We have applied regularized linear regression to map EEG signals from viewing 16740 images in an RSVP paradigm (Gifford et al., 2022) onto latent embeddings of the versatile diffusion model (Xu et al., 2022).

Earlier study

At the time of writing this, there has already been a paper on EEG image reconstruction using latent diffusion, however that dataset has already been shown to be problematic due to its blocked experiment design (Li et al., 2020). The experiment was designed so that all presentations of one object were close in time (training and test data), and separated from presentations of other objects. Because EEG is a non-stationary signal, and drifts over time, features in the EEG signals can correlate with time. This is evident in the high classifiability of that dataset, and further shown by extensive experiments in (Li et al., 2020). Notably also, visual inspection of the reconstructions from that dataset reveal similarity in class, but no similarity in visual features separate from the class.

Concurrent study

At the time of writing this, a different team has published a paper about 2 weeks in advance (Li et al., 2024). Their approach used a transformer-based model called the adaptive thinking mapper (ATM). We illustrate that it is even feasible to use a linear model for mapping.

2 Methods

We adopted the Ozcelik and VanRullen (2023) method with minimal modifications to the THINGS-EEG2 dataset. To summarize their method, the image reconstruction is a 2-stage process: the first stage maps the brain signal onto the latent space of a variational auto-encoder (VAE), specifically Very Deep Variational Auto-Encoder (VDVAE) (Child, 2020), which would provide a rough visual representation that would pass onto the diffusion model. The second stage maps the same brain signal onto the CLIP-Vision and CLIP-Text embeddings of the Versatile Diffusion model (Xu et al., 2022), which combines the CLIP and the encoded images from the VDVAE and produces the reconstructed images. The Natural Scenes Dataset (Allen et al., 2022) used in the Ozcelik and VanRullen (2023) paper used the COCO image dataset, which has captions for each image that can be used for extracting CLIP-Text. For the THINGS dataset, the images only come with their corresponding category names rather than a full-sentence description, so those category names were used to generate the training CLIP-Text embeddings.

2.1 Dataset

We used the preprocessed version of THINGS-EEG2 (Gifford et al., 2022), which has 17 posterior EEG channels compared to the 63 total channels in the raw dataset.

https://osf.io/anp5v/

Each trial in the data is from -0.2 seconds to 0.8 seconds relative to the onset of the stimulus. The training images and test images are presented in separate sessions, but within training and test images all the orders are pseudo-randomized. Each training image is shown 4 times, and each test image is shown 80 times. We took the average of all trials for each image to form the final dataset.

2.2 Computing Basic Performance Metrics

We used the same performance metrics (See Figure 1) as in Ozcelik and VanRullen (2023). Quoting from their paper:

PixCorr is the pixel-level correlation of reconstructed and groundtruth images. SSIM (Wang et al., 2004) is the structural similarity index metric. AlexNet(2) and AlexNet(5) are the 2-way comparisons of the second and fifth layers of AlexNet (Krizhevsky et al., 2017), respectively. Inception is the 2-way comparison of the last pooling layer of InceptionV3 (Szegedy et al., 2015). CLIP is the 2-way comparison of the output layer of the CLIP-Vision (Radford et al., 2021) model. EffNet-B and SwAV are distance metrics gathered from EfficientNet-B1 (Tan and Le, 2019) and SwAV-ResNet50 (Caron et al., 2020) models, respectively. The first four can be considered as low-level metrics, while the last four reflect higher-level properties. For PixCorr and SSIM metrics, we downsampled generated images from 512×512 resolution to 425×425 resolution. For the rest of the measures, generated images are preprocessed according to the input properties of each network.

Figure 1 (a) is computed by models trained on each subject’s 4-trial-averaged training data, and tested on their corresponding 80-trial-averaged test data. For Figure 1 (b), the "first 200ms", "first 400ms", "first 600ms" and "first 800ms" models use those corresponding time ranges after the onset of the stimulus. The chance level performance is computed by passing the 200ms before the onset of the stimulus onto the trained "first 200ms" model.

2.3 Model Ablation

As described in the Ozcelik and VanRullen (2023); Takagi and Nishimoto (2023), the image generation pipeline maps each trial onto 3 latent embeddings used for the reconstruction: AutoKL, CLIP-Vision, and CLIP-Text. The AutoKL is used by VDVAE, and is responsible for the general shapes and colors of the reconstructed images. Both CLIP-Vision and CLIP-Text are used by Versatile Diffusion, and the CLIP-Vision is more responsible for the visual features of the images and CLIP-Text more responsible for the general semantic alignment.

To assess the relative contribution of the 3 kinds of latent embeddings toward the model performance we tested 5 kinds of ablated models: "CLIP-Text only" uses only the CLIP-Text embedding; "no CLIP-Vision" uses AutoKL and "CLIP-Text"; "CLIP-Vision only" uses only the CLIP-Vision embedding; "no CLIP-Text" uses AutoKL and CLIP-Vision; "no AutoKL" uses CLIP-Vision and CLIP-Text.

The ablation of the AutoKL is done by feeding a blank gray image (127, 127, 127 in RGB) instead of the results from VDVAE as the input image into Versatile Diffusion, and increasing the diffusion strength from 0.75 to 0.99. The ablation of the CLIP-Vision is done by encoding a blank black image as CLIP-Vision instead of using the predicted CLIP-Vision from EEG data and setting the mixing ratio from 0.4 to 0. The ablation of the CLIP-Text is done by encoding an empty string as CLIP-Text instead of using the predicted CLIP-Text from EEG data and setting the mixing ratio from 0.4 to 0.99.

2.4 Pairwise Data Segment Replacement

Each trial in the EEG data contains 800 miliseconds worth of data, or 80 sample points at a sampling rate of 100 Hz. We took a few trials and paired them together to see the effect of swapping a small segment in the data between those two trials (See Figure 4). Take the top two rows in the figure, for example: the left-most image in the first row and the second row have their 0-50 ms segment swapped. The second from the left in the first and second row have their 10-60 ms segment swapped. The third from the left in the first and second row have their 20-70 ms segment swapped, and so forth. At 10 ms step size and the sliding window length of 50ms, there would be 75 images for the 800 ms trial. There is 1 additional image at the very right, which is from the original data without the pairwise swapping, as a reference. Each 2 subsequent rows would have the same pairwise data segment replacement procedure.

3 Results

3.1 Basic Performance Metrics

The performance across the 10 subjects are relatively consistent with a reasonable amount of variation. The performance across duration shows that using 400ms of data achieves a slightly higher performance than 200ms, despite the fact that other images have started showing by this time. See Appendix A for examples of reconstructions from subject 1. To put the performance in context, the reported THINGS-MEG data performance is slightly higher than ours (Benchetrit et al., 2024). Although they did not use the provided test set but rather took out parts of the training set as the test set, and thus did not have multiple trials to average during test time. Using 3 second duration averaged over 3 NSD presentations and 7T fMRI recording achieves significantly higher performance (Scotti et al., 2023).

(a)

(b)

(c)

Figure 1: Basic Performance Metrics.

3.1.1 Ablation

The full model achieved the best all-around performance compared to the ablated models. In general models without CLIP-Text achieved similar level of performance and are slightly better than models without CLIP-Vision.

Figure 2: Ablation Image Reconstruction Performance.

3.1.2 ICA Components

We used the ICA algorithm in the MNE-Python package (Gramfort et al., 2013) on the averaged test EEG data for subject 1. Specifically, we used the "extended-infomax" method with max iterations of 1000 (for reproducibility, we used random state 1). We then reconstruct the test EEG data using the top 1-16 independent components. We are currently investigating the enigmatic performance of Structural Similarity (SSIM), which is inconsistent with the rest of the peformance metrics. See Appendix C for more details on the ICA components.

Figure 3: Cumulative ICs Peformance. Increasing the number of IC’s increased the performance with diminishing returns.

3.2 Feature Transfer Effect from Narrow Time Segment Swapping

To investigate the salient features in the data and to find out time ranges that are most sensitive to disturbance. We used pairs of image-viewing EEG data and swapped a 50ms sliding time window between them (See Figure 4). We moved the time window along the 800ms trial to see how much each image is disturbed by the other image at each time point. Refer to Section 2.4 for more details on the method.

Take the "gorilla-gopher" pair, for example, there is a clear fur color swapping effect between the 100-210ms time range. The feature swapping effect is not always present for all pairs of images, and the exchange of features is not always symmetric. However, the example indicates that a disturbance of the reconstructed image is generally present between the 100-380ms range, as can be seen even from a distance.

The examples are hand-picked from image classes that were well visualized, but without knowing the outcome of the swapping in advance.

Figure 4: Examples of data segment swapping. Each pair of rows represents the 2 images that have parts of the EEG swapped. The images for each pair of rows from top to bottom are: "gorilla_18s.jpg" and "gopher_09s.jpg"; "chaps_18s.jpg" and "headscarf_03s.jpg"; "cat_01b.jpg" and "coverall_06s.jpg"; "sausage_04s.jpg" and "piglet_02s.jpg"; "caterpillar_03s.jpg" and "possum_05s.jpg"; "cart_09s.jpg" and "elephant_11n.jpg". Each image in a row represents the result of swapping a time window of 50ms (5 samples). The next image is the result of moving the time window by 10ms (1 sample). The last image of each row is added as a reference since it does not have any swapping. Refer to Section 2.4 for more details on the method.

4 Discussion

The similarity of the generated images to the ground truth images not only relies on the mapping algorithm but also the specific diffusion model and settings. Therefore to facilitate fair comparisons across studies, it would be helpful to adapt previous studies to newer diffusion models if there are significant improvements to the diffusion models. This study presents a baseline performance achievable using a relatively new diffusion model with a minimal amount of machine learning involved in the mapping.

EEG has well known limitations in spatial resolution which restrict the fidelity of the image decoding. The dataset used in this work used RSVP-style (rapid) presentation with stimuli lasting 100ms and new stimuli appearing every 200ms. While the rapid presentation increases the data quantity and thus the signal-to-noise ratio which can help greatly with the model performance, the restricted processing time (interrupted by new visual presentations) reduces any later cognitive processes that could be potentially useful in decoding. To investigate what later VEP components might be salient to different visual categories, it might be useful to prolong the duration of each image presentation to a little less than a second. Results from THINGS-MEG (Hebart et al., 2023) indicated that MEG data with longer ISI and presentation times achieved better reconstructions when longer windows of MEG were used (Li et al., 2024).

In its current form, the EEG image reconstruction may be used in a few limited cases, such as entertainment and artwork generation. However, the application needs to be aware of the nature of the signal as a response to sudden changes in the visual image. To apply it to real-world visual scenes, it might be necessary to add a rapid, automatic shutter goggle in front of the eyes to create artificial visually flashed stimuli that mimic the data format used in this study. Alternatively similar methods could be explored with more natural image presentation paradigms.

Future research may also look into the direction of video reconstruction as video generation AI is approaching a similar level of sophistication as image generation AI. Video stimuli would be more representative of natural visual experiences. Different motions occurring in a video may also contain decodable time courses, something that would take advantage of the temporal resolution of EEG and MEG. Decoding a sub-section of a video clip could provide insight into mechanisms of ongoing visual processing, which, compared to evoked transient visual responses, might be more similar to internal visual representations such as visual imagery and dream visuals.

Appendix A Example Reconstructions

Figure 5: Example reconstructions for subject 1. Reconstructions with pairwise correlation from best to worst for the final CLIP embedding. Each row of images with blue frames are the ground truth images. Each row of images with green frames directly under the blue-framed images correspond to the reconstruction of those images. Because there is some randomness in the diffusion process, each time the generated images would look slightly different, thus the ordering form best to worst might be different each time as well.

Appendix B UMAP of Final CLIP Embeddings

The same images from Figure 5 can also be visualized in a self-organized map 6 by taking their Final CLIP embeddings (which is the same final CLIP embeddings used in the performance metrics) and perform UMAP dimensionality reduction (McInnes et al., 2018) on them. The perfomance score based on the CLIP embeddings for each image is passed onto a sigmoid function, the result of which is then multiplied by a baseline transparency and image size. The transparency and size of ground truth images (shaded blue) are always constant.

Figure 6: UMAP of Final CLIP Embeddings for ground truth (blue) and reconstructed images (green) from subject 1. The transparency level as well as the size of the green images indicate the correlation of CLIP vector between the corresponding reconstructed image and ground truth image pair. The ground truth images themselves contain two clusters of images separated from the rest representing animals and food, which reflect the 2 most prominent clusters in the reconstructed images as well.

Appendix C ICA Components

Figure 7: ICA components for subject 1