Quantitative Biology > Neurons and Cognition
[Submitted on 10 Feb 2021]
Title:A General Framework for Revealing Human Mind with auto-encoding GANs
View PDFAbstract:Addressing the question of visualising human mind could help us to find regions that are associated with observed cognition and responsible for expressing the elusive mental image, leading to a better understanding of cognitive function. The traditional approach treats brain decoding as a classification problem, reading the mind through statistical analysis of brain activity. However, human thought is rich and varied, that it is often influenced by more of a combination of object features than a specific type of category. For this reason, we propose an end-to-end brain decoding framework which translates brain activity into an image by latent space alignment. To find the correspondence from brain signal features to image features, we embedded them into two latent spaces with modality-specific encoders and then aligned the two spaces by minimising the distance between paired latent representations. The proposed framework was trained by simultaneous electroencephalogram and functional MRI data, which were recorded when the subjects were viewing or imagining a set of image stimuli. In this paper, we focused on implementing the fMRI experiment. Our experimental results demonstrated the feasibility of translating brain activity to an image. The reconstructed image matches image stimuli approximate in both shape and colour. Our framework provides a promising direction for building a direct visualisation to reveal human mind.
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.