Shape Features Improve the Encoding Performance of High-level Visual Cortex
Pages 101 - 108
Abstract
The visual encoding model based on the convolutional neural network (CNN) realizes the prediction of brain activity from the hierarchical similarity between deep neural network and visual cortex. However, studies have shown that CNNs trained on the ImageNet have a strong texture bias, inconsistent with human's preference for shape discrimination in image recognition. Also, the image features extracted by pre-trained CNNs are not enough to encode the visual cortex, especially for the high-level visual cortex (HVC). Here, we use functional magnetic resonance imaging (fMRI) data and extract image features through different CNNs that learn texture features and shape features. Then we use ridge regression to build a linear mapping from features to voxel response to achieve the construction of the visual encoding models. The comparative analysis of different visual areas indicates that the visual encoding model constructed by CNN that learns shape features can improve the encoding performance.
References
[1]
Hubel D H, Wiesel T N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[J]. Journal of Physiology, 1962, 160(1):106--154.
[2]
Hubel D H, Wiesel T N. Receptive Fields and Functional Architecture of Monkey Striate Cortex. The Journal of Physiology[J], 1968, 195(1):215--243.
[3]
Thomas, Serre. Deep Learning: The Good, the Bad, and the Ugly[J]. Annual review of vision science, 2019, 5(1):399--426.
[4]
TC Kietzmann, P Mcclure, N Kriegeskorte. Deep Neural Networks in Computational Neuroscience[J]. bioRxiv, 2017.
[5]
Kay K N, Naselaris T, Prenger R J, et al. Identifying natural images from human brain activity[J]. Nature, 2008, 452(7185): 352--5.
[6]
Agrawal P, Stansbury D, Malik J, et al. Pixels to Voxels: Modeling Visual Representation in the Human Brain[J]. Eprint Arxiv, 2014.
[7]
Belliveau J W, Kennedy D N, Mckinstry R C, et al. Functional mapping of the human visual cortex by magnetic resonance imaging[J]. Science, 1991, 254(5032): 716--9.
[8]
Mo C, Han J, Hu X, et al. Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective[J]. Brain Imaging and Behavior, 2014, 8(1):7--23.
[9]
Ogawa S, Lee T-M, Kay A R, et al. Brain magnetic resonance imaging with contrast dependent on blood oxygenation[J]. Proceedings of the National Academy of Sciences, 1990, 87(24): 9868--72.
[10]
Kheradpisheh S R, Masoud G, Mohammad G, et al. Humans and Deep Networks Largely Agree on Which Kinds of Variation Make Object Recognition Harder[J]. Frontiers in Computational Neuroscience, 2016, 10:92.
[11]
Pospisil D A, Pasupathy A, Bair W. 'Artiphysiology' reveals V4-like shape tuning in a deep network trained for image classification[J]. eLife Sciences, 2018, 7.
[12]
Laskar M, Giraldo L, Schwartz O. Correspondence of Deep Neural Networks and the Brain for Visual Textures[J]. 2018.
[13]
Dodge S, Karam L. Can the early human visual system compete with Deep Neural Networks[C]. proceedings of the IEEE International Conference on Computer Vision Workshop, 2017.
[14]
Gruber L Z, Haruvi A, Basri R, et al. Perceptual Dominance in Brief Presentations of Mixed Images: Human Perception vs. Deep Neural Networks [J]. Front Comput Neurosci, 2018, 12(57)
[15]
Geirhos R, Temme C R M, Rauber J, et al. Generalisation in humans and deep neural networks [J]. arXiv preprint arXiv:180808750, 2018
[16]
Naselaris T, Kay K N, Nishimoto S, et al. Encoding and decoding in fMRI[J]. Neuroimage, 2011, 56(2): 400--10.
[17]
Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science, 2014.
[18]
Bay H, Ess A, Tuytelaars T, et al. Speeded-up robust features (SURF)[J]. Computer vision and image understanding, 2008, 110(3): 346--359.
[19]
A N E, A A G H, B J L G A J N. Voxelwise encoding models with non-spherical multivariate normal priors [J]. 2019, 197(482--92).
[20]
Güçlü U, van Gerven M A. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream[J]. J Neurosci, 2015, 35(27): 10005--14.
[21]
Güçlü U, Gerven M A J V J N. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects[J]. 2017, 145(Pt B): 329--36.
[22]
Siegle J H, Jia X, Durand S, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 2021, 592: 86--92.
[23]
Desimone R, Schein S J, Moran J, et al. Contour, color and shape analysis beyond the striate cortex[J]. Vision Research, 1985, 25(3):441--452.
[24]
Kourtzi Z, Tolias A S, Altmann C F, et al. Integration of Local Features into Global Shapes: Monkey and Human fMRI Studies[J]. Neuron, 2003, 37(2):333--346.
[25]
Kourtzi Z, Connor C E. Neural Representations for Object Perception: Structure, Category, and Adaptive Coding[J]. Annual Review of Neuroscience, 2011, 34(1):45.
[26]
Lowe M X, Rajsic J, Gallivan J P, et al. Neural representation of geometry and surface properties in object and scene perception[J]. Neuroimage, 2017, 157:586.
[27]
Landau B, Smith L B, Jones S S. The importance of shape in early lexical learning[J]. Cognitive Development, 1988, 3(3):299--321.
[28]
Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database[C]. CVPR, 2009.
[29]
Agrawal P, Girshick R, Malik J. Analyzing the Performance of Multilayer Neural Networks for Object Recognition[J]. arXiv:1407.1610, 2014.
[30]
Schrimpf M, Kubilius J, Hong H, et al. Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like[J]. bioRxiv, 2018, 407007.
[31]
Eickenberg M, Gramfort A, Varoquaux G, et al. Seeing it all: Convolutional network layers map the function of the human visual system [J]. Neuroimage, 2016, 152.
[32]
Cadena S A, Denfield G H, Walker E Y, et al. Deep convolutional models improve predictions of macaque V1 responses to natural images[J].bioRxiv, 2017.
[33]
Gatys L A, Ecker A S, Bethge M. Texture Synthesis Using Convolutional Neural Networks[J]. MIT Press, 2015.
[34]
Gatys, Leon A, Ecker A S, Bethge M. Texture and art with deep neural networks[J]. Current Opinion in Neurobiology, 2017, 46:178--186.
[35]
Laskar M, Giraldo L, Schwartz O. Correspondence of Deep Neural Networks and the Brain for Visual Textures[J]. 2018.
[36]
Brendel W, Bethge M. Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet[J]. 2019.
[37]
Geirhos R, Rubisch P, Michaelis C, et al. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness[J]. CoRR, 2018.
[38]
Huang X, Belongie S. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization[J]. 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
[39]
Ketkar N. Introduction to pytorch [M]. Deep Learning with Python. Springer. 2017: 195--208.
[40]
Kennard H R W J T. Ridge Regression: Applications to Nonorthogonal Problems[J]. 1970, 12(1): 69--82.
[41]
Horikawa T, Aoki S C, Tsukamoto M, et al. "Characterization of deep neural network features by decodability from human brain activity," Scientific Data, 2019.
[42]
He K, Zhang X, Ren S, et al. "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770--778, 2016.
[43]
Krizhevsky A, Sutskever I, and Hinton G. "ImageNet Classification with Deep Convolutional Neural Networks," Communications of the ACM, vol. 60, no. 6, pp. 84--90, 2017.
[44]
Ba J, Mnih V, and Kavukcuoglu K. "Multiple Object Recognition with Visual Attention," Computer Science, 2014.
[45]
Long J, Shelhamer E, and Darrell T. "Fully Convolutional Networks for Semantic Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 640--651, 2015.
[46]
Yu Z, Zhang C, Wang L, et al. "A Comparative Analysis of Visual Encoding Models Based on Classification and Segmentation Task-Driven CNNs," Computational and Mathematical Methods in Medicine, 2020.Conference Name: ACM Woodstock conference.
Index Terms
- Shape Features Improve the Encoding Performance of High-level Visual Cortex
Recommendations
Unconscious processing of unattended features in human visual cortex
Unconscious processing has been convincingly demonstrated for task-relevant feature dimensions. However, it is possible that the visual system is capable of more complex unconscious operations, extracting visual features even when they are unattended ...
Haptic shape processing in visual cortex
Humans typically rely upon vision to identify object shape, but we can also recognize shape via touch haptics. Our haptic shape recognition ability raises an intriguing question: To what extent do visual cortical shape recognition mechanisms support ...
What are we missing here? Brain imaging evidence for higher cognitive functions in primary visual cortex V1
Functional magnetic resonance imaging (fMRI) and neuron electrophysiology (neurophysiology) are two well-established ways to measure brain activity. Even though the spatial and temporal resolution of these techniques is very different, both measurements ...
Comments
Information & Contributors
Information
Published In
October 2021
593 pages
ISBN:9781450395588
DOI:10.1145/3500931
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 December 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ISAIMS 2021
ISAIMS 2021: 2nd International Symposium on Artificial Intelligence for Medicine Sciences
October 29 - 31, 2021
Beijing, China
Acceptance Rates
Overall Acceptance Rate 53 of 112 submissions, 47%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 38Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)3
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in