article

Visual saliency on networks of neurosynaptic cores

Authors:

A. Andreopoulos,

R. Alvarez-Icaza,

M. D. Flickner,

D. S. ModhaAuthors Info & Claims

IBM Journal of Research and Development, Volume 59, Issue 2-3

Pages 9:1 - 9:16

https://doi.org/10.1147/JRD.2015.2400251

Published: 01 March 2015 Publication History

Abstract

Identifying interesting or salient regions in an image plays an important role for multimedia search, object tracking, active vision, segmentation, and classification. Existing saliency extraction algorithms are implemented using the conventional von Neumann computational model. We propose a bottom-up model of visual saliency, inspired by the primate visual cortex, which is compatible with TrueNorth-a low-power, brain-inspired neuromorphic substrate that runs large-scale spiking neural networks in real-time. Our model uses color, motion, luminance, and shape to identify salient regions in video sequences. For a three-color-channel video with 240 × 136 pixels per frame and 30 frames per second, we demonstrate a model utilizing ∼3 million neurons, which achieves competitive detection performance on a publicly available dataset while consuming ∼200 mW.

References

[1]

J. K. Tsotsos, A Computational Perspective on Visual Attention. Cambridge, MA, USA: MIT Press, 2011.

[2]

S. W. Zucker, "Stereo, shading, surfaces: Curvature constraints couple neural computations," Proc. IEEE, vol. 102, no. 5, pp. 812-829, May 2014.

[3]

A. Andreopoulos and J. K. Tsotsos, "50 years of object recognition: Directions forward," Comput. Vis. Image Understanding, vol. 117, no. 8, pp. 827-891, Aug. 2013.

[4]

G. Cauwenberghs, "Reverse engineering the cognitive brain," Proc. Nat. Academy Sci., vol. 110, no. 39, pp. 15512-15513, Sep. 2013.

[5]

G. Indiveri, B. Linares-Barranco, T. J. Hamilton, A. van Schaik, R. Etienne-Cummings, T. Delbruck, S. Liu, P. Dudek, P. Häfliger, S. Renaud, J. Schemmel, G. Cauwenberghs, J. Arthur, K. Hynna, F. Folowosele, S. Saighi, T. Serrano-Gotarredona, J. Wijekoon, Y. Wang, and K. Boahen, "Neuromorphic silicon neuron circuits," Frontiers Neurosci., vol. 5, no. 73, May 2011.

[6]

E. Neftcia, J. Binasa, U. Rutishauserb, E. Chiccaa, G. Indiveria, and R. J. Douglas, "Synthesizing cognition in neuromorphic electronic systems," Proc. Nat. Academy Sci., vol. 110, no. 37, pp. E3468-E3476, Jun. 2013.

[7]

R. J. Vogelstein, U. Mallik, E. Culurciello, G. Cauwenberghs, and R. Etienne-Cummings, "Saliency-driven image acuity modulation on a reconfigurable array of spiking silicon neurons," in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 1457-1464.

[8]

R. J. Vogelstein, U. Mallik, J. T. Vogelstein, and G. Cauwenberghs, "Dynamically reconfigurable silicon array of spiking neurons with conductance-based synapses," IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 253-265, Jan. 2007.

Digital Library

[9]

S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, "The SpiNNaker project," Proc. IEEE, vol. 102, no. 5, pp. 652-665, May 2014.

[10]

K. A. Zaghloul and K. Boahen, "A silicon retina that reproduces signals in the optic nerve," J. Neural Eng., vol. 3, no. 4, pp. 257-267, Dec. 2006.

[11]

L. Itti and C. Koch, "Computational modelling of visual attention," Nat. Rev. Neurosci., vol. 2, no. 3, pp. 194-203, Mar. 2001.

[12]

A. Treisman and G. Gelade, "A feature integration theory of attention," Cognitive Psychol., vol. 12, no. 1, pp. 97-136, Jan. 1980.

[13]

C. Koch and S. Ullman, "Shifts in selective visual attention: Towards the underlying neural circuitry," Human Neurobiol., vol. 4, no. 4, pp. 219-227, 1985.

[14]

L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Analysis Mach. Intell., vol. 20, no. 11, pp. 1254-1259, Nov. 1998.

Digital Library

[15]

D. J. Berg, S. E. Boehnke, R. A. Marino, D. P. Munoz, and L. Itti, "Free viewing of dynamic stimuli by humans and monkeys," J. Vis., vol. 9, no. 5, p. 19, May 2009.

[16]

L. Itti, "Automatic foveation for video compression using a neurobiological model of visual attention," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1304-1318, Oct. 2004.

Digital Library

[17]

T. Judd, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look," in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 2106-2113.

[18]

P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada, F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura, B. Brezzo, I. Vo, S. K. Esser, R. Appuswamy, B. Taba, A. Amir, M. D. Flickner, W. P. Risk, R. Manohar, and D. S. Modha, "A Million spiking-neuron integrated circuit with a scalable communication network and interface," Science, vol. 345, no. 6197, pp. 668-673, Aug. 2014.

[19]

A. Cassidy, R. Alvarez-Icaza, F. Akopyan, J. Sawada, J. V. Arthur, P. A. Merolla, P. Datta, M. Gonzalez Tallada, B. Taba, A. Andreopoulos, A. Amir, S. K. Esser, J. Kusnitz, R. Appuswamy, C. Haymes, B. Brezzo, R. Moussalli, R. Bellofatto, C. Baks, M. Mastro, K. Schleupen, C. E. Cox, K. Inoue, S. Millman, N. Imam, E. McQuinn, Y. Y. Nakamura, I. Vo, C. Guo, D. Nguyen, S. Lekuch, S. Asaad, D. Friedmann, B. L. Jackson, M. D. Flickner, W. P. Risk, R. Manohar, and D. S. Modha, "Real-time scalable cortical computing at 46 giga-synaptic ops/watt with ∼100× speedup in time-to-solution and ∼100,000× reduction in energy-to-solution," in Int. Conf. High Perform. Comput., Netw., Storage Anal.--Supercomput., 2014, pp. 27-38.

Digital Library

[20]

A. S. Cassidy, P. Merolla, J. V. Arthur, S. K. Esser, B. Jackson, R. A Icaza, P. Datta, J. Sawada, T. M. Wong, V. Feldman, A. Amir, D. Ben-Dayan Rubin, F. Akopyan, E. McQuinn, W. P. Risk, and D. S. Modha, "Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores," in Proc. IEEE Int. Joint Conf. Neural Netw., 2013, pp. 1-10.

[21]

S. K. Esser, A. Andreopoulos, R. Appuswamy, P. Datta, D. Barch, A. Amir, J. Arthur, A. Cassidy, M. Flickner, P. Merolla, S. Chandra, N. Basilico, S. Carpin, T. Zimmerman, F. Zee, R. A. Icaza, J. A. Kusnitz, T. M. Wong, W. P. Risk, E. McQuinn, T. K. Nayak, R. Singh, and D. S. Modha, "Cognitive computing systems: Algorithms and applications for networks of neurosynaptic cores," in Proc. IEEE Int. Joint Conf. Neural Netw., 2013, pp. 1-10.

[22]

A. Amir, P. Datta, W. P. Risk, A. S. Cassidy, J. A. Kusnitz, S. K. Esser, A. Andreopoulos, T. M. Wong, M. Flickner, R. A. Icaza, E. McQuinn, B. Shaw, N. Pass, and D. S. Modha, "Cognitive computing programming paradigm: A corelet language for composing networks of neurosynaptic cores," in Proc. IEEE Int. Joint Conf. Neural Netw., 2013, pp. 1-10.

[23]

G. S. Banavar, "An application framework for compositional modularity," Ph.D. Dissertation, Univ. Utah, Salt Lake City, UT, USA, 1995.

[24]

T. M. Wong, R. Preissl, P. Datta, M. Flickner, R. Singh, S. K. Esser, E. McQuinn, R. Appuswamy, W. P. Risk, H. D. Simon, and D. S. Modha, B1014," IBM Res. Div., Armonk, NY, USA, Res. Rep. RJ 10 502, 2012.

[25]

R. Preissl, T. M. Wong, P. Datta, M. Flickner, R. Singh, S. K. Esser, W. P. Risk, H. T. D. Simon, and D. S. Modha, "Compass: A scalable simulator for an architecture for cognitive computing," in Proc. IEEE Int. Conf. High Perform. Comput., Netw., Storage Anal. (SC), 2012, p. 54.

Digital Library

[26]

P. J. Burt, "Fast filter transform for image process.," Comput. Graph. Image Process., vol. 16, no. 1, pp. 20-51, May 1981.

[27]

D. H. Hubel, "The visual cortex of the brain," Sci. Amer., vol. 209, no. 5, pp. 54-62, 1963.

[28]

Neovision2 dataset-iLab-University of Southern California, Los Angeles, CA, USA. [Online]. Available: http://ilab.usc.edu/neo2/dataset/

[29]

R. Kasturi, D. Goldgof, R. Ekambaram, G. Pratt, E. Krotkov, D. D. Hackett, Y. Ran, Q. Zheng, R. Sharma, M. Anderson, M. Peot, M. Aguilar, D. Khosla, Y. Chen, K. Kim, L. Elazary, R. C. Voorhies, D. F. Parks, and L. Itti, "Performance evaluation of neuromorphic-vision object recognition algorithms," in Int. Conf. Pattern Recog., 2014, pp. 2401-2406.

[30]

B. Shaw, A. Cox, P. Besterman, J. Minyard, C. Sassano, R. A. Icaza, A. Andreopoulos, R. Appuswamy, A. Cassidy, S. Chandra, P. Datta, E. Mcquinn, W. Risk, and D. S. Modha, "Cognitive computing commercialization: Boundary objects for communication," in Proc. Int. Conf. IDEMI, Porto, Portugal, Sep. 4-6, 2013, pp. 1-10.

[31]

J. M. Wolfe and T. S. Horowitz, "What attributes guide the deployment of visual attention and how do they do it?" Nat. Rev. Neurosci., vol. 5, no. 6, pp. 1-7, Jun. 2004.

Cited By

Tsai WBarch DCassidy ADeBole MAndreopoulos AJackson BFlickner MArthur JModha DSampson JNarayanan V(2017)Always-On Speech Recognition Using TrueNorth, a Reconfigurable, Neurosynaptic ProcessorIEEE Transactions on Computers10.1109/TC.2016.263068366:6(996-1007)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1109/TC.2016.2630683
Akopyan FSawada JCassidy AAlvarez-Icaza RArthur JMerolla PImam NNakamura YDatta PGi-Joon Nam Taba BBeakes MBrezzo BKuang JManohar RRisk WJackson BModha D(2015)TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic ChipIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2015.247439634:10(1537-1557)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1109/TCAD.2015.2474396

Recommendations

A discriminant hypothesis for visual saliency: computational principles, biological plausibility and applications in computer vision
A depth perception and visual comfort guided computational model for stereoscopic 3D visual saliency

With the emerging development of three-dimensional (3D) related technologies, 3D visual saliency modeling is becoming particularly important and challenging. This paper presents a new depth perception and visual comfort guided saliency computational ...
Visual saliency estimation using constraints
Abstract
In this paper, we propose visual saliency estimation using constraints. Based on the observations that salient regions are generally distinctive from the background, we define visual saliency as the possibility of being assigned to the ...

Comments

Information & Contributors

Information

Published In

cover image IBM Journal of Research and Development

IBM Journal of Research and Development Volume 59, Issue 2-3

March/May 2015

204 pages

ISSN:0018-8646

Editors:
Aya Soffer
IBM Research-IBM Israel, Haifa Research Lab.
,
Hui Su
IBM Research-IBM, T. J. Watson Research Center

Issue’s Table of Contents

Publisher

IBM Corp.

United States

Publication History

Published: 01 March 2015

Received: 01 May 2014

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tsai WBarch DCassidy ADeBole MAndreopoulos AJackson BFlickner MArthur JModha DSampson JNarayanan V(2017)Always-On Speech Recognition Using TrueNorth, a Reconfigurable, Neurosynaptic ProcessorIEEE Transactions on Computers10.1109/TC.2016.263068366:6(996-1007)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1109/TC.2016.2630683
Akopyan FSawada JCassidy AAlvarez-Icaza RArthur JMerolla PImam NNakamura YDatta PGi-Joon Nam Taba BBeakes MBrezzo BKuang JManohar RRisk WJackson BModha D(2015)TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic ChipIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2015.247439634:10(1537-1557)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1109/TCAD.2015.2474396

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents