Twenty-Third Annual Conference on Neural Information Processing Systems, 2009
We introduce a new family of distributions, called Lp-nested symmetric distributions, whose densi... more We introduce a new family of distributions, called Lp-nested symmetric distributions, whose densities are expressed in terms of a hierarchical cascade of Lpnorms. This class generalizes the family of spherically and Lp-spherically symmetric distributions which have recently been successfully used for natural image modeling. Similar to those distributions it allows for a nonlinear mechanism to reduce the dependencies between its variables. With suitable choices of the parameters and norms, this family includes the Independent ...
Abstract Information processing in neocortex can be very fast, indicating that neuronal ensembles... more Abstract Information processing in neocortex can be very fast, indicating that neuronal ensembles faithfully transmit rapidly changing signals to each other. Apart from signal-to-noise issues, population codes are fundamentally constrained by the neuronal dynamics. In particular, the biophysical properties of individual neurons and collective phenomena may substantially limit the speed at which a graded signal can be represented by the activity of an ensemble.
We present a probabilistic model for natural images that is based on mixtures of Gaussian scale m... more We present a probabilistic model for natural images that is based on mixtures of Gaussian scale mixtures and a simple multiscale representation. We show that it is able to generate images with interesting higher-order correlations when trained on natural images or samples from an occlusion-based model. More importantly, our multiscale model allows for a principled evaluation.
Orientation selectivity is the most striking feature of simple cell coding in V1 that has been sh... more Orientation selectivity is the most striking feature of simple cell coding in V1 that has been shown to emerge from the reduction of higher-order correlations in natural images in a large variety of statistical image models. The most parsimonious one among these models is linear Independent Component Analysis (ICA), whereas second-order decorrelation transformations such as Principal Component Analysis (PCA) do not yield oriented filters.
While there are many experiments providing evidence for synchronized neuronal activity, there is ... more While there are many experiments providing evidence for synchronized neuronal activity, there is little agreement about its functional role. Since many proposals rely on the assumption that neuronal activity can be modulated by top-down or feedback signals in a multiplicative way, it is a critical question how the dynamics of neurons may account for a selective control of their gain. In this paper we present a novel gain control mechanism based on the interplay of synaptic depression and synchronous inhibition.
Training sparse natural image models with a fast Gibbs sampler of an extended state space Jascha ... more Training sparse natural image models with a fast Gibbs sampler of an extended state space Jascha Sohl-Dickstein Redwood Center for Theoretical Neuroscience jascha@berkeley. edu Lucas Theis Werner Reichardt Centre for Integrative Neuroscience lucas@bethgelab.
abstract={Generalized linear models are the most commonly used tools to describe the stimulus sel... more abstract={Generalized linear models are the most commonly used tools to describe the stimulus selectivity of sensory neurons. Here we present a Bayesian treatment of such models. Using the expectation propagation algorithm, we are able to approximate the full posterior distribution over all weights. In addition, we use a Laplacian prior to favor sparse solutions. Therefore, stimulus features that do not critically influence neural activity will be assigned zero weights and thus be effectively excluded by the model.
Abstract Correlated trial-to-trial variability in the activity of cortical neurons is thought to ... more Abstract Correlated trial-to-trial variability in the activity of cortical neurons is thought to reflect the functional connectivity of the circuit. Many cortical areas are organized into functional columns, in which neurons are believed to be densely connected and to share common input. Numerous studies report a high degree of correlated variability between nearby cells.
Orientation selectivity is the most striking feature of simple cell coding in V1 that has been sh... more Orientation selectivity is the most striking feature of simple cell coding in V1 that has been shown to emerge from the reduction of higher-order correlations in natural images in a large variety of statistical image models. The most parsimonious one among these models is linear Independent Component Analysis (ICA), whereas second-order decorrelation transformations such as Principal Component Analysis (PCA) do not yield oriented filters.
Abstract A key hypothesis in sensory system neuroscience is that sensory representations are adap... more Abstract A key hypothesis in sensory system neuroscience is that sensory representations are adapted to the statistical regularities in sensory signals and thereby incorporate knowledge about the outside world. Supporting this hypothesis, several probabilistic models of local natural image regularities have been proposed that reproduce neural response properties. Although many such physiological links have been made, these models have not been linked directly to visual sensitivity.
Abstract Divisive normalization in primary visual cortex has been linked to adaptation to natural... more Abstract Divisive normalization in primary visual cortex has been linked to adaptation to natural image statistics in accordance to Barlow's redundancy reduction hypothesis. Using recent advances in natural image modeling, we show that the previously studied static model of divisive normalization is rather inefficient in reducing local contrast correlations, but that a simple temporal contrast adaptation mechanism of the half-saturation constant can substantially increase its efficiency.
Abstract Orientation tuning has been a classic model for understanding single-neuron computation ... more Abstract Orientation tuning has been a classic model for understanding single-neuron computation in the neocortex. However, little is known about how orientation can be read out from the activity of neural populations, in particular in alert animals. Our study is a first step toward that goal. We recorded from up to 20 well isolated single neurons in the primary visual cortex of alert macaques simultaneously and applied a simple, neurally plausible decoder to read out the population code.
Abstract. Simultaneously recorded neurons often exhibit correlations in their spiking activity. T... more Abstract. Simultaneously recorded neurons often exhibit correlations in their spiking activity. These correlations shape the statistical structure of the population activity, and can lead to substantial redundancy across neurons. Here, we study the effect of pairwise correlations on the population spike count statistics and redundancy in populations of threshold-neurons in which response-correlations arise from correlated Gaussian inputs.
Zusammenfassung Untersuchungen zur neuronalen Signalverarbeitung beruhen auf Analysen, die die Ak... more Zusammenfassung Untersuchungen zur neuronalen Signalverarbeitung beruhen auf Analysen, die die Aktivität von Nervenzellen mit Reizen korrelieren. Neue Fortschritte haben die simultane Aufnahme der Aktivitäten vieler Nervenzellen ermöglicht. Mit mathematischen Methoden werden diese dazu genutzt, die Antworten ganzer Netzwerke zu bestimmen. Dies ermöglicht Analysen, wie auf der Netzhaut eingehende Bilder von den Netzwerken der Sehbahn verarbeitet werden.
Abstract Here we derive optimal gain functions for minimum mean square reconstruction from neural... more Abstract Here we derive optimal gain functions for minimum mean square reconstruction from neural rate responses subjected to Poisson noise. The shape of these functions strongly depends on the length T of the time window within which spikes are counted in order to estimate the underlying firing rate. A phase transition towards pure binary encoding occurs if the maximum mean spike count becomes smaller than approximately three provided the minimum tiring rate is zero.
Abstract Maximum entropy analysis of binary variables provides an elegant way for studying the ro... more Abstract Maximum entropy analysis of binary variables provides an elegant way for studying the role of pairwise correlations in neural populations. Unfortunately, these approaches suffer from their poor scalability to high dimensions. In sensory coding, however, high-dimensional data is ubiquitous. Here, we introduce a new approach using a near-maximum entropy model, that makes this type of analysis feasible for very high-dimensional data—the model parameters can be derived in closed form and sampling is easy.
We present a framework for efficient, accurate approximate Bayesian inference in generalized line... more We present a framework for efficient, accurate approximate Bayesian inference in generalized linear models (GLMs), based on the expectation propagation (EP) technique. The parameters can be endowed with a factorizing prior distribution, encoding properties such as sparsity or non-negativity. The central role of posterior log-concavity in Bayesian GLMs is emphasized and related to stability issues in EP.
Here, we study two different approaches to estimate the multi-information of natural images. In b... more Here, we study two different approaches to estimate the multi-information of natural images. In both cases, we begin with a whitening step. Then, in the first approach, we use a hierarchical multi-layer ICA model [1] which is an efficient variant of projection pursuit density estimation. Projection pursuit [2] is a nonparametric density estimation technique with universal approximation properties. That is, it can be proven to converge to the true distribution in the limit of infinite amount of data and layers.
Abstract Cortical circuits perform the computations underlying rapid perceptual decisions within ... more Abstract Cortical circuits perform the computations underlying rapid perceptual decisions within a few dozen milliseconds with each neuron emitting only a few spikes. Under these conditions, the theoretical analysis of neural population codes is challenging, as the most commonly used theoretical tool—Fisher information—can lead to erroneous conclusions about the optimality of different coding schemes.
abstract={Extracellular voltage fluctuations (local field potentials, LFPs) reflecting neural mas... more abstract={Extracellular voltage fluctuations (local field potentials, LFPs) reflecting neural mass action are ubiquitous across species and brain regions. Numerous studies have characterized the properties of LFP signals in the cortex to study sensory and motor computations as well as cognitive processes like attention, perception and memory. In addition, its extracranial counterpart the electroencephalogram is widely used in clinical applications.
Twenty-Third Annual Conference on Neural Information Processing Systems, 2009
We introduce a new family of distributions, called Lp-nested symmetric distributions, whose densi... more We introduce a new family of distributions, called Lp-nested symmetric distributions, whose densities are expressed in terms of a hierarchical cascade of Lpnorms. This class generalizes the family of spherically and Lp-spherically symmetric distributions which have recently been successfully used for natural image modeling. Similar to those distributions it allows for a nonlinear mechanism to reduce the dependencies between its variables. With suitable choices of the parameters and norms, this family includes the Independent ...
Abstract Information processing in neocortex can be very fast, indicating that neuronal ensembles... more Abstract Information processing in neocortex can be very fast, indicating that neuronal ensembles faithfully transmit rapidly changing signals to each other. Apart from signal-to-noise issues, population codes are fundamentally constrained by the neuronal dynamics. In particular, the biophysical properties of individual neurons and collective phenomena may substantially limit the speed at which a graded signal can be represented by the activity of an ensemble.
We present a probabilistic model for natural images that is based on mixtures of Gaussian scale m... more We present a probabilistic model for natural images that is based on mixtures of Gaussian scale mixtures and a simple multiscale representation. We show that it is able to generate images with interesting higher-order correlations when trained on natural images or samples from an occlusion-based model. More importantly, our multiscale model allows for a principled evaluation.
Orientation selectivity is the most striking feature of simple cell coding in V1 that has been sh... more Orientation selectivity is the most striking feature of simple cell coding in V1 that has been shown to emerge from the reduction of higher-order correlations in natural images in a large variety of statistical image models. The most parsimonious one among these models is linear Independent Component Analysis (ICA), whereas second-order decorrelation transformations such as Principal Component Analysis (PCA) do not yield oriented filters.
While there are many experiments providing evidence for synchronized neuronal activity, there is ... more While there are many experiments providing evidence for synchronized neuronal activity, there is little agreement about its functional role. Since many proposals rely on the assumption that neuronal activity can be modulated by top-down or feedback signals in a multiplicative way, it is a critical question how the dynamics of neurons may account for a selective control of their gain. In this paper we present a novel gain control mechanism based on the interplay of synaptic depression and synchronous inhibition.
Training sparse natural image models with a fast Gibbs sampler of an extended state space Jascha ... more Training sparse natural image models with a fast Gibbs sampler of an extended state space Jascha Sohl-Dickstein Redwood Center for Theoretical Neuroscience jascha@berkeley. edu Lucas Theis Werner Reichardt Centre for Integrative Neuroscience lucas@bethgelab.
abstract={Generalized linear models are the most commonly used tools to describe the stimulus sel... more abstract={Generalized linear models are the most commonly used tools to describe the stimulus selectivity of sensory neurons. Here we present a Bayesian treatment of such models. Using the expectation propagation algorithm, we are able to approximate the full posterior distribution over all weights. In addition, we use a Laplacian prior to favor sparse solutions. Therefore, stimulus features that do not critically influence neural activity will be assigned zero weights and thus be effectively excluded by the model.
Abstract Correlated trial-to-trial variability in the activity of cortical neurons is thought to ... more Abstract Correlated trial-to-trial variability in the activity of cortical neurons is thought to reflect the functional connectivity of the circuit. Many cortical areas are organized into functional columns, in which neurons are believed to be densely connected and to share common input. Numerous studies report a high degree of correlated variability between nearby cells.
Orientation selectivity is the most striking feature of simple cell coding in V1 that has been sh... more Orientation selectivity is the most striking feature of simple cell coding in V1 that has been shown to emerge from the reduction of higher-order correlations in natural images in a large variety of statistical image models. The most parsimonious one among these models is linear Independent Component Analysis (ICA), whereas second-order decorrelation transformations such as Principal Component Analysis (PCA) do not yield oriented filters.
Abstract A key hypothesis in sensory system neuroscience is that sensory representations are adap... more Abstract A key hypothesis in sensory system neuroscience is that sensory representations are adapted to the statistical regularities in sensory signals and thereby incorporate knowledge about the outside world. Supporting this hypothesis, several probabilistic models of local natural image regularities have been proposed that reproduce neural response properties. Although many such physiological links have been made, these models have not been linked directly to visual sensitivity.
Abstract Divisive normalization in primary visual cortex has been linked to adaptation to natural... more Abstract Divisive normalization in primary visual cortex has been linked to adaptation to natural image statistics in accordance to Barlow's redundancy reduction hypothesis. Using recent advances in natural image modeling, we show that the previously studied static model of divisive normalization is rather inefficient in reducing local contrast correlations, but that a simple temporal contrast adaptation mechanism of the half-saturation constant can substantially increase its efficiency.
Abstract Orientation tuning has been a classic model for understanding single-neuron computation ... more Abstract Orientation tuning has been a classic model for understanding single-neuron computation in the neocortex. However, little is known about how orientation can be read out from the activity of neural populations, in particular in alert animals. Our study is a first step toward that goal. We recorded from up to 20 well isolated single neurons in the primary visual cortex of alert macaques simultaneously and applied a simple, neurally plausible decoder to read out the population code.
Abstract. Simultaneously recorded neurons often exhibit correlations in their spiking activity. T... more Abstract. Simultaneously recorded neurons often exhibit correlations in their spiking activity. These correlations shape the statistical structure of the population activity, and can lead to substantial redundancy across neurons. Here, we study the effect of pairwise correlations on the population spike count statistics and redundancy in populations of threshold-neurons in which response-correlations arise from correlated Gaussian inputs.
Zusammenfassung Untersuchungen zur neuronalen Signalverarbeitung beruhen auf Analysen, die die Ak... more Zusammenfassung Untersuchungen zur neuronalen Signalverarbeitung beruhen auf Analysen, die die Aktivität von Nervenzellen mit Reizen korrelieren. Neue Fortschritte haben die simultane Aufnahme der Aktivitäten vieler Nervenzellen ermöglicht. Mit mathematischen Methoden werden diese dazu genutzt, die Antworten ganzer Netzwerke zu bestimmen. Dies ermöglicht Analysen, wie auf der Netzhaut eingehende Bilder von den Netzwerken der Sehbahn verarbeitet werden.
Abstract Here we derive optimal gain functions for minimum mean square reconstruction from neural... more Abstract Here we derive optimal gain functions for minimum mean square reconstruction from neural rate responses subjected to Poisson noise. The shape of these functions strongly depends on the length T of the time window within which spikes are counted in order to estimate the underlying firing rate. A phase transition towards pure binary encoding occurs if the maximum mean spike count becomes smaller than approximately three provided the minimum tiring rate is zero.
Abstract Maximum entropy analysis of binary variables provides an elegant way for studying the ro... more Abstract Maximum entropy analysis of binary variables provides an elegant way for studying the role of pairwise correlations in neural populations. Unfortunately, these approaches suffer from their poor scalability to high dimensions. In sensory coding, however, high-dimensional data is ubiquitous. Here, we introduce a new approach using a near-maximum entropy model, that makes this type of analysis feasible for very high-dimensional data—the model parameters can be derived in closed form and sampling is easy.
We present a framework for efficient, accurate approximate Bayesian inference in generalized line... more We present a framework for efficient, accurate approximate Bayesian inference in generalized linear models (GLMs), based on the expectation propagation (EP) technique. The parameters can be endowed with a factorizing prior distribution, encoding properties such as sparsity or non-negativity. The central role of posterior log-concavity in Bayesian GLMs is emphasized and related to stability issues in EP.
Here, we study two different approaches to estimate the multi-information of natural images. In b... more Here, we study two different approaches to estimate the multi-information of natural images. In both cases, we begin with a whitening step. Then, in the first approach, we use a hierarchical multi-layer ICA model [1] which is an efficient variant of projection pursuit density estimation. Projection pursuit [2] is a nonparametric density estimation technique with universal approximation properties. That is, it can be proven to converge to the true distribution in the limit of infinite amount of data and layers.
Abstract Cortical circuits perform the computations underlying rapid perceptual decisions within ... more Abstract Cortical circuits perform the computations underlying rapid perceptual decisions within a few dozen milliseconds with each neuron emitting only a few spikes. Under these conditions, the theoretical analysis of neural population codes is challenging, as the most commonly used theoretical tool—Fisher information—can lead to erroneous conclusions about the optimality of different coding schemes.
abstract={Extracellular voltage fluctuations (local field potentials, LFPs) reflecting neural mas... more abstract={Extracellular voltage fluctuations (local field potentials, LFPs) reflecting neural mass action are ubiquitous across species and brain regions. Numerous studies have characterized the properties of LFP signals in the cortex to study sensory and motor computations as well as cognitive processes like attention, perception and memory. In addition, its extracranial counterpart the electroencephalogram is widely used in clinical applications.
Uploads