We investigate the asymptotic version of the Erdős–Ko–Rado theorem for the random k-uniform hyper... more We investigate the asymptotic version of the Erdős–Ko–Rado theorem for the random k-uniform hypergraph $\mathcal{H}$ k (n, p). For 2⩽k(n) ⩽ n/2, let $N=\binom{n}k$ and $D=\binom{n-k}k$ . We show that with probability tending to 1 as n → ∞, the largest intersecting subhypergraph of $\mathcal{H}$ has size $$(1+o(1))p\ffrac kn N$$ for any $$p\gg \ffrac nk\ln^2\biggl(\ffrac nk\biggr)D^{-1}.$$ This lower bound on p is asymptotically best possible for k = Θ(n). For this range of k and p, we are able to show stability as well. A different behaviour occurs when k = o(n). In this case, the lower bound on p is almost optimal. Further, for the small interval D −1 ≪ p ⩽ (n/k)1−ϵ D −1, the largest intersecting subhypergraph of $\mathcal{H}$ k (n, p) has size Θ(ln(pD)ND −1), provided that $k \gg \sqrt{n \ln n}$ . Together with previous work of Balogh, Bohman and Mubayi, these results settle the asymptotic size of the largest intersecting family in $\mathcal{H}$ k , for essentially all values of p...
Multi-agent consensus problems can often be seen as a sequence of autonomous and independent loca... more Multi-agent consensus problems can often be seen as a sequence of autonomous and independent local choices between a finite set of decision options, with each local choice undertaken simultaneously, and with a shared goal of achieving a global consensus state. Being able to estimate probabilities for the different outcomes and to predict how long it takes for a consensus to be formed, if ever, are core issues for such protocols. Little attention has been given to protocols in which agents can remember past or outdated states. In this paper, we propose a framework to study what we call memory consensus protocol. We show that the employment of memory allows such processes to always converge, as well as, in some scenarios, such as cycles, converge faster. We provide a theoretical analysis of the probability of each option eventually winning such processes based on the initial opinions expressed by agents. Further, we perform experiments to investigate network topologies in which agents...
The sharp wave ripple complex in rodent hippocampus is associated with a network burst in CA3 (NB... more The sharp wave ripple complex in rodent hippocampus is associated with a network burst in CA3 (NB) that triggers a synchronous event in the CA1 population (SE). The number of CA1 pyramidal cells participating in a SE has been observed to follow a lognormal distribution. However, the origin of this skewed and heavy‐tailed distribution of population synchrony in CA1 remains unknown. Because the size of SEs is likely to originate from the size of the NBs and the underlying neural circuitry, we model the CA3‐CA1 circuit to study the underlying mechanisms and their functional implications. We show analytically that if the size of a NB in CA3 is distributed according to a normal distribution, then the size of the resulting SE in CA1 follows a lognormal distribution. Our model predicts the distribution of the NB size in CA3, which remains to be tested experimentally. Moreover, we show that a putative lognormal NB size distribution leads to an extremely heavy‐tailed SE size distribution in CA1, contradicting experimental evidence. In conclusion, our model provides general insight on the origin of lognormally distributed network synchrony as a consequence of synchronous synaptic transmission of normally distributed input events.
The connection density of nearby neurons in the cortex has been observed to be around 0.1, wherea... more The connection density of nearby neurons in the cortex has been observed to be around 0.1, whereas the longer-range connections are present with much sparser density (Kalisman, Silberberg, & Markram, 2005 ). We propose a memory association model that qualitatively explains these empirical observations. The model we consider is a multiassociative, sparse, Willshaw-like model consisting of binary threshold neurons and binary synapses. It uses recurrent synapses for iterative retrieval of stored memories. We quantify the usefulness of recurrent synapses by simulating the model for small network sizes and by doing a precise mathematical analysis for large network sizes. Given the network parameters, we can determine the precise values of recurrent and afferent synapse densities that optimize the storage capacity of the network. If the network size is like that of a cortical column, then the predicted optimal recurrent density lies in a range that is compatible with biological measurements. Furthermore, we show that our model is able to surpass the standard Willshaw model in the multiassociative case if the information capacity is normalized per strong synapse or per bits required to store the model, as considered in Knoblauch, Palm, and Sommer ( 2010 ).
Anais do XIII Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL 2021)
This work explores speech as a biomarker and investigates the detection of respiratory insufficie... more This work explores speech as a biomarker and investigates the detection of respiratory insufficiency (RI) by analyzing speech samples. Previous work [Casanova et al. 2021] constructed a dataset of respiratory insufficiency COVID-19 patient utterances and analyzed it by means of a convolutional neural network achieving an accuracy of 87.04%, validating the hypothesis that one can detect RI through speech. Here, we study how Transformer neural network architectures can improve the performance on RI detection. This approach enables construction of an acoustic model. By choosing the correct pretraining technique, we generate a self-supervised acoustic model, leading to improved performance (96.53%) of Transformers for RI detection.
Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for de... more Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning. We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES. We propose a novel method to optimally incorporate surrogate gradient information. Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to find a descent direction that is better than the surrogate gradient. This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point. We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it significantly improves gradient estimates for general functions. Finally, we evaluate our approach empirically on MNIST and reinforcement learning tas...
Multi-agent consensus problems can often be seen as a sequence of autonomous and independent loca... more Multi-agent consensus problems can often be seen as a sequence of autonomous and independent local choices between a finite set of decision options, with each local choice undertaken simultaneously, and with a shared goal of achieving a global consensus state. Being able to estimate probabilities for the different outcomes and to predict how long it takes for a consensus to be formed, if ever, are core issues for such protocols. Little attention has been given to protocols in which agents can remember past or outdated states. In this paper, we propose a framework to study what we call memory consensus protocol. We show that the employment of memory allows such processes to always converge, as well as, in some scenarios, such as cycles, converge faster. We provide a theoretical analysis of the probability of each option eventually winning such processes based on the initial opinions expressed by agents. Further, we perform experiments to investigate network topologies in which agents...
One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies i... more One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data. Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon. In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs. Recently published approaches reduce these costs by providing noisy approximations of RTRL. We present a new approximation algorithm of RTRL, Optimal Kronecker-Sum Approximation (OK). We prove that OK is optimal for a class of approximations of RTRL, which includes all approaches published so far. Additionally, we show that OK has empirically negligible noise: Unlike previous algorithms it matches TBPTT in a real world task (character-level Penn TreeBank) and can exploit online parameter updates to outperform TBPTT in a synthet...
In computational neural network models, neurons are usually allowed to excite some and inhibit ot... more In computational neural network models, neurons are usually allowed to excite some and inhibit other neurons, depending on the weight of their synaptic connections. The traditional way to transform such networks into networks that obey Daleâs law (i.e., a neuron can either excite or inhibit) is to accompany each excitatory neuron with an inhibitory one through which inhibitory signals are mediated. However, this requires an equal number of excitatory and inhibitory neurons, whereas a realistic number of inhibitory neurons is much smaller. In this letter, we propose a model of nonlinear interaction of inhibitory synapses on dendritic compartments of excitatory neurons that allows the excitatory neurons to mediate inhibitory signals through a subset of the inhibitory population. With this construction, the number of required inhibitory neurons can be reduced tremendously.
We investigate the asymptotic version of the Erdős–Ko–Rado theorem for the random k-uniform hyper... more We investigate the asymptotic version of the Erdős–Ko–Rado theorem for the random k-uniform hypergraph $\mathcal{H}$ k (n, p). For 2⩽k(n) ⩽ n/2, let $N=\binom{n}k$ and $D=\binom{n-k}k$ . We show that with probability tending to 1 as n → ∞, the largest intersecting subhypergraph of $\mathcal{H}$ has size $$(1+o(1))p\ffrac kn N$$ for any $$p\gg \ffrac nk\ln^2\biggl(\ffrac nk\biggr)D^{-1}.$$ This lower bound on p is asymptotically best possible for k = Θ(n). For this range of k and p, we are able to show stability as well. A different behaviour occurs when k = o(n). In this case, the lower bound on p is almost optimal. Further, for the small interval D −1 ≪ p ⩽ (n/k)1−ϵ D −1, the largest intersecting subhypergraph of $\mathcal{H}$ k (n, p) has size Θ(ln(pD)ND −1), provided that $k \gg \sqrt{n \ln n}$ . Together with previous work of Balogh, Bohman and Mubayi, these results settle the asymptotic size of the largest intersecting family in $\mathcal{H}$ k , for essentially all values of p...
Multi-agent consensus problems can often be seen as a sequence of autonomous and independent loca... more Multi-agent consensus problems can often be seen as a sequence of autonomous and independent local choices between a finite set of decision options, with each local choice undertaken simultaneously, and with a shared goal of achieving a global consensus state. Being able to estimate probabilities for the different outcomes and to predict how long it takes for a consensus to be formed, if ever, are core issues for such protocols. Little attention has been given to protocols in which agents can remember past or outdated states. In this paper, we propose a framework to study what we call memory consensus protocol. We show that the employment of memory allows such processes to always converge, as well as, in some scenarios, such as cycles, converge faster. We provide a theoretical analysis of the probability of each option eventually winning such processes based on the initial opinions expressed by agents. Further, we perform experiments to investigate network topologies in which agents...
The sharp wave ripple complex in rodent hippocampus is associated with a network burst in CA3 (NB... more The sharp wave ripple complex in rodent hippocampus is associated with a network burst in CA3 (NB) that triggers a synchronous event in the CA1 population (SE). The number of CA1 pyramidal cells participating in a SE has been observed to follow a lognormal distribution. However, the origin of this skewed and heavy‐tailed distribution of population synchrony in CA1 remains unknown. Because the size of SEs is likely to originate from the size of the NBs and the underlying neural circuitry, we model the CA3‐CA1 circuit to study the underlying mechanisms and their functional implications. We show analytically that if the size of a NB in CA3 is distributed according to a normal distribution, then the size of the resulting SE in CA1 follows a lognormal distribution. Our model predicts the distribution of the NB size in CA3, which remains to be tested experimentally. Moreover, we show that a putative lognormal NB size distribution leads to an extremely heavy‐tailed SE size distribution in CA1, contradicting experimental evidence. In conclusion, our model provides general insight on the origin of lognormally distributed network synchrony as a consequence of synchronous synaptic transmission of normally distributed input events.
The connection density of nearby neurons in the cortex has been observed to be around 0.1, wherea... more The connection density of nearby neurons in the cortex has been observed to be around 0.1, whereas the longer-range connections are present with much sparser density (Kalisman, Silberberg, & Markram, 2005 ). We propose a memory association model that qualitatively explains these empirical observations. The model we consider is a multiassociative, sparse, Willshaw-like model consisting of binary threshold neurons and binary synapses. It uses recurrent synapses for iterative retrieval of stored memories. We quantify the usefulness of recurrent synapses by simulating the model for small network sizes and by doing a precise mathematical analysis for large network sizes. Given the network parameters, we can determine the precise values of recurrent and afferent synapse densities that optimize the storage capacity of the network. If the network size is like that of a cortical column, then the predicted optimal recurrent density lies in a range that is compatible with biological measurements. Furthermore, we show that our model is able to surpass the standard Willshaw model in the multiassociative case if the information capacity is normalized per strong synapse or per bits required to store the model, as considered in Knoblauch, Palm, and Sommer ( 2010 ).
Anais do XIII Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL 2021)
This work explores speech as a biomarker and investigates the detection of respiratory insufficie... more This work explores speech as a biomarker and investigates the detection of respiratory insufficiency (RI) by analyzing speech samples. Previous work [Casanova et al. 2021] constructed a dataset of respiratory insufficiency COVID-19 patient utterances and analyzed it by means of a convolutional neural network achieving an accuracy of 87.04%, validating the hypothesis that one can detect RI through speech. Here, we study how Transformer neural network architectures can improve the performance on RI detection. This approach enables construction of an acoustic model. By choosing the correct pretraining technique, we generate a self-supervised acoustic model, leading to improved performance (96.53%) of Transformers for RI detection.
Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for de... more Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning. We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES. We propose a novel method to optimally incorporate surrogate gradient information. Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to find a descent direction that is better than the surrogate gradient. This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point. We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it significantly improves gradient estimates for general functions. Finally, we evaluate our approach empirically on MNIST and reinforcement learning tas...
Multi-agent consensus problems can often be seen as a sequence of autonomous and independent loca... more Multi-agent consensus problems can often be seen as a sequence of autonomous and independent local choices between a finite set of decision options, with each local choice undertaken simultaneously, and with a shared goal of achieving a global consensus state. Being able to estimate probabilities for the different outcomes and to predict how long it takes for a consensus to be formed, if ever, are core issues for such protocols. Little attention has been given to protocols in which agents can remember past or outdated states. In this paper, we propose a framework to study what we call memory consensus protocol. We show that the employment of memory allows such processes to always converge, as well as, in some scenarios, such as cycles, converge faster. We provide a theoretical analysis of the probability of each option eventually winning such processes based on the initial opinions expressed by agents. Further, we perform experiments to investigate network topologies in which agents...
One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies i... more One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data. Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon. In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs. Recently published approaches reduce these costs by providing noisy approximations of RTRL. We present a new approximation algorithm of RTRL, Optimal Kronecker-Sum Approximation (OK). We prove that OK is optimal for a class of approximations of RTRL, which includes all approaches published so far. Additionally, we show that OK has empirically negligible noise: Unlike previous algorithms it matches TBPTT in a real world task (character-level Penn TreeBank) and can exploit online parameter updates to outperform TBPTT in a synthet...
In computational neural network models, neurons are usually allowed to excite some and inhibit ot... more In computational neural network models, neurons are usually allowed to excite some and inhibit other neurons, depending on the weight of their synaptic connections. The traditional way to transform such networks into networks that obey Daleâs law (i.e., a neuron can either excite or inhibit) is to accompany each excitatory neuron with an inhibitory one through which inhibitory signals are mediated. However, this requires an equal number of excitatory and inhibitory neurons, whereas a realistic number of inhibitory neurons is much smaller. In this letter, we propose a model of nonlinear interaction of inhibitory synapses on dendritic compartments of excitatory neurons that allows the excitatory neurons to mediate inhibitory signals through a subset of the inhibitory population. With this construction, the number of required inhibitory neurons can be reduced tremendously.
Uploads
Papers by Marcelo Matheus Gauy