Sequence learning in a spiking neuronal network with memristive synapses

In this section, we introduce our model of the ReRAM device and its control circuitry, and characterize the resulting model dynamics.

The conductance of ReRAM devices can be potentiated or depressed, mimicking the plasticity observed in biological synapses. While single memristive devices may readily emulate the inference function, they cannot emulate on their own plasticity rules such as STDP or homeostatic control. The change of the memristive conductivity depends on the momentary voltage difference between its two terminals, and the device has no memory of past spike events at either of its terminals nor of their relative timing. Hebbian learning such as STDP therefore can only be emulated using a memristive device by 'reshaping' of the pre- and post-synaptic spike events using complex voltage pulses, so that the spike-time dependency is translated into a desired instantaneous voltage difference over the device (Zamarreño-Ramos et al 2011, Wang et al 2015). As a result, the learning rule is controlled outside the actual device (see figure 1). As for implementing the learning, instead of using complex voltage pulse shapes, it is more efficient to use a controller to generate simple rectangular voltage pulses that can effectuate the desired change of the device conductance in a better, more energy efficient, and also more reliable way. The change of the device conductivity as a function of the number of applied voltage pulses can hereby be seen as an intrinsic plasticity curve of the device, where the actual pulse shape can be optimized toward desired potentiation and depression characteristics.

Zoom In Zoom Out Reset image size

Previous studies suggested both physics-based and phenomenological models for VCM-type ReRAMs. Physics-based models such as the JART model (Bengel et al 2020) capture detailed physical characteristics and predict their specific experimental behavior. They require however long simulation time and lead to convergence issues. On the other hand, the more phenomenological models give a high-level description of the operational characteristics, have good accuracy, are computationally less demanding, and can hence be combined with large-scale network models. In this study, we opt for a phenomenological model to implement both the analog and the binary ReRAM device.

The conductivity of the device (i.e. synaptic weight) is either potentiated or depressed by following learning rules similar to those outlined in the spiking TM model (Bouhadjar et al 2022). The learning rules are implemented by the control circuit (figure 1) as follows: the synapse is depressed slightly at every presynaptic spike and potentiated if a postsynaptic spike follows after a presynaptic spike. In contrast to the original spiking TM model, synapses are potentiated by a fixed amount irrespective of the relative timing between the pre- and postsynaptic spikes. The potentiation is however enabled only if the time difference between pre- and postsynaptic spikes is within the interval $[\Delta{}t_\text{min}, \Delta{}t_\text{max}]$. This prohibits synchronously firing neurons from connecting to each other and leads to improved training (Bouhadjar et al 2022). The control circuit further implements a homeostatic control mechanism (see section 2.2): in case the neuronal firing rate exceeds a certain threshold, the potentiation is disabled and instead an additional depression update is applied.

In the analog mode, the increment

$\begin{align} \Delta G_{ij} = \begin{cases} G_\text{max} \cdot\left(\lambda_{+} \cdot \left(1 -\dfrac{G_{ij}}{G_\text{max}}\right)^{\mu_{+}} + X_{ij}\right)&\text{for potentiation}\,\\[12pt] -G_\text{max} \cdot \left(\lambda_{-} \cdot \left(\dfrac{G_{ij}}{G_\text{max}}\right)^{\mu_{-}} + X_{ij}\right)&\text{for depression} \end{cases} \end{align} \tag{ 1 }$

in the conductivity G_ij of the device connecting a presynaptic neuron j to a postsynaptic neuron i following a potentiation or a depression event is modeled as in (Gütig et al 2003, Fusi and Abbott 2007), but with an additional additive noise X_ij . For each synapse and for each update, the noise $X_{ij} \sim \mathcal{N}(0,\,\sigma_\text{w}^{2})$ is randomly and independently drawn from a normal distribution with zero mean and standard deviation $\sigma_\text{w}$. The conductance G_ij evolves between a lower and an upper bound $G_{\text{min},ij}$ and G_max, and it is clipped at these boundaries, with learning rates $\lambda_{+}$ and $\lambda_{-}$ and weight dependence exponents $\mu_{+}$ and $\mu_{-}$. The conductance changes linearly with the internal state variable N_VO, thus no specification of the internal state variable is necessary. The initial conductance $G_{\text{min},ij} = G_{ij}(0)$ is drawn for every new device from a uniform distribution in the interval [$G_{0,\text{min}},G_{0,\text{max}}$].

For the binary switching behavior, we use a similar model as the structural STDP model proposed by Bouhadjar et al (2022). The switching of the conductance between the LCS and the HCS is controlled by a permanence P_ij . The permanence plays the role of the internal state variable N_VO. If it is above a certain threshold $\theta_\text{P}$, the conductance G_ij is set to G_max, otherwise it is set to $G_{\text{min},ij}$ :

$\begin{align} G_{ij}(t) = \begin{cases} G_\text{max} & \mbox{if}\ P_{ij}(t) \unicode{x2A7E} \theta_P \\ G_{\text{min},ij} & \mbox{if}\ P_{ij}(t) \lt \theta_P. \end{cases} \end{align} \tag{ 2 }$

At each potentiation or depression step, the permanence P of the synapse $j\to{}i$ is incremented by an amount

$\begin{align} \Delta P_{ij} = \begin{cases} P_\text{max}\cdot\left(\lambda_{+} \cdot \left(1 - \dfrac{P_{ij}}{P_\text{max}}\right)^{\mu_{+}} + X_{ij}\right) & \text{for potentiation}\,\\[8pt] -P_\text{max}\cdot\left(\lambda_{-}\cdot \left(\dfrac{P_{ij}}{P_\text{max}}\right)^{\mu_{-}} + X_{ij}\right)& \text{for depression}, \end{cases} \end{align} \tag{ 3 }$

similar to the conductance increment of the analog synapse. It has a lower and an upper bound $P_{\text{min},ij}$ and P_max and it is clipped at these boundaries. While the maximum permanences P_max are identical for synapses, the minimal permanences $P_{\text{min},ij}$ and conductances $G_{\text{min}, ij}$ are uniformly distributed in the intervals $[P_{0,\text{min}},P_{0,\text{max}}]$ and $[G_{0,\text{min}},G_{0,\text{max}}]$, respectively.

In addition to the write noise introduced by means of the variable X_ij , both the analog and the binary synapse models incorporate a read noise. At each presynaptic spike of neuron j, a noisy component Z is added to the synaptic current

$\begin{align} I_{ij}(t) = (G_{ij}(t) + G_\text{max} \cdot Z_{ij})\cdot V_\text{read} = \overline{G}_{ij}(t)\cdot V_\text{read}, \end{align} \tag{ 4 }$

of neuron i, where $Z_{ij}\sim\mathcal{N}(0,\,\sigma_\text{r}^{2})$ is randomly and independently drawn from a normal distribution with zero mean and standard deviation $\sigma_\text{r}$, and V_read is the applied voltage. In the course of this article, we use $\overline{G}_{i,j}$ to denote the conductance incorporating both the read and the write noise. This conductance is clamped at zero if it gets negative. In sections 2.3.3 and 3, we motivate these different types of noise both from the hardware and the biological point of view.

Figure 2 shows an exemplary switching behavior of the analog and binary synapse models for a specific set of parameters using 100 consecutive potentiation (i.e. SET) and depression (i.e. RESET) updates. We choose different learning rates ($\lambda_{+}$ and $\lambda_{-}$) for the two types of devices such that they switch from the LCS to the HCS (and back) at about the same number of updates.

Zoom In Zoom Out Reset image size

Learning in the spiking TM model is governed by a homeostatic form of STDP. Each presynaptic spike triggers a small decrease in the synaptic weight (depression). If this presynaptic spike is immediately followed by a postsynaptic spike, this weight decrease is overwritten by a larger weight increase (potentiation). This implementation ensures that synapses are potentiated only if a presynaptic spike is immediately followed by a postsynaptic spike. Presynaptic firing without subsequent postsynaptic firing weakens the synapse. While the potentiation is required to form sequence specific subnetworks, the depression is important to prune unused connections and thereby helps to acquire sparsity and context specificity. Under normal operation, a potentiation update is hence always accompanied by a small amount of depression (see figure S1 in the supplementary materials). In the case of the analog synapse, the total synaptic growth in the absence of noise is therefore governed by

$\begin{align} \Delta G_{ij} = G_\text{max} \left[ \lambda_{+} \cdot \left(1 - \dfrac{G_{ij}}{G_\text{max}}\right)^{\mu_{+}} - \lambda_{-} \cdot \left(\dfrac{G_{ij}}{G_\text{max}}\right)^{\mu_{-}}\right]. \end{align} \tag{ 5 }$

The stationary solution of the device conductance (fixed point) $G^*$, obtained by setting $\Delta G_{ij} = 0$, is always below the maximum conductance G_max (see figure S1(b) in the supplementary materials). The permanence of the binary synapse is subject to this effect, too. After a number of potentiation steps, it reaches a value $P^*$ smaller than P_max (see figure S1(c) in the supplementary materials). According to equation (2), the conductance can however still assume G_max. Only if the depression is too strong, the device may not reach the maturity threshold θ_P , and thus not switch to the HCS.

In the next sections, we evaluate the effects of different characteristics of the analog and the binary switching dynamics such as the weight dependence of the device update ($\mu_{+}$, $\mu_-$), the conductance range (G_min, G_max), the learning rates ($\lambda_{+}$, $\lambda_{-}$), as well as the write and the read variability ($\sigma_\text{w}$, $\sigma_\text{r}$) on the learning process of the spiking TM model.

Sequence learning and prediction are principal computations performed by the brain and have a number of potential technological applications. The spiking temporal memory (spiking TM) model proposed a brain-inspired network of this type of computation Bouhadjar et al (2022). In this section, we utilize the ReRAM device dynamics (see above) to replace the original synaptic model and evaluate the resulting network performance on a sequence prediction task.

We briefly describe here the main mechanisms and principles of the spiking TM model. For an in-depth analysis, we refer readers to (Bouhadjar et al 2022). The model is composed of a ${N_\textrm{E}}$ excitatory ('E') and ${N_\textrm{I}}$ inhibitory ('I') neurons, which are randomly and sparsely connected. Excitatory neurons are organized into M distinct subpopulations, where the neurons in each subpopulation represent a specific sequence element and exhibit a shared stimulus preference (figure 3(a)). Excitatory neurons are recurrently connected to the inhibitory neurons implementing a winner-take-all (WTA) mechanism. We model neurons using leaky integrate-and-fire dynamics. Excitatory neurons are additionally equipped with nonlinear dendrites mimicking dendritic action potentials (dAPs). We model the dAPs as follows: if the dendritic current a threshold $\theta_{\text{dAP}}$, it is instantly set and clamped to the dAP plateau current I_dAP for a period of duration $\tau_\text{dAP}$. The dAP threshold is chosen such that the co-activation of γ presynaptic neurons reliably triggers a dAP in the target neuron:

$\begin{align} \theta_{\text{dAP}} = V_\text{read} \cdot G_{+} \cdot \gamma \cdot p. \end{align} \tag{ 6 }$

In the case of the analog synapse, $G_{+}$ is taken to be the steady-state conductance $G^*$, and in the case of the binary synapse, it is taken to be G_max. In addition to the dendritic input, the excitatory neurons are equipped with additional inputs from external and inhibitory sources. Inhibitory neurons have only excitatory inputs. The synapses between excitatory neurons are plastic evolving according to the analog or the binary ReRAM models described in section 2.1. A homeostatic component further controls the synaptic growth: if the dAP activity, i.e. the number of generated dAPs in a certain time window, is above a target $z^*$, the potentiation is disabled and instead a depression pulse is applied (see section 5).

Zoom In Zoom Out Reset image size

During the learning process, the network is repeatedly presented with a given ensemble of sequences. Before learning, presenting a sequence element causes all neurons in the respective subpopulation to fire, except the subpopulation representing the first sequence element, where only a random subset of neurons is activated. The repeated presentation of the sequences strengthens the connections between the subpopulations representing subsequently presented elements. After sufficient learning, the activation of a subpopulation by an external input causes a specific subset of neurons in the following subpopulation to generate dAPs resulting in a long-lasting depolarization of the somata. Neurons that generate dAPs signal the anticipated sequence element and are thus referred to as predictive neurons. When receiving an external input, predictive neurons fire earlier as compared to non-predictive neurons. If a certain subpopulation contains a sufficient number of predictive neurons, their advanced spike initiates fast and strong inhibitory feedback to the entire subpopulation, ultimately suppressing the firing of the non-predictive neurons. The randomness in the connectivity supplemented by the homeostatic control enables the generation of sequence-specific sparse connectivity patterns between subsequently activated neuronal subpopulations (figures 3(a) and (b)). For each pair of sequence elements in a given sequence ensemble, there is a unique set of postsynaptic neurons generating dAPs. Consequently, after learning in response to the presentation of a sequence element, the network predicts in a context-dependent manner the next element in the sequence by activating the dAPs of the corresponding subpopulation.

Here, we study the prediction performance for the network with either the binary or the analog ReRAM synapses (figure 4). We use the synaptic parameters fitted from the exemplary data discussed in section 2.1. To quantify the sequence prediction performance, we repetitively stimulate the network using the same set of sequences {A,D,B,E,I}, {F,D,B,E,C}, {H,L,J,K,D}, {G,L,J,K,E} and assess the prediction error by comparing the anticipated next sequence element with the correct one (Bouhadjar et al 2022). To ensure the performance results are not specific to a single network, the evaluation is repeated for a number of randomly instantiated network realizations with different initial connectivities. After each new network instantiation, the initial prediction error is at 1 (figure 4). With an increasing number of training episodes, the prediction error for both networks with either the binary or the analog synapses decreases to zero as both networks learn the sequences and develop context-dependent pathways between successive sequence elements.

Zoom In Zoom Out Reset image size

ReRAM devices reported in the literature exhibit different non-idealities, including (1) limited precision or the number of synaptic levels; (2) limited dynamic range; (3) dependence of the synaptic updates on the weight, also known as synaptic nonlinearity; (4) device variability, including read and write variability (see Zhao et al 2020, for an overview). In this section, we study how these non-idealities affect the prediction performance in the spiking TM model.

2.3.1. High prediction performance obtained for a broad range of on-off ratios and learning rates

The dynamic range is defined as the on-off ratio between the maximum (G_max) and the minimum conductance (G_min). Most ReRAM devices exhibit an on-off ratio in a range of 2 to $\gt$ 10⁴ (Hong et al 2018). Within the minimum and the maximum conductance, the synapse can assume different learning rates ($\lambda_{+}$, $\lambda_{-}$). Here, we investigate the influence of different on-off ratios and learning rates on the prediction performance. Note that the learning rates directly influence the number of synaptic levels, with higher rates causing the conductance to transition more rapidly from LCS to HCS, resulting in fewer level crossings.

We first evaluate how the asymmetry in the learning rates between the potentiation and depression operations ($\lambda_{+}$ and $\lambda_{-}$) affects the prediction performance. To study this effect, we fix $\lambda_{+}$ and vary $\lambda_{-}$ with the state dependence exponents $\mu_{+}$ and $\mu_{-}$ being set to zero. The prediction error remains high if $\lambda_{-}\unicode{x2A7E}\lambda_{+}$ (see figure S2 in the supplementary materials). This is due to the plasticity dynamics of the spiking TM model: the potentiation operation is applied only when the postsynaptic spike follows after the presynaptic spike, in contrast, the RESET operation is applied every time the presynaptic neuron generates a spike. Therefore, for effective synaptic growth, the potentiation needs to be stronger than depression.

We next vary the on-off ratio between 5 and 40 by keeping G_min fixed and varying G_max. As G_min is drawn from a uniform distribution in the interval [$G_{0,\text{min}}$, $G_{0,\text{max}}$], we compute the on-off ratio as $G_\text{max}/G^{*}_\text{min}$, where $G^{*}_\text{min} = (G_{0,\text{max}}+G_{0,\text{min}})/2$. According to equation (6), a change in G_max is accompanied by a change in the dAP threshold. In addition, we vary the learning rate between 0.02 and 0.42 (figure 5). Parameters such as the read and write variability and the weight dependence exponents are taken from the exemplary data presented in section 2.1. We study the influence of the variability and the dependence of the synaptic updates on the weight more systematically in the upcoming sections. Successful learning is obtained for on-off ratios above 10 and 5 and for learning rates below 0.26 and 0.34 for the networks with analog and binary synapses, respectively (figures 5(a) and (c)). For larger learning rates, the prediction performance becomes less stable with occasional failures for some network realizations. While decreasing the learning rate yields minimum prediction error, the number of episodes to solution (i.e. learning speed, see (Bouhadjar et al 2022)) increases as either the conductances or permanences need more learning steps to reach their maximum value (figures 5(b) and (d)). For our choice of parameters (such as $\theta_\text{P}$), learning in the network with binary synapses is slightly faster, because, for identical learning rates, the number of update steps required to switch from the LCS to the HCS is lower for the binary device, as compared to the analog device.

Zoom In Zoom Out Reset image size

In general, the on-off ratio in the spiking TM network is limited due to the following: the transition of the network activity from being initially non-sparse to becoming sparse after learning requires small initial conductances to avoid spurious activation of the dAPs, but high conductances after learning to allow the sparse set of active neurons to generate the dAP reliably. If the on-off ratio is too small this distinction between high and small conductances cannot be realized. Moreover, for successful learning, the network with analog synapses requires a higher minimal on-off ratio compared to the network with binary synapses. This is due to the effect described in section 2.1 below equation (5), which prohibits the conductance from reaching G_max. Therefore, the effective on-off ratio is reduced. The learning mechanisms of the spiking TM also limit the range of possible learning rates. Increasing the learning rate bears the risk that a large fraction of neurons reaches the dAP threshold at the same time. The WTA mechanism selects then all neurons that generate dAP to become active. This leads to a loss of sparseness, which results in impairing the prediction performance. Decreasing the learning rate considerably is also not ideal as the network would learn very slowly.

2.3.2. Resilience against weight dependent updates

The conductance of realistic analog ReRAM devices grows or decays in a nonlinear manner as a function of the number of potentiation or depression update steps. The synapse model in section 2.1 captures this effect by the weight dependence exponents ($\mu_{+}$, $\mu_{-}$). During the potentiation process, the conductance tends to change rapidly at the beginning but saturates at the end of the process (see figure 6(a)). Similar behavior is also observed during the RESET. The potentiation and depression updates have, however, different dependencies on the device conductance. For high conductances, the potentiation increments are much smaller than the depression decrements. This asymmetry in the behavior can be further enhanced if the learning rates are different during the potentiation and depression operations. Similarly, it is reasonable to assume that for the binary synapses the evolution of the permanence may exhibit a weight dependence and an asymmetric behavior between the potentiation and depression dynamics (figure 6(d)).

Zoom In Zoom Out Reset image size

Here, we assess the prediction performance as a function of different weight dependence exponents for both potentiation and depression ($\mu_{+}$ and $\mu_{-}$, respectively, see figure 6). For most exponent combinations studied here, the prediction error is low and varies only mildly with $\mu_+$ and $\mu_-$ (figures 6(b) and (e)). For larger values of $\mu_{+}$, the learning slows down as it takes longer for either the conductance or the permanence to reach their maximum values (see figures 6(c) and (f)). Decreasing $\mu_{-}$ makes learning faster again as the depression becomes weaker compared to the potentiation. In the binary case, the steady-state permanence $P^*$ may end up below the maturity threshold $\theta_\text{P}$ such that the synapses can mature only due to the noise. The learning is therefore slowed down for large values of $\mu_{+}$ or even unsuccessful if the devices do not switch to the HCS. In the model, $\theta_\text{P}$ could be adjusted to $P^*$ (similarly to adjusting $\theta_\text{dAP}$ to $G^*$ in the analog synapse; see above). In this case, the learning in the analog and the binary networks may be similarly fast. In the physical device, however, the maturity threshold $\theta_\text{P}$ can hardly be changed.

2.3.3. Resilience against read and write variability

The resistive switching process (i.e. write process) of ReRAM devices involves the drift and diffusion of the oxygen vacancies. This phenomenon is highly stochastic and shows considerable variation from device to device, and even from pulse to pulse within one device (Zhao et al 2020). Further, even when no switching occurs, the oxygen vacancies exhibit random microscopic displacements resulting in read variability. In our work, we capture these effects by the read and write variability introduced in section 2.1. The influence of the read and write variability on the conductance curves is illustrated for both the analog and binary synapses in figure 7. For different trials, the write variability results in different conductance trajectories. The read variability, on the other hand, causes only a jitter in the conductance curves.

Zoom In Zoom Out Reset image size

To study how the variability influences the prediction performance, we assess the prediction error and episodes to solution for different magnitudes of the read and write variability $\sigma_\text{r}$ and $\sigma_\text{w}$, respectively. Both networks with either analog or binary synapses allow similar read and write noise levels, with the binary synapse being slightly more resilient toward the read noise (figures 7(c) and (g)). In both cases, the write noise is more detrimental as it accumulates across the different learning episodes and can therefore have a higher impact on the learning performance. The read noise tends to average out as it is independent across the learning episodes. Overall, increasing the read or write variability beyond what is acceptable leads to a spurious activation of the dAPs, i.e. predictions, and a decline in the prediction performance. The learning speed (episodes to solution) varies only slightly within the parameter region where learning is successful (figures 7(d) and (h)).

2.3.4. Robustness with respect to device failure

When operating ReRAM devices, they risk failing by getting trapped in the HCS even after applying voltage pulses with the appropriate magnitude across them (Kumar et al 2017). To study how device failure affects the prediction performance, we first train the network till it reaches zero prediction error (after 150 episodes in figure 8). Then, the conductance of a random fraction of synapses is set to the HCS. We quantify the level of device failure by the ratio between the number of failed synapses and the total number of existing synapses. In the spiking TM model, a neuron may falsely generate a dAP if a sufficient number of its synapses are randomly switched to the HCS (this number can be approximated as the ratio $\theta_\text{dAP}/G_\text{max}$, where $\theta_\text{dAP}$ is the dAP threshold and G_max is the maximum conductance). This may result in generating false positives and thus an increase in the prediction error. This is confirmed by our results presented in figures 8(a) and (b). At up to $10\%$ device failure no impact is observed on the prediction performance (figures 8(a) and (b)). At greater than $10\%$ device failure the performance of the network declines and does not recover.

Zoom In Zoom Out Reset image size

In a second experiment, instead of turning a selection of random synapses to the HCS, we turn them to the LCS. For the different levels of device failures, the performance of the network initially declines. Due to the failing synapses, which are stuck at the LCS, the neurons in certain subpopulations do not receive enough current and are thus not able to generate dAPs, i.e. make predictions. After further training episodes, the prediction errors converge back to zero as the network relearns using other synapses (figures 8(c) and (d)). At greater than $20\%$ device failure the performance does not recover due to the absence of alternative connections to form sequence-specific pathways.

In this work, we demonstrate that the learning rules of the spiking temporal memory (spiking TM) model proposed by Bouhadjar et al (2022) can be realized using memristive dynamics. We investigate this for a particular type of memristive device known as VCM ReRAM (Waser 2012a). We show that the spiking TM retains high prediction performance for a broad range of on-off ratios and learning rates. The model is resilient toward the write and read variability as well as the dependence of the synaptic updates on the weight. Moreover, our results show that the VCM-type ReRAM device can be operated either in the binary or the gradual switching regime without performance loss. We note only slight differences: the network with binary synapse is more resilient toward read noise and requires less synaptic on-off ratio. The analog synapse is more robust to larger values of the weight-dependent coefficient $\mu_{+}$. The ability of the network to retain successful performance with both types of synapses (for a broad range of parameters) is in line with the original spiking TM implementation (Bouhadjar et al 2022), which shows that the learning rule can either be implemented using structural plasticity where the weight abruptly changes between two levels or a conventional form of STDP where the weight gradually changes. This suggests that the intrinsic dynamics of the VCM ReRAM capture not only synaptic properties of biological synapses such as the variability and the dependence of the synaptic updates on the weight but also can implement known forms of plasticity in the neuroscientific literature. Our study thereby contributes to establishing a dynamical and functional correspondence between biological synapses and memristive devices.

In artificial neural networks trained by gradient-based approaches, ReRAM non-idealities can severely undermine the overall performance (Fouda et al 2020). Due to the ReRAM variability, devices can be hardly programmed to a desired state, and the asymmetry in the conductance change can affect the propagation of the gradient and lead to performance loss. Correcting for these non-idealities can be costly and may require additional circuitry (Chen et al 2015, Agarwal et al 2016, Ambrogio et al 2018, Hong et al 2018, Yu 2018, Adnan et al 2021). We know that biological neuronal networks carry out accurate computations despite their synaptic non-ideal characteristics such as variability. This suggests the existence of biological principles accommodating that, which we need to understand and port to successfully implement neuromorphic hardware. The spiking TM and other brain-inspired self-organizing networks (Lazar et al 2009, Yi et al 2022) suggest a set of biological concepts that might be at the heart of brain processing capabilities. For instance, the highly sparse connectivity and activity of the spiking TM are observed in biological networks, and they are essential for increasing the capacity of the system and decreasing energy consumption.

There are a number of biologically motivated sequence learning models that are closely related to the spiking TM, such as the self-organizing recurrent neural network model (SORN, Lazar et al 2009). Recent work incorporated memristive dynamics into the synapses and neurons of the SORN model and showed that it retains successful performance (Payvand et al 2022). The authors studied the role of variability and showed that it can improve the prediction performance. However, the other memristive non-idealities were not studied systematically. It remains also to be investigated whether the model can learn high-order sequences similar to the ones presented in our work.

Neuromorphic hardware that relies on components implemented in the analog domain is noisy and heterogeneous, similar to real brains (Zhu et al 2020). To date, there are only speculations on how the brain contributes to sensible and reliable behavior in the face of these imperfections. By using neuromorphic hardware as a test substrate, we expect to gain new insights into the neuronal principles that solve this issue. In this study, it is apparent that ReRAM devices share the same characteristics as biological synapses including the weight dependence of the synaptic updates, limited dynamic range, and variability. Throughout the work on these neuromorphic systems, we can develop intuitions of how the biological synapse exploits these different characteristics. For instance, in biology, the write noise represents the variability in the synaptic weight change following a pre- and/or postsynaptic spike, such as the variability in the postsynaptic receptor synthesis. The read noise, instead, refers to the momentary variability in the postsynaptic response amplitude, which is, for example, caused by a variability in the amount of neurotransmitter released by individual presynaptic spikes. Note that the write noise is accumulated over time, whereas the read noise affects the synaptic weight (conductance) only during the presynaptic spike. In neuroscience studies, synaptic stochasticity (including synaptic failure) is typically regarded as read noise. The role of the write noise is often ignored. So far, it is not clear how these different characteristics contribute to the learning dynamics in the biological system. Neuromorphic hardware can provide an environment where this question can be studied.

In this work, we show that the model is resilient toward synaptic variability. Other works show that synaptic variability can even have a computational benefit (Dalgaty et al 2021). For example, in probabilistic computing frameworks, the variability is considered a prerequisite for efficient probabilistic inference (Buesing et al 2011, Suri et al 2013, Maass 2014, Neftci et al 2016, Dutta et al 2022). It allows the system to explore the state space and come up with an estimate of how likely is each solution. Similarly, a recent extension of the spiking TM model shows that the model can learn to replay probabilistic sequences using noise (Bouhadjar et al 2023). The study demonstrated that, in a network context, variability can only serve exploration if it is locally correlated and hence not averaged out. Understanding whether synaptic or memristive variability could contribute to such a form of noise remains the subject of future studies.

Ultimately, the goal is to implement the spiking TM model on a standalone neuromorphic chip. In this work, we only investigate how the intrinsic properties of the memristive device affect the learning in the spiking TM. A study by (Siegel et al 2023a) provided a specific instance of an electronic circuit design of the hardware implementing the different components of the spiking TM and showed in simulations that the system supports successful prediction performance. A follow-up study by (Siegel et al 2023b) taped out a memristive synaptic array of a simplified spiking TM model and showed successful performance on a simple sequence learning problem. A future study needs to upscale the array size and task difficulty.