Tunable synaptic working memory with volatile memristive devices

The memristive devices used in this study are fabricated on top of foundry-based MOS transistors, namely one-transistor/one-resistor, allowing the control of the compliance current $I_\mathrm{C}$. The bottom electrode (BE) consists of a 70 nm × 70 nm graphitic carbon pillar, that was already demonstrated to be a good electrode material for both volatile and non-volatile devices thanks to its stability and inert behavior [25]. The oxide layer and the top electrode (TE) are fabricated by e-beam evaporation. The oxide is a 10 nm HfO$\mathrm{_2}$ active layer and the TE is a 100 nm thick Ag layer. Both HfO$\mathrm{_2}$ and Ag are deposited at room temperature and without breaking the vacuum (pressure 3×10$\mathrm{^{-6}}$ mbar).

The electrical setup allows the device to be connected either to a semiconductor device parameter analyzer (DC characterization) or to a waveform generator and an oscilloscope (pulsed characterization) through a switch matrix. An example of this configuration has been shown in [25]. DC characterization is carried out using an HP 4156 C semiconductor device parameter analyzer. A sweep from 0 V to 1.5 V is applied to the TE of the device while the BE is grounded. The characterization includes DC sweeps with different compliance current (I $\mathrm{_C}$), as in figure S1(a). The pulsed characterization as well as the WM experiment are carried out using a TTI TGA12104 arbitrary waveform generator. The voltage waveforms are applied to the TE. To measure the current, a LeCroy Waverunner 640Zi oscilloscope is connected at the BE side, and the voltage drop across a 50 Ω series resistance is probed. MATLAB® software is used for the data analysis and the control of the instruments.

The temporal dynamics are studied by first applying a semi-triangular pulse with 10 ms pulse duration and 5 V amplitude to induce the filament formation and then monitoring the state of the filament with constant −150 mV bias. To tune the temporal behavior, the I $\mathrm{_C}$ is changed from 10 µ A to 70 µ A. For each I $\mathrm{_C}$, 100 experiments are carried out. To avoid possible interference due to the previous cycles, the value of I $\mathrm{_C}$ is changed in random order.

The switching probability (figure 2(D)) is studied by applying 100 pulses for each combination of voltage amplitude–pulse duration. The conditions are applied in random order. The impact of the number of pulses (figures 2(D) and (E)) is analyzed with the same methodology, selecting the order of the combination of voltage amplitude–number of pulses randomly. Each combination is applied 100 times.

Zoom In Zoom Out Reset image size

For the WM store and recall experiment, each probability–frequency–pattern condition is repeated ten times. The voltage bias is applied only at the end of the experiments to check the retention capabilities.

Thanks to the stochasticity of the switching mechanism, the probability ${P}_\mathrm{ON}$ for a single pulse of given amplitude (figure 2(D)) follows a normal distribution and thus is fit using a cumulative distribution function:

$\begin{equation} {\Large\displaystyle{P_\mathrm{ON}\left(V\right) = \frac{1}{2}\left(1+\mathrm{erf}\left(\frac{V-\mu}{\sigma\sqrt{2}}\right)\right)}} \end{equation} \tag{ 1 }$

where the average value µ and variance σ are fitting parameters of the device and the pulse duration, respectively. Table 1 collects the data of the device presented in figure 2(D). Each point in the figure corresponds to 100 measurements.

The switching probability, for a given amplitude, as a function of the number of pulses is fit with the mathematical model:

$\begin{equation} P_\mathrm{ON}\left(N\right) = 1 - \left(1 - P_\mathrm{ON}\left(1\right)\right)^N\quad \mathrm{where} \quad P_\mathrm{ON}\left(1\right) = P_\mathrm{ON}\left(V\right) \end{equation} \tag{ 2 }$

In the WM store and recall experiments, unless otherwise noted, the following conditions apply: no read voltage between spikes during the store and the recall phases. A read voltage of −150 mV is applied at the end of the experiment. The current compliance is I $_\mathrm{C}$ = 17 µ A, which gives an average retention time of 28 ms.

To perform the computer simulations we developed a simplified phenomenological model that qualitatively reproduces the the behavior of the memristive devices. The model captures the switching probabilities and variable retention time of the devices. Each synapse i was modeled with a binary internal state variable x_i that denotes either the low ($x_i = 1$) or the high resistance state ($x_i = 0)$. The device resistance was r₁ and r₀ in the low and high resistance state, respectively. To model switching probabilities we assigned a parameter $\rho_i \in [0,1]$ to every synapse. Upon arrival of a pre-synaptic input spike, x_i was set to 1 with probability ρ_i .

To model the trial-by-trail variability of the retention times we adopted a Lognormal distribution for the retention times $t_\mathrm{ret}$, namely:

$\begin{equation} \begin{split} t_\mathrm{ret} \; &\sim\; \mathrm{Lognormal}\,\left( t_\mathrm{ret} \;|\; \mu_i, \sigma_i \right) = \; \frac{1}{\sqrt{2}\pi\sigma_i \, t_\mathrm{ret}} \, \exp \left( -\frac{\left(\log\left(t_\mathrm{ret}\right) - \mu_i\right)^2}{2\sigma_i^2} \right). \end{split} \end{equation} \tag{ 3 }$

The parameters µ_i and σ_i in equation (3) were adjusted to fit the device properties. Supplementary figure S14 shows example model fits to experimental data for different compliance currents. In the simulations we used µ = 7.24 and $\sigma_i = 0.82$, if not stated otherwise, which corresponds to a mean retention time of ${\sim}1.5$ s. Whenever a synapse was set to the low resistance state ($x_i = 1$) the retention time $t_\mathrm{ret}$ was drawn from equation (3). After the simulation time t exceeded $t+t_\mathrm{ret}$ the synapse spontaneously returned to the high-resistance state.

To reproduce the WM model introduced in [5], we used a recurrent network of 8000 excitatory and 2000 inhibitory neurons. Connection probabilities between these neurons were as in [5]. Supplementary figure S13 illustrates the detailed network structure, connection probabilities and baseline synaptic conductances. Each memory item was represented by a population of 800 excitatory neurons, which were randomly chosen from the recurrent network for each of the five memory items. Synaptic weights between these neurons were five times stronger (0.5 mV) than between other neurons (0.1 mV). All inhibitory synapses were static (no STP) and had a strength of −0.2 mV. To store and recall the WM, 1000 input neurons were set up for each memory item. Input neurons were only connected to one of the memory item populations in the network. Input neurons fired at low baseline firing rates of 0.1 Hz. To store a memory item, firing rates of input neurons corresponding to one memory item were elevated ten-fold. During recall all input neurons were set to unspecific elevated activation.

All neurons used the leaky integrate and fire (LIF) model with biologically plausible parameters [29]. The LIF neuron is a spiking point neuron model that transiently integrates synaptic inputs using a leaky membrane potential. The membrane potential u(t) follows the dynamics

$\begin{equation} \frac{\mathrm{d}u}{\mathrm{d}t} = -\frac{1}{\tau_m}\left(u\left(t\right)-u_0\right) +i\left(t\right), \end{equation} \tag{ 4 }$

where u₀ is the membrane resting potential $\tau_\mathrm{m}$ is a time constant. i(t) is the input into the neuron denoting the summed effect of afferent synapses. If the membrane potential crosses a firing threshold ϑ at time t^f a spike is emitted and the membrane potential is reset immediately after

$\begin{equation} u\left(t^f+\Delta t\right) = u_r\;. \end{equation} \tag{ 5 }$

After a spike the neuron is inactive for a brief refractory time. The firing threshold was set to 20 mV and refractory time to 2 ms. Membrane time constants $\tau_\mathrm{m}$ were 15 ms and 10 ms, resting potentials u₀ 16 mV and 13 mV, for excitatory and inhibitory neurons, respectively. Independent unit variance Gaussian noise with mean 0.5775 mV were injected into each excitatory, and with mean 0.5275 mV into inhibitory neurons. As in [5] the mean of the Gaussian noise was precisely tuned to enable multistability in the network. Example network dynamics are shown in figure 4(D), for a store–recall experiment over 8 s.

For the associative WM model in figure 5 we developed a network architecture that was capable to store bidirectional short-term associations between items of different categories. The associative memory model consisted of eight memory neurons that were connected through bidirectional associative connections as illustrated in figure 5(A). The memory neurons encoded for the three memory categories, shape, texture and color. Each memory neuron exclusively received input from one input neuron to trigger store and recall events. Memory neurons were implemented using the LIF neuron model, with firing threshold of 1.4 V and membrane time constant of 15 ms. Each group of neurons corresponding to one memory category was augmented with a single inhibitory neuron that provided lateral feedback to enforce mutually exclusive activation of one memory item at a time. Inhibitory neurons were LIF neurons with threshold of 1.4 V and membrane time constant of 10 ms. All excitatory connections had a strength of 1.5 V and inhibitory feedback was −1.5 V.

Model memristive devices were used inside the bidirectional connections to store the association between a specific pair of memory neurons (which we call here the specific neurons). Exactly one model device was used per association. Memristive devices were augmented with two auxiliary threshold gates to route the input and outputs during store and recall cycles. Gates were put in series with the devices, one before (input gate) and one after (output gate). Input gates received excitatory input from the two specific memory neurons the association corresponded to. In addition input gates received inhibitory input from all other memory neurons. Output gates recursively connected back to specific memory neurons through excitatory connections. Storage of memory items was not dependent on the activity of the neurons and threshold gates (see figure 5), but retained solely in the hidden state of the memristive devices. Model parameters of memristive devices were fit to experimental data (µ = 7.24, σ = 0.82, corresponding to ca. 1.4 s mean retention time). Switching probabilities ρ were 0.2. The synaptic conductance for the low resistance state was chosen to elicit a voltage pulse of 1.0 V in the post-synaptic neuron. The high resistance r₀ was set to be 20 times higher than the low resistance r₁.

Simulations of the WM model with volatile memristive devices were done in python (3.8) using a custom implementation of the memristive synapse model that was developed for the NEST 2.14 simulation environment [30]. The simulation time step was 1 ms. Neuron dynamics were simulated using the current-based LIF model is available in NEST. A custom model of the STP model outlined in section 2.4 was implemented based on existing synapse models. Data analysis used the numpy and matplotlib python packages in version 1.23.2 and 3.5.3, respectively. The code is available at the following link: https://gitlab.com/kappeld/stp-tunable-synaptic-working-memory.

The volatile synapse used in this work is a resistive switching C/HfO$\mathrm{_2}$/Ag memristive device (see section 2 for details). After the device fabrication, the device is in its high-resistive state (HRS) and can be switched to the low resistive state (LRS) by applying a quasi-static voltage sweep between 0 V and 1.5 V and back. Contrary to many filamentary devices, the proposed device does not require an electroforming operation (figure S1(a)), thus simplifying the electrical operation within the neuromorphic circuit. The maximum current flowing through the device (i.e. current compliance, $I\mathrm{_{CC}}$) is limited by applying a voltage to the gate of an NMOS transistor whose drain is connected to the bottom electrode of the memristive device (figure 2(A), inset). The switching to the LRS occurs because, as the voltage exceeds a given threshold voltage $V\mathrm{_{T}}$, Ag ions migrate across the oxide layer and form the CF, thus bringing the device in an LRS (set operation, figure 2(A) [27, 31, 32]. However, the filament self-sustains only in presence of a voltage higher than a critical voltage referred to as hold voltage $V_\mathrm{H}$. Below this value, the re-diffusion of the Ag atoms in the dielectric layer causes the self-disruption of the filament and, as a consequence, the device transition to an HRS (spontaneous recovery, figure 2(A). These operations are fairly reproducible (figure S1(b)). Due to the different interface between the oxide and the silver and the carbon, the switching is enabled only from the silver to the carbon, while no switching is reported from the carbon (see figure S2), since carbon is atomically stable and no other physical mechanisms are involved. The two main features that enable the use of the proposed memristive device as a volatile synapse in WM tasks are tunable volatility in biologically relevant time-scales and controllable switching probability.

The retention time is shown in figure 2(B) and the median retention time increases exponentially with the current compliance, as illustrated in figure 2(C). The device can be tuned to be in LRS for a time ranging from ms to seconds depending on $I\mathrm{_{CC}}$, which is the time-scale of interest for our application. The intrinsic stochasticity of the filament formation at a microscopic level originates high variability in the distributions of the retention times [33].

The controllability of the switching probability ($P_\mathrm{ON}$) of the device can instead be achieved by either changing the width or the amplitude of the voltage pulse applied to the device, as shown in figure 2(D). It is worth noticing that the device operates with voltages $\lt$3 V, which are compatible with standard CMOS technology and therefore suitable for integration with neuromorphic circuits. Figure 2(E) shows the stochastic behavior of the device. Due to its stochastic nature, the threshold voltage of the device is slightly variable, which results in an electrically tunable switching probability: the higher the pulse voltage, the higher the switching probability [25]. Therefore, the desired switching probability of the device can be tuned by setting a suitable voltage amplitude. Moreover, when stimulated by a burst of identical pulses (figure S4), the switching probability of the device increases with the number of pulses, as demonstrated in figures 2(F) and S5. This effect can therefore be exploited to accelerate the storing or training phase in a network. The device proposed for the experiments also shows good stability in terms of cycle to cycle variability, requiring an initial stabilization, or warm up, of the properties (more details are provided in figure S3).

3.2.1. Store and recall of features

The volatile behavior of the memristive device is used in a small-scale experiment of WM to store and recall features, as depicted in figure 3(A). In our example, our network consists of five volatile synapses all connected to the same neuron (schematic in figure 3(B). The retention time of the devices is set through the current compliance, which is obtained by applying a voltage to the gate of the transistors. The current flowing to the neuron is the sum of the currents flowing through the stimulated devices. The neuron is trained to recognize a color within a color stream. Each color is encoded by stimulating a different combination of three devices as in figure 3(C). As a result, for each color only three devices out of five are switched on. At first, the target color is stored in the network by repeatedly stimulating the network with the combination of pulses that encodes the desired color. As a consequence, the corresponding stimulated devices will be switched on and set to their LRS. (More details about the impact of the different experimental parameters are provided in figure S6). Afterward the network is fed with a stream of random stimuli, as shown in figure 3(D) (see also figures S7–S9 for more experimental traces). When the stored item is presented to the network, all the stimulated devices are in LRS and therefore the current received by the neuron overcomes a threshold, thus triggering the firing of the neuron. The current threshold is set according to the current distributions shown in figure S10. After about 1 s without stimulation, all the devices switch off and the pattern is forgotten, as shown in figure S7. The correlation plot in figure 3(E) shows the difference between the expected and the measured current when a pattern is presented. The main contribution to the current is given by the synapses that are stimulated and in LRS, therefore three different current levels can be expected. It is shown that we can reach more than 90% accuracy, defined as the percentage of correct classifications (stored/non-stored pattern) during a test sequence.

Zoom In Zoom Out Reset image size

The retention time and the switching probability of each synaptic device are electrically tunable through the gate voltage of the transistor (which sets the current compliance) and the voltage amplitude of the pulse applied to the TE of the device, respectively. These features, together with the stimulation frequency of the network, determine the accuracy of the network in recognizing the color. As shown in figure 3(F), for low $P\mathrm{_{ON}}$ ($P\mathrm{_{ON}}$  =  5%), an increase of the spike rate leads to an improvement of the accuracy because the time between two pulses is shorter than the retention time of the device. Instead, higher values of $P\mathrm{_{ON}}$ lead the network to experience a decrease in its accuracy because the probability that a non-active device is switched on is higher. Finally, the intrinsic volatility of the network results in a progressive forgetting of the stored element, which is needed in the WM to prevent the saturation of the network and, consequently, the inability to learn new experiences or recognize the ones already learned. Figure 3(G) gives an estimation on how long the network remembers by showing the average mismatch current, defined as the difference between the expected current and the measured current. The two main errors that can occur over time, i.e. the network either fails to recognize the correct color or recognizes the wrong one, depend on the combination of stimulation frequency and $P\mathrm{_{ON}}$. Indeed, for high $P\mathrm{_{ON}}$ and stimulation frequency, devices that were supposed to be in their HRS turn to their LRS, thus increasing the measured current. The opposite combination, i.e. low $P\mathrm{_{ON}}$ and stimulation frequency, results in the switching off of devices previously in LRS, thus decreasing the measured current. The results suggest that indeed a careful selection of the $P\mathrm{_{ON}}$ based on the planned stimulation frequency is advisable to fine tune the WM accuracy for a given task, as it will be discussed further in section 4.

3.2.2. Biologically inspired model of synaptic WM

The experimental results on WM based on volatile memristive devices can be used to develop a WM model for simulation of a larger scale, RNN. In [5] a model of WM in the brain is introduced that is based on STP combined with multi-stable network dynamics. We adapted this network model to implement the core dynamics of our volatile memristive synapses (figures 2 and 3) as a model of STP. Figure 4(A) illustrates our model that serves as WM model that is able to store a set of discrete patterns. For this, the network receives input from five input populations that represent the input patterns to be stored. A set of input neurons (illustrated by colored circles) are connected to the recurrent WM network (cloud) so that the memory items could be stored and reactivated. All neurons are spiking neurons, such that outputs are given by unitary events that are emitted when an internal membrane potential variable crossed the firing threshold (see section 2.5). The network has a multi-stable dynamics in that neurons of a specific population excite each other and transiently strengthen synapses within the population through STP when activated. Inhibitory neurons are added to facilitate the network multi-stability (see section 2 for details). The experimental paradigm, the network architecture, and the synapse parameters are adjusted to the characteristics of our memristive devices. Retention time parameters are fit to the measured device characteristics.

Zoom In Zoom Out Reset image size

Figure 4(B) shows typical behavior of the model over a simulation of several seconds. Spiking activity of the network and input neurons are shown. The pattern C is stored by strongly activating the corresponding input neurons. This triggers co-activation of the pool of WM neurons that encode patterns C. Through this activation, volatile memristive synapses get strengthened and cause a prominent response in WM neurons when a recall stimulus that activates all input neurons at intermediate rates is given after a timeout of 1 s. During this recall phase neurons that encode pattern C show significantly increased activity. Two repeated recalls are shown and lead to reliable memory performance. In the timeout phase WM neurons are almost perfectly silent, which is perfectly in line with the behavior of biological neurons. After the second recall phase the memory item C is being forgotten and A can be stored without interference by strongly activating pattern A input neurons. In a third recall period, A but not C neurons get strongly activated.

The WM model requires time constants in the order of behaviorally relevant time scales ($\gt$500 ms to few seconds) to function well. Figure 4(C) shows the memory recall performance of the network when different retention times are used. Recall performance is measured here as the signal-to-noise ratio (SNR) between the WM neurons. The SNR is computed here as the mean population firing rate of the neurons specific to the memory item over the mean firing rate of non-specific neurons. Population firing rates were estimated over the total duration of the recall phase (figure 4(B) illustrates firing rate estimators). Retention times below 1 s lead to degraded performance because the firing activity of the neurons is significantly lower than the retention time. This result is in accordance with the experiments shown in figure 3. Too low retention times result in too fast forgetting such that memory items cannot be reliably recalled. Also switching probabilities had to be finely tuned. Figure 4(D) shows the impact of switching probabilities on the recall performance. Switching probabilities of about 0.05 are found to work best for this memory store/recall protocol because it prevents an excessive activation of the volatile devices, which would prevent the correct storage of information, as also confirmed by the experimental results of figure 3. Too low switching probabilities prevent memory formation, whereas too high switching probabilities may result in unstable circuit dynamics. Thanks to the flexibility of our volatile memristive devices, these probabilities can be obtained with a suitable selection of the voltage amplitude and pulse time width as shown in figure 2(D).

3.2.3. Associative symbolic WM

Another important feature of WM is the ability to transiently form associations between properties, such as color or shape, to remember representations of real-world object. Figure 5(A) illustrates this associative symbolic WM model. We investigated whether the model memristive devices can be used to form transient associations between symbolic items of different categories. We used a similar store–recall paradigm as in figure 4 but here the memory item to store is given by association between features that could be dynamically bound together. These features are thought to represent the state of physical objects an associative memory networks is able to perceive through a set of sensors. Concretely we used features from three different stimulus categories, shape, texture and color (see figure 5). Objects are represented by jointly activating feature neurons in the memory network, e.g. a 'smooth, red cylinder' is represented by jointly activating the corresponding feature neurons in the network. STP model synapses as in figure 4 are used as short-term memory inside the bidirectional connections between the neurons.

Zoom In Zoom Out Reset image size

The network is able to form an autoassociative memory. After an association has been formed by presenting an object to the network, the features of that object can be recalled, after a brief time delay of length T_delay, by querying the network with only one part of the features. Figure 4(B) shows one example store and recall sequence. After the object 'smooth, red cylinder' is stored, neurons representing its features can be re-activated by triggering only the 'cylinder' or 'smooth' neurons. After a new object ('smooth, blue cone') is stored, its representation can be retrieved by only activating the 'cone' neuron. The object representations are transiently stored for and then slowly fade away. This is analyzed in figure 4(C) where we plot the decoding error as a function T_delay. The memory can be reliably retrieved during a time window of 600 ms, corresponding in our devices to $I\mathrm{_{CC}}$ = 70 µ A.