Quantifying stimulus-relevant representational drift using cross-modality contrastive learning

Abstract

Previous works investigating representational drift from sensory to central nervous systems converged to show that neural coding, especially at the population level, readily overcomes these session-to-session fluctuations. However, representational drift in the primary visual cortex is more prominent when presenting naturalistic stimuli than artificial stimuli. Animals continuously navigate natural environments during the evolutionary timescale. Why did evolution not get rid of representational drift if it was just an inconvenience? Here, we investigate how representational drift simultaneously influences the encoding of multiple behaviorally relevant features in a natural movie stimulus. Although previous work already used this natural movie stimulus, the prohibitive challenge of parameterizing a natural movie forced them to simplify their analysis by only using averaged neural activity and stimulus binned into a single coarse time scale. Because natural environments contain multiple interacting spatio-temporal features, previous works only provided incomplete understanding of representational drift because of such simplification. Here, we use cross modality contrastive learning, a cutting edge machine learning method, to circumvent this parameterization challenge. This method enables us to learn an embedding of neural activity that retains only those relevant components of the natural movie stimulus without making explicit assumptions. We also observe that our learned embedding is near-optimal in decoding a whole suite of natural features (scene, optic flow, complex spatio-temporal features, and time) and generalizable to decode those features from single-trial or novel hold-out data. Using this embedding as a surrogate model, we observe that representational drift perturbs the local geometry of the embedding, and this results in various changes in performance when we decode from a different session (90 min later) even at the population level. For example, the rate of decoding change is three times faster for optic flow features than scene features. Our work further suggests that a separate compensation mechanism may be necessary for the optic flow features, as their autocorrelation scale is shorter than the minimum time needed to discriminate scene texture features. Thus, representational drift may encourage neural processing flexibility rather than be a mere nuisance.

1 Introduction

As recording technology has improved, the neuroscience community has been able to record larger neural populations over longer periods of time. These experiments have revealed that the correlations of neural activity with external variables change between separate recording sessions, over a timescale of hours to days to weeks, in a wide range of brain areas (Marks and Goard (2021); Driscoll et al. (2017); Schoonover et al. (2021); Ziv et al. (2013); Lütcke et al. (2013); Mankin et al. (2015)). Despite the prevalence of this representational drift, previous works have also shown that behavioral variables can be stably decoded at the population level in many different brain areas (Gallego et al. (2020); Funamizu et al. (2023); Levy et al. (2021); Ziv et al. (2013); Dhawale et al. (2017); Tolhurst et al. (1983); Huber et al. (2012); Rokni et al. (2007); Arieli et al. (1996); Faisal et al. (2008); Engel and Steinmetz (2019); Clopath et al. (2017); Lütcke et al. (2013); Schölvinck et al. (2015); Cohen and Maunsell (2010); Montijn et al. (2015); Rubin et al. (2015); Sheintuch et al. (2020); Driscoll et al. (2017); Rule et al. (2020); Chambers and Rumpel (2017)). Similarly, analyses of neural activity in the primary visual cortex (V1) have attempted to establish that stimulus may also be stably decoded in the presence of week-to-week representational drift (Marks and Goard (2021); Sadeh and Clopath (2022)). However, representational drift is also more prominent in the processing of natural stimuli than the artificial stimuli. Animals continuously navigate natural environments during the evolutionary timescale. If representational drift is merely an inconvenience, Why did evolution not get rid of it?

To answer the above question, it is necessary to answer a first question: what is the effect of representational drift on the encoding of behaviorally relevant features from naturalistic stimuli in V1? Naturalistic stimuli contain multiple spatio-temporal features. These features may drive distinct behaviors. For example, detecting a transient, expansive optic flow would elicit survival behaviors like arrest or escape Zhang et al. (2014); Liang et al. (2015) whereas discriminating between different textures is crucial for foraging Glickfeld et al. (2013); Poort et al. (2015); Resulaj et al. (2018). These features fluctuate at drastically different timescales. For example, a commonly used natural movie stimulus, the "touch of evil", has its optic flow features fluctuating at 33ms whereas its scene texture features fluctuate at 500-1000 ms. The complexity of parameterizing these features prevents previous works from directly decoding them from neural activity. Instead, these works simplified their analysis of the influence of representational drift by using averaged neural activity and stimulus binned into a single, coarse (1 second long) temporal window. This kind of binning gets rid of a lot of features that are important for behavior in naturalistic stimuli (for example, 1.6-ms transient stimuli can easily cause escape or arrest behavior Liang et al. (2015)). Therefore, these analyses provided an incomplete picture of representational drift. How representational drift changes multiple behaviorally relevant features is still unexplored.

In this work, we use a new machine learning method to extract stimulus-relevant components from neural activity and circumventing the above paramterization challenge. The method that allows us to learn an embedding that only retaining stimulus-relevant components from neural activity is the cross-modality, weakly supervised contrastive learning. This method is guided by "co-occurrence", meaning it extracts the shared features between neural activity across animals and the natural movie stimulus within the same temporal window (we use the 33 ms, single frame window here). Recent theory in Bayesian risk minimization showed that embeddings learned with this method is near-optimal is near-optimal in retaining the shared features between different modalities (i.e., neural activty and natural movie in our context). Namely, the learned embedding retains the components about the natural movie only from the neural activity and removes individual bias (e.g., behavioral variability between animals) that are not present in the natural movie stimulus. Using the neuropixel recordings of V1 from the Allen Brain Observatory, we demonstrate that our learned embedding is near-optimal for decoding multiple natural features (scene, motion, time, and the combination of scene and motion features, at 99% accuracy). We also show that this learned embedding can decode natural features when we project neural activity from single-trial and novel pseudomouses into the trained model. Therefore, the learned embedding extracts stimulus relevant components available in the neural code while compressing away irrelevant variability from individual behaviors or mental states.

Using this learned embedding as a surrogate model, we quantify the representational drift across the two sessions that are 90 min apart in this dataset using the learned embedding. Using the linear decoders trained on the embedding of session 1 neural activity to read out stimulus features from the embedding of session 2 neural activity, we observe a nearly 50% decrease in decoding performance for all features. We also find that the decoding decay of features fluctuating between fast and slow timescales is different: the decay ratio for fast features is three times greater than that for slow features. We further elucidate the geometry of the learned embedding and how it changes from session 1 to session 2, revealing a potential cause for the drop in decoding features fluctuating at tens of milliseconds.

2 Result

We use the publicly available recording of mouse primary visual cortex responding to a natural movie within the Allen Brain Observatory. This recording contains two sessions of neural responses during passive viewing of a natural movie that are 90 minutes apart. Previous work reported the presence of representational drift, even with such a short 90-minute interval. Although the visual coding dataset in the Allen Brain Observatory includes neurons from both V1 and higher order visual areas de Vries et al. (2019); brain map.org (2019), we focus on V1 in this study. V1 is the visual area that has been sampled the most thoroughly within this dataset. In addition, V1 not only sends behaviorally relevant features to higher visual areas, it also connects directly to subcortical structures that are responsible for driving survival behaviors Zhao et al. (2014); Liang et al. (2015). As a result, we hypothesize that V1 encodes the behaviorally relevant features that are used at both higher order areas of the cortex and subcortical structures.

2.1 The natural movie stimulus contains features with multiple spatiotemporal scales

Different from artificial stimuli, naturalistic stimuli consist of features that fluctuate across multiple spatiotemporal distributions. Previous works showed that similar multi-scale structures may be present for both scene textures (Saremi and Sejnowski (2013)) and dynamic motion (Salisbury and Palmer (2023)) in natural visual environments. These features give natural environments their complex structures within which organisms must navigate and maximize their chances of survival. Understanding how the neural population encodes stimulus features within the natural movie requires us to first understand the variety of features and their spatiotemporal scales in the natural movie stimulus.

Parametrizing a natural movie is computationally prohibitive. Here we dissect the natural movie into two streams: one corresponds to the static, textural components, and the other corresponds to the dynamic (optic flow) components (Figure. 1A, see methods for details). Using independent unsupervised clustering on static and dynamic components, we group frames with similar static or dynamic features together into separate clusters. Therefore, the clustering labeling gives us an estimate of how many discrete static and dynamic features are present in the natural movie stimulus. As a result, we decode these cluster labels as a proxy for decoding static or dynamic features in subsequent analyses. We also decode joint features, whose clustering is a result of combined clustering by aggregating the scene and flow clustering together.

When we compare the autocorrelation time scales between scene and flow labels, we find that the flow labels fluctuate one order of magnitude faster than those scene labels. In particular, the autocorrelation time scale of the flow labels is as short as 33–100 ms (1-3 frames in the 30 Hz movie), while the scene labels fluctuate at either 500 ms or 1000 ms. This emergence of fast and slow autocorrelation time scales may result from the changes in optic flow (e.g., moving left or right) in the natural movie being much faster than the switch between different scenes. Therefore, our work aims to understand how representational drift changes the neural coding of these features and whether it influences features fluctuating at fast time scales differently from features fluctuating at slower time scales.

Because the autocorrelation time scale of flow labels is only a few frames, consecutive frames in the movie may contain different flow labels. Hence, differentiating neighboring frames that are merely 33 ms apart is necessary to trace changes in stimulus features in the movie. Indeed, when we cluster the natural movie into a broad window clustering by first binning movie frames into a 1 second window (analogous to previous works (Xia et al. (2021); Sadeh and Clopath (2022)), we observe that the resulting clusters do not trace the change of stimulus features. For example, even if similar optic flow features occur at different time windows, they are still clustered into distinct clusters. As a result, whether a neural activity belongs to a broad binned temporal window may not reflect which stimulus features are encoded. Instead, we develop new tools to decode neural activity at 33ms, i.e., a single frame, in the subsequent analysis. Decoding features at single frame resolution allows us to observe and compare the impact of representational drift on stimulus features whose autocorrelation scales span a few tens to a few hundred milliseconds.

Figure 1: A) Three example (static) frames from the movie (top), with their corresponding optic flow frames (bottom). We perform hierarchical clustering on the static and optic flow frames to obtain discrete "scene" and "flow" labels (separate clusterings on scene and optic flow frames; see main text and Supplementary for details). All three example static frames here have the same scene label, but their optic flow frames each have a different flow label. Thus the optic flow provides more information for discriminating these three frames from each other than the textures in the static frames. B) These two behaviorally-relevant features of interest in the natural movie stimulus vary over different timescales: the autocorrelation decays differently for scene and flow labels (schematized at the top, with actual data below). The optic flow labeling has a fast change, i.e., it decays to near zero after 33-100 ms (1-3 frames). In comparison, the decay for scene labeling is much slower. It only reaches zero after 400 500 ms and reaches a negative peak around 1s (likely corresponding to scene change in the movie). This figure shows results for the first 400 frames (1st half of the movie), and we observe the same difference in decay timescales for the second half of the movie (see Supplementary Information).

2.2 Learning a generalizable representation from weakly supervised contrastive training

Because of the complex structure present in the natural movie, we aim at learning an embedding of features of the natural movie selectively encoded in V1 without making explicit assumptions. This encourages us to use weakly supervised contrastive learning to extract stimulus-relevant components from neural activity and the natural movie. The weak supervision is guided by “co-occurrence” (Radford2021), i.e., whether samples of neural activity or samples of a natural movie happen within the same temporal window. Thus, it extracts shared features across animals and maps them to features of natural movie frames within the same temporal window. We use 33 ms, i.e., a single frame, as the temporal window because flow features only autocorrelate over a couple of frames. This means that given a sample of neural activity $a$ from pseudomouse A and a sample $b$ from pseudomouse B, along with the natural movie features $m$ that happen at frame $t$, contrastive learning tries to bring the tuple $(a,b,m)$ together.

To obtain broad enough V1 populations that cover the entire visual field, we generate two pseudomouses by pooling neural activity from five different mice together; each pseudomouse consists of data from a non-overlapping set of five mice. We then use two training phases to obtain a neural representation that is relevant to the stimulus (see Figure 2). In the first phase, we use contrastive learning to extract features encoded by both $a$ and $b$ into the embedding, while removing the bias from individual pseudomouses. Because the embedding retains features shared by $a$ and $b$, the resulting embedding pulls $a$ and $b$ together. In machine learning, this is self-supervised learning Chen et al. (2020). Meanwhile, we also trained a separate backbone to obtain an efficient representation of a movie by contrasting its scene and optic flow features. In other words, the single-modality phase learns both efficient representations of neural activity and the natural movie separately. In the second phase, we align the representations of neural activity with the natural movie. To do this, we contrast samples from all possible pairs of modalities (neural activity, scene, and optic flow frames). Similar to the single-modality phase, this cross-modality contrastive learning learns a compressed representation that only keeps the shared features between the natural movie and the neural activity.

Because we construct contrastive pairs based on whether they co-occur within a single frame, time at the single frame level, a.k.a. frame number, is a natural choice of decoding feature (i.e., the mapping between neural activity and the changes in the natural scene). In Figure 3, we show that we can decode time in a held-out session 1 test set with nearly 99% accuracy using both the single-modality and cross-modality trained models. We get almost perfect results on our held-out session 1 test set. This shows that the neural activity responding to different frames in the movie is sufficiently different from one another for our contrastive learning backbone to be able to learn an embedding where each sample can be easily (linearly) differentiated from the others. Demonstrating the near-optimality of our learned embedding (for linearly decoding on time) is a necessary step to ensure that we can use it to quantify the effect of drift based on the change in decoding performance, as we do in the next section.

Additionally, the learned embedding generalizes to decode time using other ”out of distribution” formats of the session 1 neural activity. Our single-trial decoding accuracy reached around 93% for known pseudomouses (i.e., trained using the PSTH) and 92% for a novel pseudomouse (trained without PSTH or single-trial data). Therefore, our encoding models trained with PSTH capture the majority of the stimulus-relevant components of neural activity. Notably, we compare this performance to simple methods (PCA, NMF, see Supplementary) based on the neural population activity directly. The difference between our decoding performance and these off-the-shelf methods shows that our learned embedding uniquely reformats stimulus-relevant features to facilitate linear decoding among them.

Figure 2: Our two-phase contrastive training procedure. We created two pseudomice (“mouse 1” and “mouse 2”) by aggregating the V1 units from 5-6 animals, to ensure coverage of the visual field stimulated by the movie. We focus on the 2 sessions of the natural movie with 30 trials in each session. The natural movie used here is the first 30 seconds of “Touch of Evil." We use the InfoNCE loss proposed in van den Oord et al. (2018) in both single modality and cross-modality training. This InfoNCE loss uses contrastive pairs between neural activity of different mice, different features in the natural movie, or neural activity and the natural movie based on weak supervision guided by co-occurrence (see main text for details). We also find that the cross modality training aligns the single modality representations from both neural and the natural movie. As a result, the learned representations of the two modalities share decision boundaries for decoding stimulus features (e.g., the $t_{1}$ shown in green and $t_{2}$ shown in blue) while matching their supports (see supplementary for details).

Figure 3: A) We evaluate the learned representation via whether we can train a linear decoder $WX+b$ to differentiate the learned features that belong to $t_{i}$ and $t_{j}$.B) The linear decoder learns multiple hyperplanes (up to $n$, where $n$ is the dimensionality of the representation space). When the neural representation corresponding to different times form distinct clusters in the high-dimensional representation space, these hyperplanes jointly partition the space into $n$ partitions ($n$ is the total number of frames) (top schematic). Each distinct time is contained within a unique partition and is thus perfectly linearly decodable. By contrast, when the learned representation for different times contains overlap, these hyperplanes cannot find a unique subspace for each distinct time, and there will be decoding errors (bottom schematic). C) The decoding performance of both phases of training (single- or cross-modality) using neural population PSTHs (on the hold-out test sets) OR their corresponding single trial neural activity. These decoding performances are approaching 99% (PSTH) or 92-93% (single-trial) for decoding time.

2.3 Quantify the magnitude of representational drift via changes in decoding performance

Figure 4: A) Quantification of representational drift as the decrease in decoding performance across sessions. After we pretrain the model with neural activity from session 1 (training set) and freeze the weights, we obtain the neural representation without drift by mapping the held-out neural activity of session 1 (test set) into the pretrained model and the neural representation with drift by mapping the neural activity of session 2 into the pretrained model. We then train linear decoders on four natural features with the neural representation of session 1 test set, and show the decoding performance of session 1 (the dashed line; all decoding performances are close to 99%) and session 2 (the bar plots, indicating significant drift). B) The representational drift changes decoding of features fluctuating at fast/slow time scals differently. We recalculate the decoding accuracy as accuracy within some window of error tolerance. Using a specific temporal window of $n$ frames, we treat the predicted label for frame $t$ as the correct label if it matches any true labels within such a $n$-frame window, i.e., $(t-n,t+n)$ (as opposed to only regarding the predicted label as correct if it matches the true label of frame $t$). Errors beyond the $\tau_{3}$ autocorrelation scale can be easily corrected via retraining the linear classifier (see supplementary).

Because our learned embedding is nearly optimal for decoding a whole suite of natural features, we use it as a surrogate model. We then use the corresponding change in decoding performance between different sessions to quantify how much representational drift changes the encoding of stimulus features. This decoding is different from previous works on decoding. Previous works either only decoded averaged neural activity in a broad window of 1 second, and such decoding cannot track the change of stimulus features that fluctuate at faster time scales, or they only reported pairwise discriminability between different binned windows, which is a much easier decoding task than what we pose here. When we decode on session 2 with a linear decoder trained on session 1, we observe significant accuracy decay, i.e., up to 50%, in decoding all four natural features (time, scene, flow, and joint features combining scene and flow; see Fig 4). The results are symmetric across the two sessions: training on session 2 data and decoding session 1 data yields a quantitatively similar drop in decoding performance (see supplementary Table 2).

We also investigate whether the change in decoding flow features (or features fluctuating at fast time scales) is different from the change in decoding scene features (or features fluctuating at slow time scales). We look into how much improvement in decoding performance one may obtain from forgiving decoding errors within a short temporal window. We refer to this particular decoding performance as "error-tolerating accuracy”. For any given sample of time $t$, this error tolerating accuracy within an $n$-frame temporal window will regard the predicted time $t^{\prime}$ as correct if $t^{\prime}\in(t-n,t+n)$, as opposed to regarding $t^{\prime}$ as correct only if $t^{\prime}=t$. We expand the error tolerance window up to 30 frames (corresponding to the longest autocorrelation scales) to understand if any specific decoding errors occur within a certain autocorrelation timescale (shown in Figure 5B). We find that this error tolerance accuracy increases by 12.1% for flow features within its autocorrelation scale $\tau_{1}$ (and overall 27.2% towards the 1 second window). This increase in accuracy is also present in two other features that contain components of flow features, i.e., joint features and time. Meanwhile, the change in decoding scene features is smaller in magnitude as we expand the error tolerance window to 1 second. Comparing the difference in decoding accuracy between two autocorrelation scales for the scene, i.e., $\tau_{2}$ and $\tau_{3}$, we observe a change of 10%. Therefore, the representational drift changes the encoding of flow features much more than scene features. In the next section, we further analyze how representational drift changes the geometry of learned embedding to gain insights on why it leads to such a substantial change in the decoding of flow features.

2.4 The representational drift perturbs the local geometry helpful for decoding fast features

Figure 5: A) The global geometry of the learned embedding: Principal components (PC) with or without drift show fast decays. These decays are faster than the threshold $-1-2/d$ ($d$ as the stimulus dimension), suggesting that both embedding are smooth manifolds. Two neural encodings of close stimulus features are nearby in the embedding space because of this smoothness. B) Large proportions of samples follow $K$-nearest neighbor in local $K$ neighborhood (i.e., $k\in(0,4)$). The presence of representational drift perturbs this local geometry. The cyan line plot shows the proportion of the test set that has a local $k$-nearest neighborhood for $k\in(0,4)$. For a given sample belonging to time $t$, it has a $k$-nearest neighborhood if its nearest $k$-th neighbor in the learned embedding is a cluster within $(t-k,t+k)$. When $k=0$, this plot shows that 99% of the samples can be correctly classified by having a linear decoder with weight for class $K$ as the mean of all samples belonging to class $K$ (see supplementary). When $k=1$, this plot shows that for 96% of the samples, their closest neighbor cluster (after the cluster they were assigned to/decoded as) is also one of those that are temporally closest, i.e., for a sample of time $t$, its closest neighbor cluster is either $t+1$ or $t-1$. The magenta line plot shows the corresponding $k$-nearest neighborhood in the presence of representational drift for $k\in(0,4)$. Dashed line shows the chance level performance. The gap between this dashed line and the cyan/magenta lines echos A) that the gloabl geometry of the learned embedding is smooth with and without drift. The inset shows that there is no significant difference if we expand the neighborhood $k$ beyond $k=4$. C)Illustration of the change in the local geometry (see supplementary).

In this section, we analyze whether the geometry of the learned embedding changes in the presence of representational drift. We focus on two geometric features that are relevant to decoding stimulus features. The first geometry is the ”coding smoothness” for similar stimulus features, as first characterized in (Stringer et al. (2019a)). This smoothness geometry suggests that the neural representation of similar stimulus features is located nearby in the representation space (Stringer et al. (2019a)). The scale of the $n$-th principal component variance determines whether such a smoothness exists. Specifically, if the variance of principle components decays faster than the ”smoothness threshold”, i.e., $(-1-2/d)$ for a stimulus with d dimensions, then the representation is smooth. Stringer et al. (2019a) showed that any smooth representation is also a differentiable manifold. This allows two similar stimuli features to be found close to each other on the manifold (see Stringer et al. (2019a) for more information). In addition, Stringer et al. (2019a) also showed that neural activity generally obeys such a smoothness condition in encoding static stimuli like natural images. Here, we investigate whether our learned embedding inherits this coding smoothness from the neural activity for encoding spatio-temporal features from the natural movie. As we show in Fig. 5A (we use $d=400$ to calculate the smoothness threshold), the variance explained by the PC dimensions of our learned embedding decays much faster than the corresponding threshold, indicating that the learned embedding for session 1 is globally smooth. Meanwhile, representational drift does not change such a global smoothness, suggesting that global geometry does not play a role in the drop in decoding performance from session 1 to session 2.

Next, we investigate whether representational drift changes a local geometry between the embedding of similar stimulus features. Because our learned embedding is near-optimal for decoding (see Section 2.2), it exhibits another geometry (Papyan et al. (2020)). This geometry arranges the samples corresponding to the same stimulus feature to form a distinct cluster separable from other stimulus features (see supplementary for theoretical analysis). Within such a geometry, recent theory (Papyan et al. (2020)) suggests a specific optimal linear decoder whose weight for time $t$ is the mean activation of all samples belonging to $t$. In our case, this mean activation linear decoder achieves 99% accuracy on decoding time (see Section 2.2 and supplementary). Because of the global smoothness, this mean activation clustering suggests that the learned embedding also has a local $K$-nearest neighborhood within which the $k$-th closest neighbor encodes the $k$-th similar stimulus feature. For example, the 2nd nearest cluster mean for 96% of the test samples are also the clusters that are temporally adjacent (i.e., either $t-1$ or $t+1$ for a sample belonging to time $t$, shown as the top portion of the cyan line in Figure. 5B). Such a local neighborhood pertains to 75% of samples for $t\pm 2$ and quickly drops when it is beyond 2 frames (shown as the magenta line in Figure. 5B). This local neighborhood is much weaker in the presence of representational drift (magenta line) within the range $k\in(1,4)$ (shown as the cyan line in Figure. 5B). First, only 61.5% of the samples can be linearly decoded by the respective mean activation of all samples belonging to $t$ in the drifted activations. Second, only 41.9% and 30.3% of the test samples can be read out as the 2nd or 3rd nearest neighbor. Note that this pertubation from the representational drift is prominent in local neighborhood, but not as obvious in global structure. The inset of Figure 5B shows that if we expand $K$ into 10-30 frames, the neighborhood structures become similar with and without the representational drift.

We further illustrate why such a difference may occur (Figure 5C and supplementary). Without the representational drift, samples belonging to time $t$ and nearby time (e.g., $t\pm 1$ or $t\pm 2$) form well separated clusters with regular shapes. With representational drift, the shapes of these clusters become irregular. The overlap of these irregularly-shaped clusters results in errors for differentiating time $t$ with its immediate neighbors (see supplementary for theoretical analysis). Notably, the width $K$ corresponds to $30-60$ ms. Within this neighborhood, it is likely that flow labels would change (because of the fast autocorrelation scale of optic flow). The representational would make it challenging to use the optimal linear decoder to read out these different flow labels because of the confusion between these nearby clusters. Because the scene labels fluctuate at much longer time scales, different times within this local temporal window may share the same scene label; thus, representational drift does not change the linear decoding of scene labels as significantly as flow labels (shown in Section 2.3). Therefore, the perturbation of this local neighborhood is a possible cause of decoding decay in flow features.

3 Discussion

This study shows that representational drift brings various changes to multiple natural features simultaneously present in a natural movie stimulus. Observing that there are features that fluctuate from tens (fast, a single frame) to hundreds (slow, 30 frames) of milliseconds, we first use weakly supervised contrastive learning to extract all available stimulus-relevant components from neural activity. Because this embedding is near-optimal for linearly decoding a whole suite of natural features, we are able to directly quantify the magnitude of that drift in decoding. This is unique because previous works only investigated averaged neural activity on a single time scale without considering the intricate structure of the natural movie stimulus. We observe a 50% decrease in decoding performance on all natural features, suggesting that there is a significant change in the encoding of these features with representational drift. In addition, the magnitude of decoding decay for the fast fluctuating flow features is three times faster than the slow fluctuating scene features, suggesting that representational drift changes the encoding of fast and slow features differently. We then analyze the learned embedding geometry to show that the substantial decoding decay of the fast features corresponds to representational drift perturbing the local geometry, which may support near-optimal linear decoding.

The fast and slow features in the natural movie stimulus may support different visually guided behaviors, considering their distinct autocorrelation time scales. The fast features fluctuate as fast as 33 ms (or possibly faster, but 33 ms is the limit from the sampling rate of the neural activity), which corresponds to transient changes in the visual field. Previous experiments demonstrated that V1 can modulate the superior colliculus (SC) in the presence of such transient stimuli (even as brief as 1.6 ms, Liang et al. (2015)) to induce temporal arrest. Meanwhile, the same circuit was hypothesized to be also responsible for detecting looming threats Zhang et al. (2014), suggesting that the V1 encoding of these fast features is informative to drive survival critical behaviors. The slow features that fluctuate at 500-100 ms are scene textures. Such slow time scales allow temporal integration from V1 to discriminate target textural features from distractions Resulaj et al. (2018); Goris et al. (2018). They may play important roles in foraging when animals navigate complex natural environments.

Our research encourages future research to explore the potential impact of representational drift in the primary visual cortex (V1) on visually guided behaviors. There are some thought-provoking inquiries. A crucial initial inquiry is whether there is a requirement for compensation mechanisms for representational drift. This is contingent upon whether the accuracy of V1 in encoding naturalistic stimuli is necessary for stable behavioral outputs. Prior studies have demonstrated that V1 encodes orientation selectivity with a precision of 0.1 degrees upon drifting grating stimuli. However, downstream systems that directly drive behaviors can only use 5 degrees of orientation selectivity. This represents a 50-time coarsening. Representational drift significantly changes the encoding of naturalistic stimuli more than artificial stimuli, with a 50% difference in our case. Therefore, it is currently unknown whether behavioral output will remain stable when using an encoding that is 100 times coarser (or two orders of magnitude coarser) than V1. This is an intriguing and unexplored future question. Our findings suggest that if compensating methods are required, then the encoding for optic flow features is distinct from that of scene texture features. Because V1 requires a minimum of 80 ms to exhibit discriminability (Resulaj et al. (2018)), and considering that optic flow characteristics change at a rate of 33 ms or even shorter, it is possible that mechanisms designed to compensate for the encoding of texture data may not be fast enough to handle optic flow features. Thus, if the superior colliculus uses those rapid characteristics to stimulate appropriate actions, it must develop a compensation mechanism distinct from mechanisms known to improve discriminability, like pooling (Stringer et al. (2019b)).

This approach has the unique advantage of extracting stimulus-relevant components from neural activity without making explicit assumptions. Alternatively, we show that the learned embedding exhibits Bayesian optimality. This optimality ensures that the resulting embedding retains all stimulus features commonly encoded across multiple animals. This optimality also makes this method suitable for learning neural representation for natural behaviors. In neural recording experiments of sensory systems, the experience is usually consistent across multiple trials. The variance we observe at the single trial level is mostly noise in neural representation. This is not the case with natural behaviors. Natural behaviors themselves exhibit significant variability in trial-by-trial execution. As a result, it is unclear whether the variability observed in neural representation for natural behaviors comes from noise in the neural code or a difference in behavioral variables. Moreover, it is unknown whether all features exhibited in natural behavior execution (based on video) are encoded by the motor relevant brain areas (i.e., the motor cortex) or can be contributed to stereotypical motor tape within local control circuits. We can adapt this method to discover behaviorally relevant features represented by the neural population. For example, we can first use single modality training to obtain an efficient representation of natural behaviors based on video or posture recording. We can then obtain another efficient representation through cross-modality training between video and neural representation. Bayesian optimality suggests that the efficient representation after cross-modality training retains all behavioral features selectively encoded by the neural population. We can determine which features the motor cortex selectively encodes by comparing this representation with the representation from single-modality training.

This Bayesian optimality also distinguishes our approach from previous methods that focused on learning compressed latent embeddings Schneider et al. (2023); Azabou et al. (2021); Zhou and Wei (2020) to reconstruct neural populations. These encoder-decoder based methods obtain latent embedding through training both an encoder and a decoder. Because of the extra training for the decoder, those embeddings are not Bayesian optimal. Furthermore, the dimensionality of the latent embedding is manually controlled as a parameter. Although the learned embedding may include stimulus features encoded by the neuronal population, There is no guarantee that all of these features will be retained in the learned embedding. Therefore, these previous methods could not be used to discover novel features about external variables.

The finding that representational drift brings different changes in the encoding of features that fluctuate at different time scales implies that representational drift may serve a useful purpose rather than being only a nuisance in neural processing. Features that fluctuate at different time scales enable various behaviors. Organisms may develop a specific compensation mechanism to compensate for the unique effect of representational drift on a single stimulus feature rather than relying on a general mechanism that applies to all features. As previously discussed, the range of time scales across these features suggests that the compensating mechanisms may also differ. Consequently, if one specific compensating mechanism fails, it would only impact one behavior rather than all behaviors that are essential for survival. Hence, our observation corroborates the notion that the brain carries out flexible, robust, and efficient computations as organisms explore and interact with the external world.