Brant-2: Foundation Model for Brain Signals

Abstract.

Foundational models benefit from pre-training on large amounts of unlabeled data and enable strong performance in a wide variety of applications with a small amount of labeled data. Such models can be particularly effective in analyzing brain signals, as this field encompasses numerous application scenarios, and it is costly to perform large-scale annotation. In this work, we present the largest foundation model in brain signals, Brant-2. Compared to Brant, a foundation model designed for intracranial neural signals, Brant-2 not only exhibits robustness towards data variations and modeling scales but also can be applied to a broader range of brain neural data. By experimenting on an extensive range of tasks, we demonstrate that Brant-2 is adaptive to various application scenarios in brain signals. Further analyses reveal the scalability of the Brant-2, validate each component’s effectiveness, and showcase our model’s ability to maintain performance in scenarios with scarce labels. The source code and pre-trained weights are available at: https://github.com/yzz673/Brant-2.

1. Introduction

Brain signals refer to the biometric information collected from the brain (Zhang et al., 2021). Their patterns provide valuable insights towards understanding the physiological function of the brain and the mechanism of related diseases, leading to various applications like neurological disorders (Alturki et al., 2020; Chen et al., 2022), sleep health research (Supratak and Guo, 2020; Phan et al., 2021, 2023), emotion recognition (Song et al., 2023, 2020) and so on. Brain signals are usually measured by invasive methods like stereoelectroencephalography (SEEG) or non-invasive methods like scalp electroencephalography (EEG). SEEG requires extra surgeries to implant the recording devices, resulting in a high cost (Kovács et al., 2021). However, it manifests advantages by providing stereotactic and detailed information about deep brain structures. As a non-invasive method, EEG fails to capture deep brain information accurately and contains more noise due to the placement of electrodes on the scalp. However, compared to SEEG, EEG is more accessible to implement without surgery, leading to more application scenarios. Despite the differences, both SEEG and EEG data use the same principle of electrical activity recording (Caune et al., 2012) and share the same physiological basis.

The field of brain signals encompasses a wide array of downstream tasks. It represents a cutting-edge domain that will continue to unveil new research directions and application scenarios in the future (Zhang et al., 2021). In addition, after collecting the brain signals, the annotation work highly relies on experts in the corresponding field, making large-scale data labeling infeasible (Diachenko et al., 2022). However, existing works in this field are mainly designed to solve specific tasks (Chen et al., 2022; Yuan et al., 2023; Zheng et al., 2023). Moreover, many of them (Jia et al., 2023; Lopes et al., 2023) require training models from scratch, which tends to have a high dependency on labels. Very limited research provides an off-the-shelf model, known as the foundation model, that can be applied to multiple scenarios in this field and serve as a tool for further investigation of the brain. Foundation models have shown great potential in language (Touvron et al., 2023a, b; Yang et al., 2023) and vision (Wang et al., 2023; Yuan et al., 2021), which not only allow for customization for diverse applications but also reduce costs for data annotation (Bommasani et al., 2022). Furthermore, by leveraging foundation models, researchers and developers no longer need to build models from scratch, which saves time and costs, benefiting the advancement of the field. Therefore, we aim to build a foundation model for brain signals that can effectively solve numerous downstream tasks for both SEEG and EEG data. However, when building a foundation model for brain signals, it is inevitable to encounter challenging issues along the way.

Firstly, different data exhibit differences in terms of sampling rates as well as the positions and quantities of electrodes. SEEG data is often recorded at a high sampling rate of at least 1000Hz (Gavvala et al., 2022) and exhibits significant inter-individual variability in electrode numbers and locations. EEG data is usually sampled at a lower frequency than SEEG (Dasgupta et al., 2022) and can vary significantly in terms of montage (the number and the places of electrodes placed on the scalp) (Yi et al., 2023). Secondly, brain signals collected from different scenarios contain distinct physiological characteristics, leading to varying modeling scales. For example, a sleep stage in sleep studies is often defined as lasting up to 30 seconds (Supratak and Guo, 2020; Phan et al., 2021, 2023; Jia et al., 2023), seizure detection may utilize a time scale of less than 10 seconds (Chen et al., 2022; Yuan et al., 2023), and existing works for emotion recognition adopt modeling scales of 5 seconds or shorter (Yi et al., 2023; Song et al., 2023). Thirdly, there is substantial diversity among different tasks in the field of brain signals. For example, in seizure detection (identifying whether a segment includes seizure waves), the model is required to extract information from the target signal, such as capturing spikes and sharp waves within the signal. On the other hand, in seizure prediction (predicting whether there will be future epileptic seizures), the model needs to anticipate future changes of the target signal.

Figure 1. Overview of our work. We initially utilized approximately 4 TB of brain neural data from over 15k subjects to construct our pre-training corpus. Subsequently, we employ the corpus to train Brant-2 using two pre-training tasks. Then the pre-trained model can be fine-tuned and applied to various application scenarios of brain signals.

As Fig. 1 shows, to build such a foundation model, the first step is to gather a large amount of unlabeled brain neural data, which is then used for large-scale pre-training, overcoming the challenges above. For applications, as an off-the-shelf model, the pre-trained model can be applied to various downstream scenarios through fine-tuning. In the field of brain signals, Zhang et al. (2023) propose a foundation model for SEEG, Brant, which can capture long-term dependency, spatial correlation, and time-frequency information from SEEG signals. However, Brant has some limitations that prevent it from addressing the above challenges. Therefore, we propose Brant-2, a foundation model for brain signals, which excels in three main aspects. Firstly, the pre-training corpus of Brant-2 is large and diverse. As shown in Fig. 1(a), Brant-2 utilizes nearly 4 TB of mixed SEEG and EEG data with more than 15k subjects. Due to the large volume and diversity of data, Brant-2 contains over 1 billion parameters. Secondly, Brant-2 is robust to data variations and different modeling scales. Brant is pre-trained on multi-channel data with a fixed sampling rate and window length, by which it struggles to handle data variations and adapt to changes in modeling scales. During the pre-training process of Brant-2, we design a data augmentation module to further expand the diversity of the pre-training corpus, which enhances the robustness of Brant-2 towards data variations and modeling scales. Thirdly, Brant-2 can be applied to a broad range of tasks and scenarios. As shown in Fig. 1(b), compared to Brant, which is only pre-trained with mask-prediction, Brant-2 learns more comprehensive semantic knowledge through two pre-training tasks, leading to better generalization abilities to a wider set of downstream tasks (shown in Fig. 1(c)). In summary, our key contributions comprise:

• We propose a foundation model Brant-2, the first off-the-shelf model that can be applied to scenarios of both SEEG and EEG. Brant-2 is the largest model in brain signals pre-trained with nearly 4 TB brain signal data from more than 15k subjects.

• The pre-training framework we designed not only enhances the robustness of the model to significant data variations and different modeling scales but also empowers the ability to adapt to diverse downstream tasks in brain signals.

• We evaluate Brant-2 on a wide range of downstream tasks to illustrate the generalization ability of our model. By conducting additional analysis experiments, we demonstrate our model’s scalability, confirming each component’s efficacy and highlighting its ability to sustain performance in scenarios with limited labels.

2. Method

As aforementioned, building a foundation model for brain signals primarily requires handling data variations, different modeling scales, and diverse tasks. To tackle the variations of the data and modeling scales, we employ data augmentation during pre-training to enhance the diversity of the training data, improving our model’s robustness. To learn complex semantic representations and adapt to diverse downstream tasks, Brant-2 integrates time and frequency information and simultaneously focuses on reconstructing the input and forecasting future signals based on partial observations.

Notations. We use $\bm{s}_{i}\in\mathbb{R}^{C\times(B+F)},\ i\in\{1,2...,N\}$ with $C$ channel(s) and $B+F$ time steps to represent a segment of SEEG or EEG signals obtained from the pre-training corpus, where $N$ denotes the total sample number. The sample $\bm{s}_{i}$ is divided into two consecutive parts: the look back window $\bm{x}_{i}\in\mathbb{R}^{C\times B}$ and the future values $\bm{x}_{i}^{\text{fut}}\in\mathbb{R}^{C\times F}$, where the look back window $\bm{x}_{i}$ serves as the input.

Figure 2. The architecture and pre-training framework of Brant-2. The input raw signal $\bm{x}_{i}$ is first processed to subseries-level patches $\bm{p}_{i}$. Then we conduct data augmentation to increase the diversity of the training data and mask a subset of patches. We combine the information from both time and frequency domains to obtain the input embedding $\bm{h}_{i},\bm{\hat{h}}_{i}$, which are then fed into the temporal and spatial encoder sequentially. The output representations $\bm{z}_{i},\bm{\hat{z}}_{i}$ are linearly mapped to reconstruct the masked patches and forecast future signals.

2.1. Overall Architecture

The overall architecture of Brant-2 is shown in Fig. 2, which mainly involves four modules: 1) patching; 2) data augmentation and masking module; 3) input embedding module; 4) encoder.

Patching. Aggregating time steps into subseries-level patches can not only enhance the locality and capture comprehensive semantic information, but also reduce computation cost (Nie et al., 2022; Zhang et al., 2023). Thus, we divide the input sample $\bm{x}_{i}$ into non-overlapped patches with length $P$ and generate a set of patches $\bm{p}_{i}\in\mathbb{R}^{C\times L\times P}$, where $L=\lfloor B/P\rfloor$ is context length (i.e., the number of consecutive patches).

Data augmentation and masking. The quality of the data is of paramount importance for training a foundation model (Li et al., 2023). When assessing data quality, the diversity of the data is a crucial metric. High data diversity is beneficial for improving model performance, while low diversity can introduce biases and inaccuracies in the training process. In the field of language, LLMs (Large Language Models) are pre-trained using text corpus sourced from various domains (Touvron et al., 2023a, b; Yang et al., 2023). Furthermore, Lee et al. (2023) measure the diversity of publicly available LLM datasets and conclude that these datasets are highly diverse, which emphasizes the significance of data diversity for a foundation model.

In view of the significance of data diversity during pre-training, we introduce a data augmentation module to enhance the pre-training corpus in both the temporal and spatial dimensions, aiming to generate more diverse data. For the obtained patches $\bm{p}_{i}\in\mathbb{R}^{C\times L\times P}$, the temporal augmentation involves a random resampling to adjust the sampling rate of the input sample. The variation in sampling rates enriches the temporal scale of the samples, allowing the model to become more robust to handle changes in modeling scales. Formally, we choose an adjustment factor $m_{k}$ from $\mathcal{M}=\{m_{1},...,m_{K}\}$ uniformly at random, then the input patches and future values are resampled by a factor of $m_{k}$, which are denoted as $\hat{\bm{p}}_{i}\in\mathbb{R}^{C\times L\times P_{k}}$ and $\hat{\bm{x}}_{i}^{\text{fut}}\in\mathbb{R}^{C\times F_{k}}$, where $P_{k}=P\times m_{k}$, $F_{k}=F\times m_{k}$. For the spatial augmentation, our goal is to enable the model to handle various numbers of channels, including single-channel data. Specifically, we first select a channel number $C_{k^{\prime}}$ from $\mathcal{C}=\{C_{1},...,C_{K^{\prime}}\}$ ($C_{k^{\prime}}$ is set equal to $C$ when $C_{k^{\prime}}>C$), where $C_{1}=1$. Then, we shuffle the channel dimension of the resampled patches $\hat{\bm{p}}_{i}$ and the future values $\hat{\bm{x}}_{i}^{\text{fut}}$ according to the same rule and the data is reorganized into $n$ non-overlapping subset(s) along the channel dimension with $C_{k^{\prime}}$ channel(s), where $n=\lfloor C/C_{k^{\prime}}\rfloor$. In the masking procedure, we randomly mask a subset of $C_{k^{\prime}}\times L$ patches with a fixed masking ratio, where the values of the masked patches are replaced by zeros. We denote the outputs of data augmentation and masking as $\tilde{\bm{p}}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times P_{k}}$ and $\tilde{\bm{x}}_{i}^{\text{fut}}\in\mathbb{R}^{n\times C_{k^{\prime}}\times F_{k}}$.

Input embedding. For neural recordings, the time domain provides insights into the amplitude and duration of neural signals, while the frequency domain unveils oscillatory patterns and rhythmic activity (Kalaivani et al., 2014). By modeling neural signals in both domains, we can obtain a more comprehensive understanding of the underlying neurophysiological mechanisms (Morales and Bowers, 2022). Therefore, as shown in the top left corner of Fig. 2, we combine the features from both time and frequency domains to obtain the input embedding. We generate the frequency features $\tilde{\bm{p}}_{i}^{\text{F}}$ from the augmented data $\tilde{\bm{p}}_{i}$ by calculating the power spectral density (Youngworth et al., 2005) (PSD) that reveals the spectral power distribution in different frequency bands of the signals, which is associated with different brain functional states (Zhang et al., 2023). For example, during wakefulness, $\alpha$ (8-13 Hz) and $\beta$ (13-30 Hz) waves are more active; during sleep, $\delta$ (less than 4 Hz) and $\theta$ (4-8 Hz) waves are more prominent.

We use non-linear encoders to map the time and frequency data $\tilde{\bm{p}}_{i},\ \tilde{\bm{p}}_{i}^{\text{F}}$ to $\frac{D}{2}$-dimensional latent representations $\tilde{\bm{h}}_{i},\ \tilde{\bm{h}}_{i}^{\text{F}}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times\frac{D}{2}}$. which are then concatenated and added with a learnable positional encoding $\mathbf{W}_{\text{pos}}\in\mathbb{R}^{L\times D}$ which monitors the temporal order of patches to obtain the input embedding $\bm{h}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times D}$:

(1) $$\bm{h}_{i}=\text{Concat}(\tilde{\bm{h}}_{i},\tilde{\bm{h}}_{i}^{\text{F}})+\text{Broadcast}(\mathbf{W}_{\text{pos}}),$$

where the $\text{Broadcast}(\cdot)$ operator broadcasts $\mathbf{W}_{\text{pos}}$ to the same shape as $\text{Concat}(\tilde{\bm{h}}_{i},\tilde{\bm{h}}_{i}^{\text{F}})$. Finally, we make a clone of the input embedding $\bm{h}_{i}$ and obtain $\bm{\hat{h}}_{i}$, preparing for the subsequent encoding process.

Encoder. In order to generalize to various scenarios, we incorporate both mask-prediction and forecasting tasks during pre-training to learn representations with rich semantic information. For the purpose of simultaneously accomplishing these two pre-training tasks, we design a multi-feed-forward (multi-FFN) Transformer block, as illustrated in the top right corner of Fig. 2. The block contains two FFNs, where one is used for signal reconstruction, and the other is employed for forecasting. We utilize a temporal encoder to capture time dependencies and a spatial encoder to capture channel correlations, both of which are composed of stacked multi-FFN Transformer blocks.

For temporal encoding, we model series of patches of length $L$: $\bm{h}_{i,j}\in\bm{h}_{i},\ \bm{\hat{h}}_{i,j}\in\bm{\hat{h}}_{i},j=1,2,...,n\times C_{k^{\prime}}$, where $\bm{h}_{i,j},\ \bm{\hat{h}}_{i,j}\in\mathbb{R}^{L\times D}$. We denote the outputs of the $l-1$-th layer of the temporal encoder as $\bm{o}_{i,j}^{l-1},\ \bm{\hat{o}}_{i,j}^{l-1}\in\mathbb{R}^{L\times D}$, where $\bm{o}_{i,j}^{0}=\bm{h}_{i,j},\ \bm{\hat{o}}_{i,j}^{0}=\bm{\hat{h}}_{i,j}$. For the $l$’th layer, outputs from the last layer $\bm{o}_{i,j}^{l-1},\ \bm{\hat{o}}_{i,j}^{l-1}$ go through the same multi-head attention (Vaswani et al., 2023) followed by a residual addition and a normalization to obtain the attention outputs $\bm{a}_{i,j}^{l},\ \bm{\hat{a}}_{i,j}^{l}\in\mathbb{R}^{L\times D}$, which will be separately fed into two FFNs (denoted as $\text{FFN}_{\text{mask}}^{l}$ and $\text{FFN}_{\text{fcst}}^{l}$). Due to the incomplete data resulting from the mask operation during pre-training, we utilize the information reconstructed by $\text{FFN}_{\text{mask}}^{l}$ to assist forecasting. Therefore, we establish a residual connection without gradients between the two attention outputs $\bm{a}_{i,j}^{l}$ and $\bm{\hat{a}}_{i,j}^{l}$:

(2) $$\bm{f}_{i,j}^{l} =\text{FFN}_{\text{mask}}^{l}(\bm{a}_{i,j}^{l}),$$

(3) $$\bm{\hat{f}}_{i,j}^{l} =\text{FFN}_{\text{fcst}}^{l}(\bm{\hat{a}}_{i,j}^{l}+\text{Detach}(\bm{a}_{i,j}^{l}))),$$

where $\text{Detach}(\cdot)$ operator returns a clone of the original input without gradients. Followed by a residual addition and a normalization, we derive the outputs of the $l$-th layer $\bm{o}_{i,j}^{l},\ \bm{\hat{o}}_{i,j}^{l}\in\mathbb{R}^{L\times D}$. The whole outputs of the temporal encoder are denoted as $\bm{t}_{i},\ \bm{\hat{t}}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times D}$. For spatial encoding, we take the outputs of the temporal encoder and model sets of patches of length $C_{k^{\prime}}$ from different channels: $\bm{t}_{i,j}\in\bm{t}_{i},\ \bm{\hat{t}}_{i,j}\in\bm{\hat{t}}_{i},\ j=1,2,...,n\times L$, where $\bm{t}_{i,j},\ \bm{\hat{t}}_{i,j}\in\mathbb{R}^{C_{k^{\prime}}\times D}$. The encoding process of the spatial encoder is the same as the temporal encoder which is described above. The outputs of the spatial encoder $\bm{z}_{i},\ \bm{\hat{z}}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times D}$ are served as the latent representations of Brant-2.

2.2. Pre-training and Fine-tuning

Pre-training. We adopt mask-prediction and forecasting tasks during pre-training to fully extract rich semantic information to adapt to different downstream tasks. The mask-prediction allows the model to understand the patterns within a certain segment of the signal. On the other hand, the forecasting task enables the model to learn future trend changes from the current observed series. We utilize two linear heads $\mathbf{W}_{\text{rec}}\in\mathbb{R}^{D\times P_{k}},\mathbf{W}_{\text{fcst}}\in\mathbb{R}^{D\times F_{k}}$ to map the latent representations to the original signals. During pre-training, the model conducts a patch-level reconstruction and a series-level forecasting:

(4) $$\bm{p}_{i}^{\text{rec}} =\bm{z}_{i}\mathbf{W}_{\text{rec}},$$

(5) $$\bm{x}_{i}^{\text{fcst}} =\text{MeanPool}(\bm{\hat{z}}_{i})\mathbf{W}_{\text{fcst}},$$

where $\bm{p}_{i}^{\text{rec}}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times P_{k}},\bm{x}_{i}^{\text{fcst}}\in\mathbb{R}^{n\times C_{k^{\prime}}\times F_{k}}$, the $\text{MeanPool}(\cdot)$ operation aggregates each $L$ consecutive patches in $\bm{\hat{z}}_{i}$. Then following the masked modeling and forecasting paradigms, Brant-2 is supervised by two MSE losses in the pre-training stage:

(6) $$\mathcal{L}_{rec} =\sum_{i=1}^{N}\|\tilde{\bm{p}}_{i}-\bm{p}_{i}^{\text{rec}}\|_{2}^{2},$$

(7) $$\mathcal{L}_{fcst} =\sum_{i=1}^{N}\|\bm{x}_{i}^{\text{fut}}-\bm{x}_{i}^{\text{fcst}}\|_{2}^{2},$$

where $N$ is the number of training samples. The objective of joint optimization is obtained by adding the losses $\mathcal{L}_{rec}$ and $\mathcal{L}_{fcst}$.

Fine-tuning. When fine-tuning the model, we first use a mean pooling operation to gather each $L$ consecutive patches of the latent representations $\bm{z}_{i},\ \bm{\hat{z}}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times L\times D}$, then the representations are aggregated by a weighted sum:

(8) $$\bm{r}_{i}=\lambda\text{MeanPool}(\bm{z}_{i})+(1-\lambda)\text{MeanPool}(\bm{\hat{z}}_{i}),$$

where $\bm{r}_{i}\in\mathbb{R}^{n\times C_{k^{\prime}}\times D}$ and $\lambda$ is a learnable parameter. The aggregated representation $\bm{r}_{i}$ will be fed into a linear or non-linear head for the downstream tasks.

3. Experiments

3.1. Pre-training Setup

Pre-training datasets. The pre-training corpus of Brant-2 incorporates a mixed dataset of SEEG and EEG data from over 15k subjects with a total data size approaching 4 TB. The SEEG corpus used for pre-training contains intracranial neural data recorded from 26 subjects. The original corpus has a total size of 12 TB. After removing unused channels and applying preprocessing like denoising and filtering, we obtain 2.3 TB of SEEG data for pre-training. The surgical procedure involves the implantation of invasive electrodes with 47 to 238 channels. The corpus contains SEEG data at 1000Hz, 2000Hz, and 4000Hz sampling rates. The EEG corpus utilized in the pre-training is a publicly available dataset, TUEG(Harati et al., 2014), which comprises 1,643 GB of clinical recordings from 14,987 individuals with a total of 27,063 hours of data. The dataset contains over 40 different channel configurations, in which approximately 95% of the data includes a 10/20 configuration as a subset of the available channels. The sampling rate of the recordings varies between 250Hz and 1024Hz.

Pre-training details. In the encoder block of Brant-2, we apply RMSNorm(Zhang and Sennrich, 2019) and use the Swish activation function(Ramachandran et al., 2017). We set the context length $L$ of Brant-2 as 16 patches, the masking ratio as 40%, and the forecasting length as 1/4 of the context length (The hyperparameter analysis of the masking ratio and forecasting length is shown in App.B). The adjustment factor of the sampling rate is uniformly chosen from $\mathcal{M}=\{0.25,0.5,1.0,2.0\}$ and the reorganized channel number is chosen from $\mathcal{C}=\{1,2,4,8,16,32,64,128\}$. Brant-2 is trained using the AdamW optimizer(Loshchilov and Hutter, 2017), with $\beta_{1}=0.9$, $\beta_{2}=0.95$, $eps=10^{-5}$. For the learning rate scheduling, we use a linear warmup of 1k steps to reach a peak learning rate of $1.0\times 10^{-5}$, followed by a cosine decay of 150k steps to decay the final learning rate to $0$. During the pre-training process, the model parameters are updated for a total of 105k steps. Our models are trained on a Linux system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 4 GPUs (NVIDIA Tesla A100 80G). Brant-2 contains 1 billion parameters, which takes over 100 hours to pre-train.

3.2. Evaluation Setup

We conduct evaluation experiments on nine diverse SEEG and EEG datasets, encompassing five downstream tasks: seizure detection, seizure prediction, sleep stage classification, emotion recognition, and motor imagery classification. We divide each dataset into several non-overlapping groups and conduct n-fold cross-validation for all groups. Each time, we set one group for evaluation and the others for fine-tuning. The fine-tuning process mainly involves updating the last two layers of the temporal encoder and the classification head, while freezing the remaining parameters of the Brant-2 (more details are shown in App. D.2).

Seizure detection. Accurate seizure detection is crucial for diagnosing and treating individuals with epilepsy and other seizure-related disorders. Seizure detection aims to identify and classify instances of seizures in brain signals recorded from epilepsy patients, which is formalized as a binary classification to classify between physiological and pathological samples. We employ 2 SEEG and 2 EEG datasets to evaluate the model performance on seizure detection. The SEEG datasets, MAYO and FNUSA (Nejedly et al., 2020), contain 5000Hz SEEG recordings from 18 and 13 subjects, respectively. The subjects are divided into six groups for each dataset, and the data is segmented into 3-second data clips. We preserve the physiological and pathological activities and remove the artifacts and power line noise. CHB-MIT (Guttag, 2010; Shoeb, 2009) consists of 23-channel EEG recordings with a sampling rate of 256Hz from 22 subjects with intractable seizures. We divide the subjects into four groups and segment the signals into 8-second data clips. Siena (Detti et al., 2020; Detti, 2020) consists of 27-channel EEG recordings of 14 patients with a sampling rate of 512Hz. The subjects are divided into five groups, and the signals are segmented into 4-second data clips. We use precision, recall, F1 and F2 scores as evaluation metrics. In the scenario of epilepsy, F2 is more valued than F1 since ignoring any seizure is costly in diagnosis.

Seizure prediction. Different from seizure detection, seizure prediction is conducted under a more challenging setting where the task is to predict the likelihood of future seizures based on the current observations. Seizure prediction is crucial for providing early warnings and alerts for individuals with epilepsy. We utilize a clinical SEEG dataset from a first-class hospital labeled by professional neurosurgeons. The dataset contains 5 subjects with a sampling rate of 1000Hz, and we adopt a 5-fold cross-validation. We sample 16-second segments and predict whether a seizure occurs within the next 1 minute. We adopt the same evaluation metrics as in seizure detection (i.e., precision, recall, F1 and F2 score) to measure the performance of the models.

Sleep stage classification. In sleep health research, sleep staging plays a critical role in enhancing our understanding of sleep states and patterns, contributing to the prevention and diagnosis of sleep-related disorders (Phan and Mikkelsen, 2022). The American Academy of Sleep Medicine (AASM) manual defines sleep into five stages: wake, N1, N2, N3, and REM  (Berry et al., 2017). Thus, sleep stage classification is formalized as a 5-class classification. We choose 2 EEG datasets, SleepEDF (Kemp et al., 2000) and Haaglanden Medisch Centrum sleep staging database (Alvarez-Estevez and Rijsman, 2021, 2022)(referred to as HMC), to verify the model performance on sleep stage classification. For SleepEDF, we adopt the SleepEDF-78 dataset which contains 153 whole-night polysomnographic sleep recordings from 78 subjects during sleep cassette studies. The EEG data with a sampling rate of 100Hz contains 1 EEG channel, which is segmented into 30-second epochs. We randomly divide the subjects into five groups. HMC is a collection of 151 whole-night polysomnographic (PSG) sleep recordings from 151 subjects. The data is sampled at 256Hz and contains 4 EEG channels. The subjects are split into five groups, and the signals are segmented into 30-second epochs. As for the evaluation metrics, we utilize accuracy, sensitivity, specificity, macro F1 score, and Cohen’s kappa $\kappa$.

Emotion recognition. Emotion recognition using EEG is becoming an interesting topic among researchers, which has made advancements in various domains, including biomedical research, brain-computer interfaces (BCIs), etc (Kamble and Sengupta, 2023). The SEED dataset (Zheng and Lu, 2015; Duan et al., 2013) contains 62-channel EEG data from 15 subjects while watching film clips. The film clips are carefully selected to induce different types of emotion (positive, negative, and neutral). Thus, we conduct discrete emotion recognition formalized as a 3-class classification. The data is down-sampled to 200Hz, segmented into 5-second segments, and split into five groups. The evaluation metrics include accuracy and macro F1 score.

Motor imagery classification. Motor imagery classification is to classify brain activity patterns related to imagined movements, which has gained significant attention due to its potential applications in BCIs, rehabilitation therapies, and assistive technologies. We select EEG Motor Movement/Imagery (Schalk et al., 2004; Goldberger et al., 2000)(referred to as Motor Imagery) as the dataset for this task. Motor Imagery consists of over 1500 one- and two-minute 64-channel EEG recordings with a 160Hz sampling rate obtained from 109 volunteers. For each subject, a target appears on either the left or the right side of the screen, and the subject imagines opening and closing the corresponding fist until the target disappears. The model aims to differentiate whether the subject is imagining opening and closing their left fist or right fist based on the collected EEG signals. We split the subjects into five groups and clip the data into 6.4-second segments. The evaluation metrics include accuracy and F1 score.

3.3. Baselines

We extensively compare our model with 12 advanced methods, which are divided into three categories, including 1) 3 methods aimed at time series universal modeling, 2) 3 methods based on self-supervised pre-training on brain signals, and 3) 6 methods specifically designed for each task. The methods in the first two categories are evaluated on all downstream tasks, while the methods in the third category are only evaluated on the specific tasks.

To be precise, we choose TF-C (Zhang et al., 2022) SimMTM (Dong et al., 2023) and One Fits All (Zhou et al., 2023) as universal modeling methods for time series. For the pre-training works on brain signals, we choose BrainBERT (Wang et al., 2022), Brant (Zhang et al., 2023) and MBrain (Cai et al., 2023). In addition, we select PPi (Yuan et al., 2023) (seizure detection on SEEG data), ScatterFormer (Zheng et al., 2023) (seizure detection on EEG data), Lopes et al. (2023) (seizure prediction), SleepHGNN (Jia et al., 2023) (sleep stage classification), EEG Conformer (Song et al., 2023) (emotion recognition) and TSFF-Net (Miao and Zhao, 2023a) (motor imagery classification) as the task-specific methods for each downstream task. More details of the baselines are shown in App. C.

Figure 3. The overall performance comparison of our model and other baseline methods on all downstream datasets.

Table 1. Average performance on seizure detection.

	MAYO				FNUSA
	Pre.	Rec.	F1	F2	Pre.	Rec.	F1	F2
TF-C(Zhang et al., 2022)	55.12 $\pm$10.37	70.72 $\pm$14.49	55.84 $\pm$8.89	61.48 $\pm$9.43	69.80$\pm$4.91	81.46 $\pm$7.63	74.76 $\pm$6.30	78.50 $\pm$8.16
SimMTM(Dong et al., 2023)	51.03 $\pm$10.75	78.97 $\pm$9.31	56.66 $\pm$10.44	67.13 $\pm$10.04	69.33*$\pm$5.71	82.44 $\pm$12.05	74.74 $\pm$6.03	79.00 $\pm$9.39
One Fits All(Zhou et al., 2023)	59.60$\pm$11.21	77.62 $\pm$10.71	66.26*$\pm$3.23	72.22 $\pm$4.02	63.65 $\pm$11.03	82.18 $\pm$9.22	70.90 $\pm$6.00	76.95 $\pm$5.02
BrainBERT(Wang et al., 2022)	54.47 $\pm$11.25	89.70*$\pm$11.43	64.65 $\pm$7.54	76.02 $\pm$6.31	58.11 $\pm$8.60	87.62*$\pm$8.79	67.91 $\pm$6.46	77.67 $\pm$4.95
Brant (Zhang et al., 2023)	49.24 $\pm$12.56	93.30$\pm$7.84	63.38 $\pm$8.22	77.92*$\pm$2.36	48.15 $\pm$5.13	94.72$\pm$4.94	63.61 $\pm$3.72	79.07 $\pm$3.31
MBrain(Cai et al., 2023)	55.39 $\pm$10.46	80.08 $\pm$10.97	61.81 $\pm$4.55	70.32 $\pm$10.12	67.88 $\pm$5.43	85.89 $\pm$9.51	75.25*$\pm$6.78	81.10*$\pm$10.68
PPi(Yuan et al., 2023)	68.94$\pm$12.14	88.73 $\pm$8.49	73.77$\pm$6.24	80.74$\pm$4.63	72.77$\pm$5.06	86.60 $\pm$13.67	78.39$\pm$8.29	82.93$\pm$9.49
Brant-2	55.41*$\pm$9.74	95.88$\pm$3.86	69.90$\pm$8.45	83.30$\pm$6.07	68.78 $\pm$9.86	95.96$\pm$4.43	79.76$\pm$7.61	88.59$\pm$5.21

	CHB-MIT				Siena
	Pre.	Rec.	F1	F2	Pre.	Rec.	F1	F2
TF-C(Zhang et al., 2022)	17.82 $\pm$8.44	61.36 $\pm$10.66	18.01 $\pm$6.72	27.61 $\pm$6.52	7.98 $\pm$1.34	61.63 $\pm$10.92	13.68 $\pm$3.20	24.92 $\pm$3.78
SimMTM(Dong et al., 2023)	54.31 $\pm$8.12	36.77 $\pm$8.71	42.82 $\pm$8.95	38.84 $\pm$8.79	32.30 $\pm$10.51	43.37 $\pm$11.72	27.47 $\pm$6.10	32.97 $\pm$9.35
One Fits All(Zhou et al., 2023)	51.14 $\pm$12.20	56.58 $\pm$5.23	51.21 $\pm$9.95	53.69 $\pm$5.47	47.27 $\pm$10.26	43.68 $\pm$8.55	43.00 $\pm$5.42	42.97 $\pm$5.29
BrainBERT(Wang et al., 2022)	52.47 $\pm$8.04	59.44 $\pm$12.25	55.09 $\pm$8.95	57.41 $\pm$10.06	47.33*$\pm$9.81	60.61 $\pm$10.15	48.54*$\pm$6.20	53.52 $\pm$6.92
Brant (Zhang et al., 2023)	57.42$\pm$8.42	61.41*$\pm$10.07	57.99*$\pm$7.63	59.65*$\pm$8.09	43.07 $\pm$11.05	66.04*$\pm$6.81	48.36 $\pm$8.42	55.27*$\pm$4.82
MBrain(Cai et al., 2023)	53.39 $\pm$14.42	52.87 $\pm$8.08	49.35 $\pm$7.40	50.38 $\pm$6.18	38.94 $\pm$8.83	60.99 $\pm$5.61	45.88 $\pm$7.06	53.32 $\pm$6.62
ScatterFormer(Zheng et al., 2023)	55.73*$\pm$8.03	64.92$\pm$10.14	59.81$\pm$8.80	62.72*$\pm$9.52	50.33$\pm$11.38	67.49$\pm$7.21	53.49$\pm$7.04	58.98$\pm$4.15
Brant-2	56.44$\pm$7.53	73.04$\pm$7.83	63.42$\pm$7.48	68.78$\pm$7.65	50.35$\pm$8.52	70.42$\pm$4.53	58.14$\pm$4.77	64.70$\pm$1.92

Table 2. Average performance on sleep stage classification.

	SleepEDFx					HMC
	Acc.	Sens.	Spec.	Macro F1	Kappa	Acc.	Sens.	Spec.	Macro F1	Kappa
TF-C(Zhang et al., 2022)	65.96 $\pm$1.28	51.42 $\pm$2.37	90.26 $\pm$0.47	49.37 $\pm$1.92	50.73 $\pm$2.04	48.04 $\pm$4.89	35.30 $\pm$4.67	84.35 $\pm$1.31	30.15 $\pm$6.67	23.90 $\pm$7.08
SimMTM(Dong et al., 2023)	63.85 $\pm$2.30	36.29 $\pm$1.54	88.97 $\pm$0.68	31.37 $\pm$2.62	44.40 $\pm$3.54	44.64 $\pm$2.01	31.52 $\pm$2.48	83.18 $\pm$0.76	27.27 $\pm$3.51	17.76 $\pm$3.90
One Fits All(Zhou et al., 2023)	68.45 $\pm$1.95	56.32 $\pm$3.04	91.20 $\pm$0.71	54.77 $\pm$1.70	55.03 $\pm$2.75	58.64 $\pm$1.55	51.12 $\pm$2.85	88.42 $\pm$0.70	50.54 $\pm$2.97	43.52 $\pm$3.00
BrainBERT(Wang et al., 2022)	69.56 $\pm$1.85	59.40*$\pm$2.40	91.80 $\pm$0.55	58.66*$\pm$1.66	57.13 $\pm$2.51	60.69 $\pm$1.67	53.06 $\pm$2.13	89.04 $\pm$0.61	51.95*$\pm$2.01	46.48 $\pm$2.58
Brant (Zhang et al., 2023)	69.06 $\pm$2.69	58.25 $\pm$3.62	91.63 $\pm$0.83	56.84 $\pm$3.32	56.55 $\pm$3.77	51.02 $\pm$3.15	41.90 $\pm$2.20	85.89 $\pm$0.60	38.12 $\pm$3.74	30.42 $\pm$2.90
MBrain(Cai et al., 2023)	71.91*$\pm$0.98	58.35 $\pm$1.99	92.16*$\pm$0.38	58.22 $\pm$3.63	59.83*$\pm$2.10	62.33*$\pm$1.24	53.97*$\pm$1.87	89.50*$\pm$0.14	51.48 $\pm$2.94	48.65*$\pm$0.88
SleepHGNN(Jia et al., 2023)	77.56$\pm$2.06	70.38$\pm$2.57	94.18$\pm$0.60	69.79$\pm$3.98	69.72$\pm$3.79	64.87$\pm$2.34	57.27$\pm$2.48	90.24$\pm$0.56	56.93$\pm$3.50	52.21$\pm$3.56
Brant-2	77.15$\pm$1.39	67.01$\pm$2.51	93.90$\pm$0.43	67.20$\pm$2.42	68.05$\pm$2.22	68.76$\pm$2.41	63.74$\pm$1.74	91.52$\pm$0.43	63.87$\pm$1.94	58.20$\pm$2.65

Table 3. Average performance on seizure prediction.

	Clinical
	Pre.	Rec.	F1	F2
TF-C(Zhang et al., 2022)	34.92 $\pm$6.61	49.35$\pm$13.53	37.28 $\pm$9.50	42.88 $\pm$11.10
SimMTM(Dong et al., 2023)	60.39 $\pm$10.76	37.94 $\pm$6.72	45.37 $\pm$7.69	40.42 $\pm$6.99
One Fits All(Zhou et al., 2023)	55.84 $\pm$8.48	43.41 $\pm$10.39	48.04*$\pm$9.42	45.01*$\pm$10.02
BrainBERT(Wang et al., 2022)	61.89*$\pm$13.92	29.38 $\pm$12.47	39.05 $\pm$13.98	32.55 $\pm$13.05
Brant (Zhang et al., 2023)	55.39 $\pm$11.28	40.89 $\pm$9.97	40.01 $\pm$8.80	39.29 $\pm$9.29
MBrain(Cai et al., 2023)	54.75 $\pm$13.29	44.26 $\pm$11.86	41.79 $\pm$9.59	41.67 $\pm$9.82
Lopes et al. (2023)	62.82$\pm$9.17	46.86*$\pm$10.21	50.84$\pm$8.46	47.85$\pm$10.36
Brant-2	62.67$\pm$7.25	49.94$\pm$8.04	55.20$\pm$7.93	51.87$\pm$8.05

Table 4. Average performance on emotion recognition.

	SEED
	Acc.	Macro F1
TF-C(Zhang et al., 2022)	82.87 $\pm$5.21	82.13 $\pm$5.66
SimMTM(Dong et al., 2023)	81.69 $\pm$7.06	81.26 $\pm$7.20
One Fits All(Zhou et al., 2023)	87.80 $\pm$3.35	87.67 $\pm$3.38
BrainBERT(Wang et al., 2022)	85.98 $\pm$5.46	85.81 $\pm$5.98
Brant (Zhang et al., 2023)	89.50*$\pm$3.57	89.43*$\pm$3.71
MBrain(Cai et al., 2023)	84.60 $\pm$7.47	84.52 $\pm$7.45
EEG Conformer(Song et al., 2023)	93.17$\pm$4.20	93.10$\pm$4.22
Brant-2	93.47$\pm$3.09	93.42$\pm$3.08

Table 5. Average performance on motor imagery classification.

	Motor Imagery
	Acc.	F1
TF-C(Zhang et al., 2022)	60.06 $\pm$1.62	57.79 $\pm$3.00
SimMTM(Dong et al., 2023)	57.48 $\pm$1.82	57.37 $\pm$2.80
One Fits All(Zhou et al., 2023)	71.25 $\pm$3.50	72.56*$\pm$3.06
BrainBERT(Wang et al., 2022)	64.84 $\pm$4.19	70.32 $\pm$3.55
Brant (Zhang et al., 2023)	72.00*$\pm$1.93	71.84 $\pm$2.42
MBrain(Cai et al., 2023)	61.06 $\pm$2.09	60.42 $\pm$4.65
TSFF-Net(Miao and Zhao, 2023a)	73.00$\pm$4.32	73.87$\pm$2.10
Brant-2	74.33$\pm$3.61	74.30$\pm$3.83

3.4. Evaluation Results

Main Results. Fig. 3 summarizes the overall results of Brant-2 compared with the baseline methods on all the downstream tasks. The radar chart shows that Brant-2 outperforms all universal time series modeling methods and pre-training methods on brain signals, surpassing a majority of task-specific methods, indicating that our method exhibits strong generalization ability across various scenarios of brain signals. Detailed statistics and comparisons of each task will be discussed in the following paragraphs, where in all the tables, we mark values ranking the first (v), second (v), and third (v*) in each column.

Seizure Detection. Tab. 1 shows the results of seizure detection on SEEG and EEG datasets. In the results of MAYO and FNUSA, Brant-2 achieves the best recall and F2 score over other models, demonstrating our model’s ability in seizure detection on SEEG data. Regarding the F2 score, PPi secures the second position, which can be attributed to the fact that PPi contains a specifically designed pre-training framework for seizure detection and dedicated techniques to address inter-subject variability. From the results of CHB-MIT and Siena, we can observe that Brant-2 ranks the first in almost all performance metrics, showing a strong ability in EEG-based seizure detection.

Seizure Prediction. Tab. 3 shows the average performance of seizure prediction task on the clinical dataset. In Tab. 3, Brant-2 achieves the first place in terms of F1 and F2 scores, which are improved by 37.97% and 32.02% compared to Brant, indicating that the pre-training forecasting task enhances the predictive capability. Lopes et al. (2023) achieves the second-best F1 and F2 scores, demonstrating the effectiveness of combining original signals and handcrafted features. However, as a fully supervised method, Lopes et al. (2023) relies heavily on labels, which are often scarce in clinical settings. We will investigate the impact of scarce labels on model performance in Sec. 4.3. Apart from Brant-2 and Lopes et al. (2023), One Fits All achieves the highest F2 score, which could be attributed to its utilization of a pre-trained GPT-2 (Radford et al., 2019) as the backbone with strong predictive abilities.

Sleep Stage Classification. The results of sleep stage classification on SleepEDFx and HMC are shown in Tab. 2. Overall, Brant-2 and SleepHGNN exhibit comparable performance, with Brant-2 outperforming SleepHGNN on HMC and SleepHGNN having a slight edge on SleepEDFx. As a specialized model designed for sleep stage classification, SleepHGNN incorporates EEG signals and synchronously collected EOG, ECG, and EMG signals, thereby leveraging multiple modalities for improved performance. Brant-2 achieves comparable performance using only EEG signals, highlighting the effectiveness of our large-scale pre-training.

Emotion Recognition. Tab. 4 contains the results of emotion recognition on SEED dataset. Our model obtains the best results and EEG Conformer achieves the second place. Like Brant-2, EEG Conformer also considers the temporal dependency and spatial correlations by designing a convolution module with temporal and spatial convolutional layers.

Motor Imagery Classification. The performance of motor imagery classification is shown in Tab. 5. The achievement of the highest accuracy and F1 score by Brant-2 demonstrates the effectiveness of our model in motor imagery classification. The utilization of time-frequency spectrograms in TSFF-Net, along with its second-best accuracy and F1 score, highlights the significance of time-frequency domain information in this scenario.

4. Analysis

4.1. Scalability Analysis

Large language and vision models(Bai et al., 2023; Yang et al., 2023; Touvron et al., 2023a, b) have shown strong scalability behavior. As a large model in brain signals, we aim to investigate the scaling behavior of our model in terms of the pre-training loss and downstream task performance.

Setup. In addition to Brant-2, we pre-trained two smaller versions with 200 million, 460 million parameters, following the same training configurations described in Sec.3.1. Then we evaluate the models on all five downstream tasks, with each task utilizing one dataset (CHB-MIT for seizure detection, Clinical for seizure prediction, SleepEDFx for sleep stage classification, SEED for emotion recognition and Motor Imagery for motor imagery classification).

Figure 4. The results of scalability analysis.

Results. The training losses of the two pre-training objectives (the losses are calculated every 5k steps) are shown in Fig.4(a). One can observe that as training progresses: 1) the training losses of the models, regardless of their size, continue to decrease; 2) as we increase the model size, the losses decrease faster. These observations indicate that Brant-2 shows scalability behavior during pre-training. Furthermore, as shown in Fig.4(b), larger models attain better performance across all tasks, showcasing that our scalable overall performance transfers to a range of downstream tasks.

4.2. Ablation Study

We perform ablation experiments to assess the effectiveness of the architectural design of the model and the pre-training tasks.

Setup. We set three model variants to validate the effectiveness of our architectural design: 1) Brant-2 w/o temporal encoder: remove the temporal encoder; 2) Brant-2 w/o spatial encoder: remove the spatial encoder; 3) Brant-2 w/o multi-FFN: replace the multi-FFN Transformer encoder block of Brant-2 with the vanilla Transformer encoder block (Vaswani et al., 2023). For each model variant, we control the parameter count of the models to be approximately the same to ensure fair comparison. To illustrate the usefulness of both pre-training tasks, we perform two sets of experiments: 4) Brant-2 w/o mask: pre-train with forecasting; 5) Brant-2 w/o forecast: pre-train with mask-prediction. Brant-2 and the above five variants are evaluated on all the five downstream tasks, with each task utilizing the same dataset as the one used in the scalability analysis (i.e., CHB-MIT for seizure detection, Clinical for seizure prediction, SleepEDFx for sleep stage classification, SEED for emotion recognition and Motor Imagery for motor imagery classification). Since each model variant requires pre-training and such process for a 1-billion scale model alone takes over 100 hours as described in Sec. 3.1, all the experiments in the ablation study are based on Brant-2-460M.

Figure 5. The results of ablation study.

Results. The ablation results are shown in Fig.5, in which Brant-2 outperforms the other variants across all five downstream tasks, demonstrating the effectiveness of each component of our work. Brant-2 w/o temporal encoder exhibits overall poor performance in these downstream tasks, highlighting the crucial importance of temporal dependency for brain signals. In certain tasks (e.g., seizure detection, emotion recognition), Brant-2 w/o mask outperforms Brant-2 w/o forecast, indicating that these tasks require a better understanding of patterns within a signal segment. On the other hand, in some tasks (e.g., seizure prediction), Brant-2 w/o forecast performs better, demonstrating that these tasks prioritize predicting future changes based on the current observed series. Therefore, joint training of the two pre-training tasks enhances the adaptive ability to different downstream tasks.

4.3. Label Scarcity Scenario Exploration

The results in Sec.3.4 have demonstrated that Brant-2 can generalize well to various tasks. As a foundation model, we also aim to investigate whether our model can address the issue of over-reliance on labels and be applicable to scenarios with scarce labels.

Setup. We choose to conduct experiments on the Clinical dataset originating from real-world clinical scenario of epilepsy, where the annotation cost is high. By choosing this dataset, we intend to simulate real-world scenarios closely and address the challenges associated with expensive annotations in clinical settings. We compare our model with the best-performed baseline method Lopes et al. (2023), which is fully supervised. We conduct three sets of experiments on each model with 100%, 10%, and 1% of training data.

Figure 6. the performance changes of the model as the training labels decrease.

Results. The variation in model performance with decreasing training labels is shown in Fig.6. Overall, as the training labels decrease, the performance exhibits a certain degree of decline. When transitioning from 100% to 1% labels, Brant-2 and Brant-2-460M show F1 and F2 scores decreases of less than 10% and 15%, respectively, In contrast, the F1 and F2 scores of Lopes et al. (2023) decline 50.6% and 32.6%, respectively. The results indicate that Brant-2 can reduce reliance on labels, thereby ensuring performance in scenarios with scarce labels.

5. Conclusion

We propose a foundation model Brant-2, the first off-the-shelf model that can be applied to scenarios of both SEEG and EEG. Brant-2 is able to handle significant data variations and generate powerful representations of brain signals from a broad range of application scenarios. We experiment on five downstream tasks to illustrate the generalization ability of Brant-2. In addition, Brant-2 shows a scalability behavior in both pre-training and downstream tasks. Furthermore, we explore the change of model performance in low-resource labeled scenarios, in which the performance of Brant-2 remains much more stable than the supervised SOTA method designed for the scenario, indicating that our model alleviates the issue of label efficiency. The field of brain signals is continuously evolving, with emerging research directions and scenarios. In the future, we aim to train our model on a more diverse and extensive corpus, enabling its application in more research areas and scenarios.

Ethics Statement

The data collection and experiments conducted in our work on the private datasets (i.e., the pre-training SEEG corpus and the clinical dataset for seizure prediction) have been approved by the Institutional Review Board (IRB) and passed ethical review. All participants have signed informed consent forms. All publicly available datasets used in this paper are not associated with any privacy or security concerns. Furthermore, we have followed guidelines on responsible use specified by the primary authors of the datasets used in the current work.

Appendix A Related Work

Scenario-specific methods for brain signals. In view of the diverse application scenarios of brain signals, researchers have designed various methods specifically tailored for these contexts. Yuan et al. (2023) propose a self-supervised learning framework and two techniques for SEEG-based patient-independent seizure detection. Zheng et al. (2023) propose a model, ScatterFormer, for patient-independent seizure detection on EEG data, which is an invariant scattering transform-based hierarchical Transformer that specifically pays attention to subtle features. Lopes et al. (2023) conduct seizure prediction by encoding the original signals and hand-crafted features with a deep and shallow networks, respectively. Jia et al. (2023) propose a novel Sleep Heterogeneous Graph Neural Network (SleepHGNN) to capture the heterogeneity and the interactivity of physiological signals for sleep stage classification. Song et al. (2023) design a compact Convolutional Transformer, named EEG Conformer, to encapsulate local and global features for emotional recognition and motor imagery classification. Miao and Zhao (2023b) propose a shallow, lightweight decoding architecture (TSFF-img) based on time-frequency spectrograms for motor imagery classification.

Universal modeling for brain signals. Universal modeling exhibits great advantages by learning highly general representations to enable customization for various applications. Developing such techniques for brain signals with a broad range of applications is suitable. Wang et al. (2022) propose an off-the-shelf model, BrainBERT, that provides embeddings for intracranial recordings. Zhang et al. (2023) propose a foundation model Brant for SEEG modeling, which is the largest model for intracranial recordings. Both BrainBERT and Brant are limited to SEEG data with a relative narrow range of application scenarios. Cai et al. (2023) design a unified self-supervised learning framework for brain signals which can be utilized on either SEEG or EEG data. However, their work cannot model different kinds of brain signals simultaneously.

Appendix B hyperparameter Analysis

The pre-training performance of a foundation model is of utmost importance as it significantly impacts the performance on downstream tasks. Therefore, it is necessary to select the optimal masking ratio and forecasting length which are two crucial hyperparameters of Brant-2. However, due to limited computational resources, it is impractical to perform a grid search for these two parameters on a model with over 1 billion parameters (as the pre-training time for Brant-2 exceeds 100 hours). Therefore, we conduct experiments on two smaller-scale models (100M and 200M) to observe their performance on downstream tasks and determine the optimal hyperparameters, which are then applied to larger-scale models. For the searching strategy, since the two pre-training tasks are relatively independent, we adopted the following strategy to determine the two optimal hyperparameters instead of grid search: First, we solely focus on the mask-prediction task during pre-training and select the optimal masking ratio. Next, with the chosen masking ratio fixed, we determine the optimal forecasting length.

Setup. To comprehensively evaluate the model to determine the optimal hyperparameters, we pre-trained two models with 100M and 200M parameters and conducted experiments on all five downstream tasks. Each task is evaluated on the same dataset as the one used in the scalability analysis (i.e., CHB-MIT for seizure detection, Clinical for seizure prediction, SleepEDFx for sleep stage classification, SEED for emotion recognition and Motor Imagery for motor imagery classification) described in Sec. 4.1. We follow the same pre-training configurations described in Sec.3.1. For the masking ratio, we experiment with settings of 20%, 40%, 60%, and 80%. Regarding the forecasting length, we try lengths of 1/16 $L$, 1/4 $L$, 1/2 $L$, and $L$, where $L$ is the context length.

Figure 7. The hyperparameter analysis.

Results. Fig.7 illustrates the results of the hyperparameter analysis, in which the top five graphs represent the analysis for the masking ratio, and the bottom five graphs depict the analysis for the forecasting length. It can be observed that with a masking ratio of 40% and a forecasting length of 1/4 $L$, the models achieve the best overall performance among the five downstream tasks.

Since brain signals are natural signals with heavy redundancy, a missing patch can be recovered from neighboring patches with little high-level understanding of the semantic information. Thus, masking with a relative low ratio (20% or lower) or conducting very short-term forecasting (e.g. 1/16 $L$) may lead to ineffectiveness of high-level representation learning, which is crucial for time series classification(Wu et al., 2022). However, as non-stationary time series, the values and associations between variables in brain signals significantly change over time. Therefore, a excessively high masking ratio (e.g., 80%) poses great challenges for the model to reconstruct the original signals. Similarly, when the forecasting length is equal to the context length, the prediction becomes extremely challenging, both of which may unstabilize the pre-training process and lead to a decline in performance on downstream tasks.

Appendix C Details of baselines

We extensively compare our model with 12 advanced methods which are divided into three categories, including 1) 3 methods aimed at time series universal modeling; 2) 3 methods based on self-supervised pre-training on brain signals and 3) 6 methods specifically designed for each downstream task. The detailed information of these methods are described below.

For the first category:

• TF-C (Zhang et al., 2022): A decomposable pre-training model for general time series modeling, where the self-supervised signal is provided by the distance between time and frequency components.

• SimMTM (Dong et al., 2023): A pre-training framework on time series to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold.

• One Fits All (Zhou et al., 2023): A unified model that leverages language models for time series analysis, leading to a comparable or SOTA performance in all main time series analysis tasks.

For the second category:

• BrainBERT(Wang et al., 2022): A reusable transformer for intracranial field potential recordings enables classifying complex concepts and decoding neural data.

• Brant(Zhang et al., 2023): A foundation model for intracranial neural recordings which is a large-scale, off-the-shelf model for medicine.

• MBrain(Cai et al., 2023): A multi-channel self-supervised learning framework which explicitly capture the spatial and temporal correlations of brain signals to learn a unique representation for each channel.

For the third category:

• PPi(Yuan et al., 2023): A pre-training-based model for patient-independent seizure detection on SEEG data, which contains two novel self-supervised tasks to extract rich information from abundant SEEG data and two techniques to tackle the domain shift problem.

• ScatterFormer(Zheng et al., 2023): An invariant scattering transform-based hierarchical Transformer that specifically pays attention to subtle features which is designed for patient-independent detection of epileptic based on visual spectral representation of continuous EEG.

• Lopes et al. (2023): A deep convolutional neural network-based EEG artefact removal model designed for seizure prediction using a deep convolutional neural network connected to a bidirectional long short-term memory layer (CNN-BiLSTM) using time series as input and a shallow artificial neural network trained using established handcrafted features.

• SleepHGNN(Jia et al., 2023): A novel sleep heterogeneous graph neural network designed to capture interactivity and heterogeneity of physiological signals for accurate sleep stage classification.

• EEG Conformer(Song et al., 2023): A compact convolutional Transformer to encapsulate local and global features in a unified EEG classification framework for motor imagery and emotion recognition.

• TSFF-Net(Miao and Zhao, 2023a): A novel network architecture designed for motor imagery classification that integrates time-space-frequency features, effectively compensating for the limitations of single-mode feature extraction networks based on time-series or time-frequency modalities.

For TF-C, SimMTM, BrainBERT, Brant and MBrain which need to be pre-trainined and applied on all the downstream tasks, we utilize the same pre-training corpus of Brant-2 to pre-train these baselines for fair comparison. During fine-tuning, we fine-tune all the parameters of these baselines.

Appendix D Details of Experimental setup

D.1. Evaluation Metrics

In the seizure detection and prediction tasks, following the existing works(Cai et al., 2023; Yuan et al., 2023; Chen et al., 2022), we adopt precision, recall, F1 score and F2 score as the evaluation metrics. For the sleep stage classification, following the existing works (Supratak and Guo, 2020; Phan et al., 2021, 2023), we use accuracy, sensitivity, specificity, macro F1 score, and Cohen’s kappa $\kappa$ as evaluation metrics. For emotion recognition and motor imagery classification, we use accuracy and F1 score as evaluation metrics. Detailed information of these metrics are given as follows:

• Precision: Also known as positive predictive value (PPV), precision is the proportional accuracy of correctly identified positive outcomes out of all predicted positive outcomes. It’s a crucial metric when the cost of a false positive is high. The higher the value, the more relevant the results returned by the model. A lower value would mean that the model returns more false positives.

  (9) $$Precision=\frac{TP}{TP+FP},$$

  where $TP$ is the number of true positives and $FP$ is the number of false positives.

• Recall (Sensitivity): Also known as the true positive rate or sensitivity, recall measures the proportion of actual positive observations that are correctly identified as such. It helps us understand the predictive capacity of the model concerning the positive class. The higher the sensitivity, the fewer real positive cases the model will miss. A value of 1 means the model has perfect sensitivity and is not missing any real positives.

  (10) $$Sensitivity=Recall=\frac{TP}{TP+FN},$$

  where $TP$ is the number of true positives and $FN$ is the number of false negatives.

• Specificity: Also known as the true negative rate, specificity measures the proportion of actual negatives that are correctly identified. This provides insight into the predictive capacity of the model for the negative class. A higher specificity value means that the model is good at avoiding false positives, whilst a lower specificity indicates that the model often predicts a positive outcome when it’s actually negative.

  (11) $$Specificity=\frac{TN}{TN+FP},$$

  where $TN$ is the number of true negatives and $FP$ is the number of false positives.

• F-measure: The F-measure is a metric defined as the weighted harmonic mean of precision and recall, with the following equation:

  (12) $$\text{F}\beta=\frac{(1+\beta^{2})\times precision\times recall}{\beta^{2}\times precision+recall}.$$

  In the scenario of epilepsy, F2 is more valued than F1, since ignoring any seizure is costly in diagnosis. While in other scenarios like sleep or emotion, F1 score is more valuable as a reference as it seeks a balance between precision and recall. For multi-class problems, macro $\text{F}\beta$ score calculates $\text{F}\beta$ for each class independently and then averages them. A higher macro $\text{F}\beta$ score indicates that the classifier has both good precision and good recall.

• Cohen’s Kappa: Kappa $\kappa$ is a statistic that measures inter-rater agreement for qualitative items. It generally measures how well the model is performing over the random prediction. The value lies between -1 to 1. A high positive value (close to 1) signifies that the model’s predictions align well with the actual results beyond what would be expected by chance, a value of 0 indicates alignment similar to random chance, and a negative value indicates agreement less than chance.

D.2. Fine-tuning Details

For most of the datasets (MAYO, FNUSA, CHB-MIT, Siena, Clinical, SEED and Motor Imagery), we find that only fine-tuning the last two layers of the temporal encoder and freezing the remaining parameters of the Brant-2 encoder, we can achieve satisfactory results. Thus, for the evaluation on the above datasets, we only fine-tune the last two layers of the temporal encoder with a learning rate of $1.0\times 10^{-5}$ and the classification head with a learning rate of $1.0\times 10^{-3}$. For datasets SleepEDFx and HMC, we fine-tune all the parameters of Brant-2 with a learning rate of $1.0\times 10^{-6}$.

D.3. Model Configurations

Table 6. Configurations of Brant-2 with different sizes.

	Temporal/Spatial Encoder Layer	Model Dimension	Inner Dimension	Parameter Count
Brant-2-100M	10/2	768	2304	115M
Brant-2-200M	10/2	1024	3072	204M
Brant-2-460M	10/2	1536	4608	459M
Brant-2	8/2	2560	7680	1065M

The model configurations of Brant-2 with different sizes are shown in Tab. 6.