A Review of Common Online Speaker Diarization Methods

There are already two papers [1], [3] that provide a review of speaker diarization systems. These papers discuss the historical development, evaluation metrics, different diarization methods, common datasets and current use cases of speaker diarization. However, these papers do not explicitly address online speaker diarization methods and systems. This paper aims to fill this gap by conducting a review of common online speaker diarization methods.
To give a good overview of online diarization methods, the rest of this paper is structured as follows. The next section will give a general introduction to online diarization. For this purpose, the historical development is presented, the taxonomy for the online diarization systems of this paper is introduced and common evaluation metrics are presented. Next various online diarization systems are briefly summarized. Finally, challenges of online speaker diarization will be presented and an outlook will be given.

The first preliminary work on online speaker diarization was published in 1999. In their work, Daben Liu et al. [4] present an algorithm for recognizing speaker changes in real time. Their system is based on Hidden Markov Models (HMM) in combination with Gaussian Mixture Models (GMM) to define audio classes. A maximum likelihood distance is calculated to recognize a speaker change.

A few years later in 2003, Daben Liu et al. [5] published a paper on online speaker clustering algorithms that have comparable performance to offline speaker clustering algorithms. Two of the newly developed algorithms even outperform the offline hierarchical clustering chosen as a baseline.

In the following years, the first online speaker diarization systems were introduced. These systems are generally structured as follows, with minor deviations. For SAD, these systems use energy-based SAD systems or consider non-speech as a separate class. For segmentation, the audio file is divided into Mel Frequency Cepstral Coefficients (MFCC) of constant length. GMMs are used as audio and speaker representations. Depending on the system, the GMMs are combined with a Universal Background Model (UBM), to also represent the speaker-independent part of the acoustic features. Agglomerative Hierarchical Clustering (AHC) is the most commonly used clustering algorithm [6][7][8][9][10]. Some other systems also include the speaker’s physical location in the diarization process to further improve the results. [11][12].

With the introduction of i-vectors [13] and d-vectors [14], audio and speaker representations in the form of GMMs were replaced in online speaker diarization [15]. The use of neural network-based d-vectors in particular led to a leap in performance in speaker diarization [16] [17].

End-to-end systems are the latest development in the context of speaker diarization. Here, a single neural network takes over the individual sub tasks of speaker diarization. These end-to-end systems are also used for online speaker diarization. Especially in combination with self-attention, these systems lead to an improvement in performance [18][19][20][21] [22].

In this paper, online diarization systems are divided into two categories. All systems with a modular structure are assigned to the first category. This includes all systems that process at least one sub-task separately. All pure end-to-end systems are assigned to the second category. These include systems that process all sub-tasks with a single machine learning model. Table I provides an overview of the systems. Further systems are not considered in more detail in this paper, but are included in the assigned category.

The metrics described here are not purely online speaker diarization metrics. They are also used in offline speaker diarization.

The datasets for online speaker diarization are the same as for offline diarization systems. However, the datasets are handed to the online speaker diarization systems as a stream. The following datasets are used most frequently:

• •

  CALLHOME: This dataset contains 500 telephone multilingual language sessions with 2 to 7 speakers [28].

• •

  2003 NIST Rich Transcription: This dataset is a collection of different datasets. It contains a total of 13 hours of annotated multilingual speech with several speakers [29].

• •

  DIHARD (I/II/III): These datasets build on each other. The DIHARD datasets are multilingual and contain 1-8 speakers [30][27][31].

• •

  VoxConverse: This dataset contains over 50 hours of multilingual speech extracted from YouTube videos [32].

As already described in section II-A, the first online diarization systems were primarily built with GMMs. GMMs are essentially an iterative soft clustering. First a Gaussian distribution is randomly initialized for each cluster. Next the probabilities of belonging to the Gaussian distribution are then calculated for each data point. Data points that belong to one of the Gaussian distributions with a very high probability (above a certain threshold) are assigned to this Gaussian distribution or cluster. All other data points are assigned to several clusters. After that new Gaussian distributions are calculated from the assigned data points, this process is repeated iteratively until the Gaussian distributions no longer show any major changes. In speaker diarization, MFCCs are taken as data points and a speaker is represented by the mean vectors and the covariance matrices of a GMM. Due to their iterative nature, GMMs can also be used well in online speaker diarization systems [33] [34].

As an example of an online diarization system based on GMMs, the work of Markov et. al. [7] will now be described in more detail. Markov et. al. have developed a system that decides for each new speech segment whether it matches a known speaker GMM. If no matching speaker GMM is found, a new speaker GMM is created. Speaker GMMs are deleted if no new speech segment has been assigned to the GMM for a long time.

The system uses three different GMMs as SAD component. The first represents non-speech. The other GMMs represent voice characteristics of the two biological sexes.

Segmentation is performed using a logic that includes parameters such as minimum segment length (MSL), maximum pause in segment (MPS) and maximum speech in pause (MSP).

In the subsequent clustering component, the system decides whether it is a new speaker or a known speaker. This decision is made using a likelihood ratio. If it is a known speaker, the GMM of this speaker is updated with the new data point. If it is a new speaker, a new speaker GMM is created from the associated gender GMM. This allows the system to integrate new speakers online. By deleting speaker GMMs that are no longer used, the system is able to process audio endlessly. From a latency of 3 seconds, the system delivers solid performance on the evaluation dataset with a DER <12%.

I-vectors have developed from the problems of GMMs with intersession variability. In this context, intersession variability means that the same speaker sounds different in different recordings. Joint Factor Analysis (JFA) was proposed to counteract this problem [35]. The JFA breaks down the vector of the GMM into individual components. The result is a speaker independent component, speaker dependent component, channel dependent component and a residual component.

In a subsequent study, it was found that the channel dependent component also contains information about the speaker [36]. As a result, the speaker component and the channel component were combined into a common variability matrix. The column weights of this combined matrix are also referred to as i-vectors. In the following years, these i-vectors were used as a representation for audio segments in both online and offline speaker diarization systems.

Dimitriadis et al. [15] have developed an online diarization system that uses i-vectors, among other things. The system consists of a SAD, a segmentation and a clustering component. In addition, an ASR component is integrated to improve the performance of the system. The system architecture of Dimitriadis et. al. is visualized in figure 2. The components of this system will be described in more detail in the following.

The SAD component is based on a neural network and was inspired by the work of Thomas et al. [37]. The output of the SAD component is passed directly to the ASR module. This validates that no words are contained in non-speech segments. Thereby the ASR module can eliminate many false positives from the SAD component.

The segmentation component receives continuous audio segments in the form of cepstral acoustic features and the ASR transcript as input. The audio segment is then split into two segments at each word boundary. Two Gaussians are fitted with the two sub-segments and compared with the BIC algorithm [38]. If the two sub-segments differ sufficiently, a speaker change is set here. The additional ASR module ensures that speaker changes are only set at word boundaries.

For the clustering, i-vectors are generated from the homogeneous audio segments. These are then clustered using the x-means algorithm. The x-means is a variation of the k-means algorithm [39]. The x-means has a linear complexity proportional to the number of audio segments to be clustered. However, in order to ensure online diarization, long audio files cannot be clustered on the entire history. Therefore, Dimitriadis et al. introduce an active window to limit the history. However, this means that clustering of two different active windows is no longer consistent. To solve this problem, the system provides a fast speaker label based on the clustering of the active window. A more accurate speaker label is then delivered with a slightly higher latency. The accurate label is generated using a reconciliation algorithm [40], which minimizes the hamming distance of the speaker labels between two adjacent active windows. This approach gives the end user the option of receiving speaker labels with a short latency and updating them with the more accurate labels at a later point in time.

As already described in Section II-A Historical development, the current speaker diarization systems are based on neural networks (NN). There is one approach of replacing the entire system with a single neuronal network. These end-to-end systems are discussed in more detail in the next section IV. Another approach is to replace individual sub-tasks with trainable neural networks. In the online diarization system by Zhang et. al. [16], the focus is on the use of a fully supervised clustering component. This system is described in more detail below.

As SAD, two simple Gaussian distributions are used to filter out audio segments that do not contain speech. Subsequently, overlapping audio segments are cut. For the segments, representations are generated in the form of d-vectors [14]. These d-vectors are used as input for supervised clustering. The algorithm unbounded interleaved-state recurrent neural network (UIS RNN) was developed as a clustering component for this system. UIS RNN receives three sets as input in the training scenario:

• •

  $X=(x_{1},x_{2},x_{3},\text{…},x_{T})$ where each $x_{t}$ is the d-vector of an audio segment

• •

  $Y=(y_{1},y_{2},y_{3},\text{…},y_{T})$ where each $y_{t}$ is the speaker-id of the corresponding audio segment

• •

  $Z=(z_{2},z_{3},\text{…},z_{T})$ where $z_{t}=1$ if there is a speaker change at time $t$. In all other cases, $z_{t}=0$

The aim of UIS RNN is to represent the combined probability of $p(X,Y,Z)$. UIS RNN can therefore be split into three individual components.

Supervised speaker diarization systems need a lot of training data in order to deliver good results. The annotation of audio data is time consuming and therefore cost intensive. As a rule, you can expect to spend 2 hours on a 10 minute audio file [23]. In their work, Xia et. al. [23] present a system that can be trained on the basis of speaker turn labels. For this purpose, $<st>$ tokens are inserted into the ASR transcript of the audio file at each speaker turn. This means that an exact timestamp no longer needs to be set during annotation. This speeds up labeling many times over. In addition, the semantic information in the audio data can be processed better.

The system from Xia et al. is structured as follows. A transformer transducer model takes over the ASR and speaker turn detection. The detected segments are then fed into a Long short-term memory model (LSTM) in order to calculate the corresponding d-vectors. An online variant of spectral clustering is used to cluster the d-vectors, which also includes the speaker turns for decision making. The individual system components are described below.

In their work, Liang et al. [20] present an online speaker diarization system that processes the audio stream frame by frame and delivers the corresponding diarization results in real time. The system consists of an audio encoder and an attractor decoder. In this case, an attractor is the representation of a speaker. Finally, diarization results are generated by a similarity comparison of the attractors with the audio embeddings. The system architecture can be seen in figure 3.

The decoder is implemented with non-autoregressive self-attention. On the one hand, self-attention ensures that the speaker labels remain consistent over time. On the other hand, self-attention makes it possible to better distinguish the speakers. In order to achieve better performance, a look-ahead mechanism is added to the system. This mechanism receives some future frames as input and thus makes it possible to recognize new unknown speakers in real time. The architecture of the whole system is analyzed in more detail below.

In their work, Lin et al. [25] present a benchmark for speaker diarization systems based on online learning. In addition they develop an online learning speaker diarization system. In the following, both the benchmark and the online diarization system will be examined in more detail.

The Minivox Benchmark converts classic speaker diarization datasets into audio streams. To do this, random speaker n and associated audio sequences m are attached to each other. The parameter p can be used to control how many of the speaker labels the system receives as feedback. The parameter p thus simulates a real user who does not always provide feedback. A result can look like this, for example:

• •

  $X=[1,1,1,2,2,2,1,1,3,3,3,3,…]$

• •

  $Y=[1,\_,\_,2,\_,\_,\_,1,\_,\_3,\_,…]\ for\ p\ =\ 0.3$

Training can be carried out either with or without an oracle. The oracle specifies the maximum number of speakers as initial input. The performance of the system is measured with the DER.

The presented Online Speaker Diarization System is based on the concept of the Contextual Bandit [44]. For each decision option, the bandit has an arm that stands for an unknown reward. The bandit tries to find a good trade-off between exploiting known options and discovering new, possibly better options. In case of the contextual bandit, the bandit is given a context (e.g. audio embeddings) that he can incorporate into his decision. The context in this work is the maximum number of speakers.

If the online diarization system starts with the maximum number of speakers N, the bandit initially receives N arms, as well as an additional arm for no speech. Without this information, the bandit starts with one arm for new speaker and one arm for no speech. The system then begins the learning and inference process. The following cases must be distinguished:

• •

  The system selects new speaker and the user confirms the selection: The system initializes a new speaker.

• •

  The system selects no speech and the user objects to the selection: The system initializes a new speaker.

• •

  The system selects speaker $n_{x}$ and the user objects with new speaker: The system initializes a new speaker by copying speaker $n_{x}$.

In addition, it may be the case that the user does not provide any feedback. A self-supervision module has been added to the system for this purpose. This clusters the audio segments that have already been labeled and the current audio segment. Then the cluster label is used as a substitute for the user feedback. Online variants of GMM, k-means and KNN are used as cluster algorithms. The authors are aware that these perform poorly compared to there offline variants, but want to ensure the online suitability of the system.

The presented online diarization system has a comparatively poor performance. However, the online learning scenario poses a particular challenge, as the system never gets to see the real labels of the audio segments unless the user confirms an assigned speaker label.

In many current online diarization systems, it can be observed that the error is reduced if a higher latency is accepted. The reason for this is that more audio information can be included in the decision if a larger input sequence is available. Depending on the system, this trade-off is handled differently. Coria et al. [18] use an active window that is controlled by the parameter $\lambda$. The larger $\lambda$ is selected, the higher the latency and the lower the DER. Morrone et al. [45] use a similar approach in their work with a CSS window based on the sliding window of Chen et al. [46]. Here it can be also observed that the DER decreases when the window is enlarged. Liang et al. [20] show the system some future frames via the look-ahead mechanism. As described in section IV-A, the latency is controlled via the kernel size of the convolution that makes the future frames available. Dimitriadis et al. [15] solve the problem by returning a fast label and later delivering a more accurate label. Further examples of the accuracy latency tradeoff can be found in [19] [47] [22].

However, it can be observed that the DER no longer drops significantly above a certain latency. This sweet spot latency differs from system to system. For Coria et al. [18] it is 3s, for Morrone et al. [45] it is a window size of 15s. Nevertheless, there is a need for further research to decouple latency from accuracy.

Current supervised online and offline diarization systems need a lot of high quality training data to deliver good results. Such training data is only available to a limited extent in English and is usually associated with high costs [28] [29] [48]. For some languages, there is still no corresponding training data for speaker diarization. An approach such as Xia et al. Turn to Diarize [23] attempts to simplify the annotation of the training data. Lin et al. [25] try to perform the training process through online learning parallel to inference time. These are good approaches, but further research and work is needed to reduce this problem.

Many end-to-end online diarization systems can only process a limited number of speakers. Horiguchi et al. [43] state in their work that end-to-end online diarization systems always have problems when the number of speakers for inference is higher than the number of speakers in the training process. In their work, they also propose a solution by adding blockwise clustering to the end-to-end system. However, they move away from the actual core idea of end-to-end systems, which is to solve the entire speaker diarization with a single neural network. Also, the maximum number of speakers in online speaker diarization cannot be determined in advance, as the audio file arrives as a stream and additional speakers may be added at a later point in time. This is a research gap that needs to be closed in the future in order to be able to handle a flexible number of speakers for inference.