2.1. Source Localization

When there is at most one active sound source at a time, the problem of audio-visual source association is also known as source localization, i.e., indicating the location of the sound source in the video. For audio-visual speech, source localization is helpful for speaker face segmentation (Liu and Sato, 2008). Early work on speaker localization correlates audio energy changes with pixel motion via non-linear diffusion (Casanovas and Vandergheynst, 2010) or with semantic regions via video segmentation and tracking (Li et al., 2014). Other methods include time-delayed neural networks (Cutler and Davis, 2000), probabilistic multi-modal generative models (Fisher and Darrell, 2004), and Canonical Correlation Analysis (CCA) (Kidron et al., 2007; Izadinia et al., 2013).

More recent work proposes to localize semantic objects in unconstrained videos by learning deep multi-modal representations. Owens and Efros (2018) propose a fused multi-sensory network to learn an audio-visual representation, which further localizes the sound objects on the video frames. Senocak et al. (2018) employ a similar two-stream network structure, where an attention mechanism is developed for sound source localization. A similar idea is adopted by Arandjelović and Zisserman (2018) for cross-modal retrieval and source localization, and by Tian et al. (2018) for both spatial and temporal localization.

2.2. Source Association for Separation

Other work deals with mixtures of active sources, where cross-modal association can be applied to isolate sounds that correspond to each visual object. Barzelay and Schechner (2007, 2010) detect drastic changes (i.e., onsets of events) in audio and video and then use their coincidence to associate audio-visual components that belong to the same source of harmonic sounds. Sigg et al. (2007) reformulate CCA by introducing non-negativity and sparsity constraints on the coefficients of the projection directions to locate and separate sound sources in movies. In (Casanovas et al., 2010), auditory and visual modalities are decomposed into relevant structures using redundant representations for source localization. Segments, where only one source is active, are used to learn a timbre model for the separation of the source. Ephrat el al. (2018) propose a deep network-based model to isolate single speech signals from a mixture of sounds given the target speaker from the video. Gao et al. (2018) map audio frequency basis to individual visual objects via an audio-visual object model, which further guides audio source separation. Most of these methods, however, either deal with mixtures with at most two active sources or only focus on isolating one source from multiple active sources (e.g., background noises). The association problem for each individual source is not addressed.

2.3. Source Association for Chamber Ensembles

The source association problem for music ensembles is more challenging since all the available sound sources (the players) are active almost all the time, and the difficulty increases dramatically as the number of sources increases. Although each score part is performed by one player in chamber music, the same kind of instruments are often used for different score parts (e.g., a violin duet). Therefore, approaches aiming at learning deep representations that map acoustic features with visual appearances to localize each source (Owens and Efros, 2018; Senocak et al., 2018; Arandjelović and Zisserman, 2018) are not applicable. Instead, one needs to recognize the distinct motion of different players and correlate them with the music content to achieve association.

Bazzica et al. (2014) first propose to distinguish play from non-play conditions for each player in an orchestra, which are compared with each score part to solve the temporal alignment. In our previous work (Li et al., 2017a), we propose an approach to solving the association problem for string ensembles with up to five simultaneously active sources in a score-informed fashion. The approach analyzes the bowing motion and correlates it with note onsets in score parts. The assumptions are that many note onsets correspond to the beginning of bowing strokes and that different instrumental parts often have different rhythmic patterns. When these assumptions are invalid, for example, when multiple notes are played within a single bow stroke (i.e., legato bowing) or when different parts show a similar rhythmic pattern, the approach becomes less robust. Later we propose a complementary approach (Li et al., 2017c) which correlates the fingering hand rolling motion with pitch fluctuations of vibrato notes for the association of string instruments. However, the method only works when vibrato notes are played. To our best knowledge, there is neither an existing work on integrating the bowing motion and vibrato motion for source association for string instruments, nor any extensions of the concept to deal with non-string instruments.

3.1. Performance-Score Alignment

As the proposed approach is score informed, a preliminary step for the system is to temporally align the music score with the dynamic timing of the audio-visual ensemble performance (assuming audio and video are pre-synchronized). The temporal alignment is achieved through audio-score alignment on the harmonic content (Müller, 2015). To do so, the audio is first converted to short-time Fourier spectral magnitudes with a 42.7 ms frame length (2048 samples for a 48 kHz sampling rate), 10 ms hop size, Hamming window, and zero padding to produce 4 times the original length. The short-time Fourier spectral magnitudes are then mapped to 12-dimonsional chroma vectors, where each element represents a pitch class. Each chroma vector is normalized by its root mean square (RMS) value. A similar operation is applied to the score, which is segmented into non-overlapping frames of the same duration using the default tempo notated in the score. A 12-D binary chroma vector is calculated for each frame to indicate the presence (taking more than 50% of the frame) or absence of each pitch class. The chroma vector is then normalized by its RMS value.

In offline scenarios where the entire performance is available beforehand, the alignment can be obtained by the dynamic time warping (DTW) algorithm (Müller, 2007), which is robust and efficient (Müller et al., 2006). In online scenarios where the performance data arrives as a live stream, one commonly used framework is an online DTW algorithm (Dixon, 2005), which provides options such as the “forward-backward strategy” to reconsider past decisions (Arzt el al., 2008), or incorporating a tempo model (Arzt and Widmer, 2010) for robustness. An alternative framework employs a stochastic model (Grubb and Dannenberg, 1997; Duan and Pardo, 2011b), where the score position hypotheses are represented by a probability density function. In this paper, to deal with online video streaming scenarios, we apply the online method proposed by Duan and Pardo (2011b), which is based on a hidden Markov model with a 2-D continuous state space to represent the score position and tempo. This framework is previously evaluated on the Bach10 dataset (Duan et al., 2010) showing decent results. A further qualitative check confirms a good alignment performance on the URMP dataset used in our experiments.

3.2. Onset Correspondence with Body Motion

3.2.1. Body Motion Extraction

In music performances, body motion of performers conveys important musical expressions and ideas, e.g., the head nodding at leading notes. For some instruments, body motion directly articulates notes (e.g., strings, drums) or controls the pitch (e.g., trombones). To capture body motion from video recordings, one approach is optical flow estimation. In our previous approach (Li et al., 2017a) we apply optical flow estimation to extract bowing motion of string players. However, we argue that this pixel-level analysis may not be ideal for semantic-level understanding of body gestures and movements, and can be less robust to occlusions and camera viewpoint changes.

In this paper, we propose to apply OpenPose (Cao et al., 2017) on each frame, a multi-person pose estimation approach to extract body skeleton coordinates for all the players on stage without pre-segmentation of the video recording. A skeleton in each frame is represented as a 20-D vector y(t) corresponding to the horizontal and vertical coordinates of the 10 upper body joints, including nose, neck, shoulders, elbows, wrists, and hips. We do not include lower body joints as they are usually less relevant to music events. Figure 2 shows video frames of several instrumentalists with the extracted body skeletons. To form a continuous skeleton sequence across time, we eliminate joint coordinates if the confidence score from OpenPose is smaller than 0.2 and the L₂ distance between a joint in consecutive frames is larger than 10% of the head-hip distance, which is considered the maximal regular movement in a ≈30-FPS video without shot transition. We also temporally smooth their coordinates using a moving average with a 5-frame window size. These post-processing steps are referred from (Li et al., 2018b) where the same approach is applied to extract skeletons for pianists. We then take the two hips as reference coordinates to align the body position across frames. Finally, we calculate motion velocities z(t) as the derivative of y(t) w.r.t. time. Compared to optical flow estimation, this gesture-based motion analysis approach is semantically more meaningful, less computationally expensive, and more robust to occlusions and camera viewpoint changes such as camera zooming or panning.

Figure 2 

Body motion extraction. Upper body skeletons (second row) are extracted with OpenPose (Cao et al., 2017) in each video frame (first row) followed by temporal smoothing.

To extract motion related to note onsets in each video frame, for each player we denote the motion velocities of n frames in the past as Z ∈ ℝ^n×20 and apply principal component analysis (PCA) by eigenvalue decomposition Z^TZ = V∑V^T, where V and ∑ represent the matrix of eigenvectors and the corresponding eigenvalues respectively. We then project the motion velocity z(t) onto the principal component direction (first column of V) and take its absolute value as the motion salience s(t). Choosing the salient motion discards the direction information of the motion (e.g., up/down-bow for violinists), which is less relevant to timing than the amplitude information. We set n to 150 frames, i.e., 5 seconds in time, assuming a player’s pose stays consistent in this range. To reduce the computational cost, we update V every 1 second (assuming consistent motion patterns within a short period).

3.2.2. Onset Likelihood

From the motion salience s(t), we infer the timings of the motion strokes that are potentially related to score note onsets. As a note onset often corresponds to the beginning or ending of a sound articulation motion (e.g., a bowing stroke for string instruments), the motion speed at the onset is often small. Therefore, local minima of the motion salience s(t) are often indicative of note onsets. Let Ω be the set of all the local minima throughout a piece. For each local minimum τ ∈ Ω, we represent the likelihood of a note onset as a(τ) = max_{γ∈[τ,τ+30]}s(γ) – s(τ) that is determined by the maximum speed of the motion stroke considering the following 30 frames: the larger the value of a(τ) the more likely that a note onset is activated by the motion stroke. Here 30 frames are considered to span the high energy part of most notes. Therefore, we can define an onset likelihood curve φ_b(t) derived from body motion analysis as

(1)

${\varphi }_b\left(t\right)\hspace{0.17em}\hspace{0.17em}=\hspace{0.17em}\hspace{0.17em}\left({\displaystyle \sum _{\tau \in \hspace{0.17em}\Omega }a}\left(\tau \right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\cdot \hspace{0.17em}\hspace{0.17em}\delta \left(t-\hspace{0.17em}\hspace{0.17em}\tau \right)\right)\hspace{0.17em}\hspace{0.17em}*\mathcal{N}\left(t\right),$

M1 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\phi _b}\left(t\right)\,\, = \,\,\left({\sum\limits_{\tau\in \,\Omega} a \left(\tau\right) \cdot \delta \left({t - \,\,\tau} \right)} \right)\,\,*{\cal N}\left(t\right), \] \end{document}

where δ(t) is the Dirac delta function, * is the convolution operation, and $\mathcal{N}(t)$M2 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\cal N}(t) \] \end{document} is a Gaussian function to give each predicted onset time a tolerance (width) with a standard deviation of 3 frames (100 ms) (considering some slight non-synchronization between different modalities in the recording file). It is noted that φ_b(t) can be calculated in an online fashion, with a delay of up to 1 second due to the search for the local maximum after each local minimum. Figure 3 plots the onset likelihood curve φ_b(t) along with the associated and temporally aligned score part as piano-roll, where the note onset timings are marked as red circles. We find that many of the note onsets can be associated with peaks of φ_b(t). The correspondence between the notes and peaks sets the basis for the association between score and motion, as described below.

Figure 3 

Example correspondence between body motion and note onsets. Top: temporally aligned score part with onsets marked by red circles. Middle: extracted motion salience (primarily bowing motion) from the visual performance of a violin player. Bottom: derived onset likelihood curve from the motion salience.

3.3. Onset Correspondence with Finger Motion

3.3.1. Finger Motion Extraction

While note articulation is visible in the body movements of string instrumentalists, this is generally not the case for woodwind and brass players, where notes are articulated by blowing into the reed or mouthpiece, showing little visible motion around the mouth. However, pitch changes of these instruments are mostly controlled by finger-operated keys, resulting in synchronized events between finger movements and note onsets (Palmer et al., 2007). Compared to body motion, finger motion is more subtle and more prone to occlusion. In this section, we propose to extract finger motion and correlate it with note onsets.

We apply OpenPose again to extract the positions of all the finger joints from each player. Due to the limited video resolution and occlusion, the result is not robust enough to estimate the motion. Inspired by our previous work (Li et al., 2017c), we use optical flow estimation (Sun et al., 2010) to capture this subtle motion at the pixel level. To reduce the computational cost, we set a region of interest (ROI) around the detected finger joints from OpenPose for optical flow estimation. The ROI is centered at the median of all the finger joints for each hand, and spans to cover all the joints. Similar to body skeletons, we smooth the joint coordinates using a moving average filter with a window size of 5 frames. Then we compute the optical flow estimation inside the ROI. Again, to eliminate the rigid and affine motion, each optical flow vector is reduced by the average motion vector of the ROI, resulting in a motion vector u^{(i j)}(t) at pixel (i, j) and frame t. Figure 4 takes one flute player and one clarinet player as examples to visualize the optical flow estimation of one-hand finger motion in five consecutive frames, where the estimated finger joint positions are overlaid on the first video frames.

Figure 4 

Optical flow visualization of finger motion in five consecutive frames corresponding to note changes. The color encoding scheme is adopted from Baker et al. (2011).

3.3.2. Onset Likelihood

On each frame we take the maximum value of pixel-wise motion magnitude |u^{(i j)}(t)| across all the pixels in the ROI as the motion flux, which captures the finger movements corresponding to pitch changes and is directly considered as onset likelihood φ_f(t) from finger motion. Figure 5 plots an onset likelihood curve φ_f(t) along with the associated and temporally aligned score part as piano-roll. We can observe salient motion flux around most note onset frames. Compared to Figure 3, the correspondence of note onsets to fingering motion for woodwind and brass instruments is not as robust as that to body motion for string instruments. The observation can be attributed to the fact that fine-grained motion is more sensitive to irrelevant motion. In addition, repeated notes for wind instruments are usually not reflected by finger maneuvers on the keys.

Figure 5 

Example correspondence between finger motion and note onsets of a flute player. Top: temporally aligned score part with onsets marked by red circles. Bottom: extracted motion flux from finger movements.

Analogous to Eq. (2), the pair-wise matching score from finger motion can be calculated as:

(3)

$M_f^{[p,\hspace{0.17em}q]}\left(t\right)={\displaystyle \sum _{\tau =\hspace{0.17em}\hspace{0.17em}0}^t{\varphi }_f^{[p]}}\left(\tau \right)\hspace{0.17em}\hspace{0.17em}\cdot \hspace{0.17em}\hspace{0.17em}{\psi }^{[q]}\left(\tau \right).$

M5 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ M_f^{[p,\,q]}\left(t\right) = \sum\limits_{\tau= \,\,0}^t {\phi _f^{[p]}} \left(\tau\right) \cdot {\psi ^{[q]}}\left(\tau\right). \] \end{document}

3.4. Pitch Correspondence with Vibrato Motion

In addition to the onset time, variations of acoustic features throughout the entire process of some note articulations show correspondence with specific motion. Vibrato is one such feature. Vibrato is a commonly used artistic note articulation method to color a tone and express emotions in music performances. Physically, vibrato is generated by pitch modulation of a note in a periodic fashion. For string instruments, vibrato is often visible as the left hand rolling motion on the fingerboard. The relationship between visible motion and vibrato motivates us to follow our previous work (Li et al., 2017c) to extract the fine motion and find the correspondence with pitch contours extracted from the audio modality.

3.4.1. Vibrato Motion Extraction

We retrieve the finger motion u^{(i j)}(t) as computed from the previous section. Although the vibrato motion is mostly a rigid motion (fingers move together with little relative movement), it is periodic and very fast (usually about 4–7.5 Hz (Geringer et al., 2010)), and hence it is not removed as slower rigid/affine motions are. Figure 6 illustrates several frames of the optical flow estimation of the vibrato hand motion for the two players. For each frame t, we take the average motion vector across all pixels within the ROI as u(t) = [u_x(t), u_y(t)]^T, where the motion direction is preserved for vibrato detection.

Figure 6 

Optical flow visualization of hand motion corresponding to vibrato articulation. The color encoding scheme is adopted from Baker et al. (2011).

The vibrato detection module works as a binary classifier as proposed and trained by Li et al. (2017b). The classifier is implemented as a support vector machine (SVM) that takes as input an 8-D feature extracted from each sample, including the zero crossing rate of the x- and y- motion velocities and their auto-correlations, the energy in the 3–9 Hz frequency range, and the auto-correlation peaks. According to Li et al. (2017b), this method achieves a vibrato detection accuracy of over 90%, regardless of the polyphony number and instrument type within the string instrument family. Here each input sample is a 1-second segment of u(t) (again introducing an average 0.5-second delay of the association system).

For each detected vibrato segment, we perform PCA on u(t) within this 1-second segment to obtain the 1-D principal motion velocity curve v(t). We then integrate v(t) over time to calculate a motion displacement curve, d(t), which corresponds to the length fluctuation of the vibrating string, and hence the pitch fluctuation of the note. We normalize each vibrato segment of d(t) to zero mean and unit variance. We set the non-vibrato segments of d(t) to zero.

3.4.2. Pitch Contour Extraction

Utilizing the score information, we apply Soundprism (Duan and Pardo, 2011a), an online score-informed source separation system, to separate the polyphonic audio mixture into individual sources. Note that although audio recordings of individual instrumental tracks are available in the dataset, we do not use them as they are not generally available in real concert scenarios. To extract the pitch contour, we perform a score-informed pitch estimation step on each separated audio source, as described by our previous work (Li et al., 2017c). The pitch contour of each note segment is normalized to have zero mean and unit variance, and is denoted as f(t). The normalization operation discards the original pitch height information, and only preserves the pitch changes from the central frequency within each note. Figure 7 plots a 1-second segment of the normalized pitch contour overlaid with a motion displacement curve from the associated track (left) and a random track (right). Similar to Eqs. (2) and (3), we calculate the vibrato correspondence as:

(4)

$M_v^{[p,\hspace{0.17em}q]}\left(t\right)={\displaystyle \sum _{\tau =\hspace{0.17em}\hspace{0.17em}0}^t{d}^{[p]}}\left(\tau \right)\cdot f^{[q]}\left(\tau \right).$

M6 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ M_v^{[p,\,q]}\left(t\right) = \sum\limits_{\tau= \,\,0}^t {{d^{[p]}}} \left(\tau\right) \cdot {f^{[q]}}\left(\tau\right). \] \end{document}

Figure 7 

The same segment of normalized pitch contour f(t) (green) overlaid with the motion displacement curve d(t) (black) from the associated track (left) and another random track (right).

3.5. Integrating All Correspondences

We integrate the three modules to calculate the pair-wise correspondence between visual motion and score or audio events considering both onset timing and the entire note articulation process. The calculation is presented as

where ${\overline{M}}_b^{[p,\hspace{0.17em}q]}(t)$M8 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_b^{[p,\,q]}(t) \] \end{document} , ${\overline{M}}_f^{[p,\hspace{0.17em}q]}(t)$M9 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_f^{[p,\,q]}(t) \] \end{document} , and ${\overline{M}}_v^{[p,\hspace{0.17em}q]}(t)$M10 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_v^{[p,\,q]}(t) \] \end{document} represent the normalized correspondence across all of the pair-wise combinations between N players and N tracks as:

and w_b, w_f, w_v represent the weighting parameters to re-scale the normalized correspondences from different modules. Weight w_v is set as 2w_f, to place greater emphasis on finger motion with vibrato patterns. Weights w_b and w_f are linearly related to their motion salience/flux in the past frames as

The linear relationship recovers the original scale of body and finger motion to weight the correspondences M_b(t) and M_f(t). It allows the system to focus on the part with stronger motion cues, such as body motion for string instrumentalists and finger motion for wind instrumentalists. In Section 4, we test the components in isolation as well as some combinations of them.

For an ensemble with N players, the number of possible associations is the factorial of N. Let σ(·) be a permutation function from p ∈ [1, N] to q ∈ [1, N] that represents one association candidate, where the p-th player is associated with the σ(p)-th track. For each association candidate σ, we calculate an overall association score as the product of the N pair-wise correspondence values. The final association solution $\sigma ˆ$M13 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \hat{\sigma} \] \end{document} is returned to maximize the association score as:

The replacement of product with sum of negative logarithms makes the efficient Hungarian algorithm (Kuhn, 1955) directly applicable for finding the best association.

4.1. Dataset

The proposed source association system is evaluated on the URMP dataset (Li et al., 2019). To our best knowledge, this is the only publicly available multi-track audio-visual music performance dataset that is suitable for our evaluations. It contains 44 classical chamber ensemble pieces ranging from duets to quintets, assembled from 149 individually recorded tracks. Each piece comes with an audio recording (48 kHz, 24 bits) of the ensemble performance along with the audio recording for each individual instrument track, an assembled video recording (1080P, 29.97 FPS) of all instrumentalists as a whole, pitch and note annotations for each track, and the corresponding MIDI file as music score. In the assembled video recording, players are arranged horizontally from left to right, with the right-front side exposed to camera. The video has a static view without camera panning or zooming or shot transitions during the whole performance. The whole dataset is accessible from (Li et al., 2018a).

We further expand the dataset by creating all possible track combinations within each piece. In the expanded set, audio is remixed from the provided individual audio tracks. For videos, we directly use the estimated pose of each player from the original video ensembles for augmented track combinations. This process gives equivalent results as if we first create the assembled videos of the augmented instrumental combinations and then run OpenPose on these assembled videos, but requires less computation in the experiments. For the example of a quartet, we further generate 6 duets and 4 trios from the 4 original tracks. Note that we do not combine tracks across pieces, to ensure the naturalness of the expanded set. The total expanded dataset comprises 171 duets, 126 trios, 47 quartets, and 7 quintets. The number of pieces for different instrument arrangements are listed in Table 1.

To further understand the dataset, we calculate the onset overlap rate for each original piece. This statistic is defined as the percentage of onset positions that are shared by two or more tracks for each piece. This statistic is relevant to the performance of the proposed source association approach, as two out of the three motion analysis modules rely on onset patterns to associate players with tracks. Figure 8 plots this statistic for all of the original 44 pieces. While the rate varies much from one piece to another, we see a general increasing trend as the polyphony increases.

4.2. System Setup

For implementation, the audio is processed with a frame length of 42.7 ms and a hop size of 10 ms for score following and pitch contour extraction. When calculating the vibrato correspondence, the motion curve extracted from the 29.97-frame-per-second (FPS) video is up-sampled to 100 FPS, enabling a synchronized time resolution between the audio and video. As vibrato detection is performed on 1-second segments and the onset likelihood curve from body motion is derived from a local maximum within future 30 video frames (≈1 second), the system has a 1-second inherent delay when it runs for real-time applications. The past 5 seconds of body and finger motion velocities are stored in memory to apply PCA (described in Section 3.2.1) and to calculate the weighting parameters in Eq. (7).

For evaluation, we first address each track independently to investigate the quality of the extracted onset likelihood features, using the traditional onset detection measures. Then we evaluate the association performance on the expanded set of ensemble pieces. The results are grouped by different ensemble types and sizes, from duets to quintets, which directly correlates with the difficulty level. Note that whatever number of tracks are present in the performance, only one association is correct. We do not include a quantitative evaluation of the score following and vibrato detection modules in this paper, since they have been fully evaluated in the previous work.

4.3. Onset Detection Evaluation

As two modules of the proposed system rely on the synchronization cues of onset timing between different modalities, we evaluate the quality of our proposed onset likelihood curves that are extracted from body motion and finger motion. To do so, we set up an onset detection task. We take the onset likelihood curve as the onset detection function (Bello et al., 2005), and perform peak-picking to retrieve the onsets. A true positive detection is counted when a detected onset is within a tolerance window of 3 video frames (100 ms). This is wider than the standard 50ms in the literature, since the precise timing is not the main focus of the source association system.

Figure 9 plots the precision versus recall by varying the peak-picking threshold on the onset likelihood curves extracted from body motion and finger motion respectively. Precision and recall are calculated for each instrument across all pieces in the original dataset. Observing Figure 9 reveals that the onset likelihood curve extracted from body motion shows better correlation with the ground-truth onset timings for string instruments, while, that from finger motion shows better correlation for woodwind and brass instruments. An exception is trombone, where the onset likelihood curve extracted from body motion shows better correlation than that from finger motion. This observation is not surprising as the trombone pitch change (hence note transition) is mainly performed by moving the slide using the right arm (body motion).

Another interesting observation is that although the onset likelihood curve φ_b(t) in Figure 3 is visually less noisy than φ_f(t) in Figure 5, the recall calculated from φ_b(t) for string instruments cannot reach as high a value as that of wind instruments calculated from φ_f(t). We argue that this is because legato bowing (i.e., articulating a sequence of notes from one sustained bowing action) is widely used in string instrument performances, where onset detection from bow motion misses some true positives. This explains the upper bound of recall rates (around 80% as in Figure 9(a)) for string instruments. For wind instruments, there are also onsets that are not visible such as repeated notes, but the number is much smaller, which explains why the recall rates can reach closely to 100% in Figure 9(b).

4.4. Source Association Evaluation

In this section we evaluate the source association performance, first for each module (corresponding to each component in Eq. (5) independently, then for the finally integrated approach. We use association accuracy as the evaluation measure, which is defined as the percentage of correctly associated pieces among all testing pieces. A piece is considered correctly associated if the exactly correct bijection between players and score or audio tracks is retrieved. Note that the difficulty of source association increases dramatically from small to large ensembles. In a quintet ensemble, there are in total 5! = 120 bijection candidates, and only one is considered correct. Therefore, we divide our evaluation based on the size of ensembles.

Besides the ensemble size, the length of the performance also affects the difficulty of the association problem, assuming longer pieces provide richer cues. In an online setting, we hope that the proposed system can retrieve the correct association as quickly as possible. Therefore, in the experiments, we segment the testing pieces into non-overlapping excerpts for each of the following lengths: 5, 10, 15, 20, 25, and 30 seconds. When doing so, we first remove the beginning and the last 5 seconds of each piece as the performance may not cover the entire length of those segments. This segmentation further expands the testing pieces to a large number of evaluation samples, totaling 17,574 samples, as presented in Table 2.

4.4.1. Body Motion

We first evaluate the source association performance using the normalized onset correspondence ${\overline{M}}_b$M15 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{b} \] \end{document} between score parts and body motion (the first component of Eq. (5)). Figure 10(a)–(c) shows the association accuracy for ensembles consisting of string, wind, and mixed instruments with various levels of polyphony. Note that the “All Ensembles” evaluated in Figure 10(c) and (f) contain all the instrument categories from Table 2, i.e., String+Wind+Mixed. For each piece, we plot how the association accuracy varies as the duration of the input stream increases from 5 to 30 seconds. Each marker in the figure is the association accuracy calculated from the number of excerpts shown in Table 2.

Figure 10 

(a)–(c): Source association accuracy only using onset correspondence between score parts and body motion (the first component ${\overline{M}}_b$M16 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{b} \] \end{document} in Eq. (5)). (d)–(f): Source association accuracy only using onset correspondence between score parts and finger motion (the second component ${\overline{M}}_f$M17 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{f} \] \end{document} in Eq. (5)).

Comparing different ensemble sizes, the association accuracy decreases as the number of players or tracks increases. From Figure 10(a), we find that correlating onsets with body motion is beneficial for string instruments. Note that this evaluation is reproduced from our previous work (Li et al., 2017a) as one baseline system here. The accuracy increases as the duration of video stream increases, which provides more cues to solve the association. The accuracy reaches around 90% for all ensemble sizes when the video stream duration reaches 30 seconds. This strategy based on onset correspondence from body motion, however, is not effective for wind instruments, where the association accuracy remains around random guess accuracy as shown in Figure 10(b), e.g., 1/6 for trios. This observation is consistent with our expectations and the onset detection evaluations in Figure 9.

4.4.3. Vibrato Motion

The correspondence between pitch fluctuations and vibrato motion (denoted as ${\overline{M}}_v$M19 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{v} \] \end{document} , the third component of Eq. (5)) helps to retrieve the source association on a finer level for string instrumentalists. The evaluation result is plotted in Figure 11(a) for the same set of pieces performed by string ensembles used for evaluations plotted in Figure 10(a). Note that this baseline is the same system as proposed in our previous work (Li et al., 2017c). We do not include the wind instrument group here since no vibrato pattern can be detected from finger motion. We find that the source association reaches a high accuracy from shorter video clips, i.e., 90% after 10 seconds. The limitation of this approach is that vibrato articulation is not guaranteed to be always present in the performance. We thus combine this module with the onset correspondence from body motion, the two dominant cues to solve association for string instruments, to evaluate the association accuracy as shown in Figure 11(b). The two components from ${\overline{M}}_b$M20 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{b} \] \end{document} and ${\overline{M}}_v$M21 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{v} \] \end{document} work together to reach a high association accuracy from a short video stream.

Figure 11 

Source association accuracy of string ensembles by (a) only using vibrato correspondence between pitch fluctuation and hand motion (${\overline{M}}_v$M22 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{v} \] \end{document} in Eq. (5)), and (b) combining vibrato correspondence with onset correspondence from body motion (${\overline{M}}_b$M23 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{b} \] \end{document} and in ${\overline{M}}_v$M24 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\overline M}_{v} \] \end{document} Eq. (5)).

4.4.4. The Integrated System

Finally, we evaluate the proposed complete source association system after integrating all the modules together, as presented in Eq. (5). The evaluated pieces are the same as the ones plotted in Figure 10. This presents a universal source association system for common melodic instruments. Overall wind instruments have less chance to retrieve the correct association than string instruments, since only the subtle finger motion contributes to the correspondence with onset events. This correspondence is often inaccessible due to overlapping onsets across tracks or repeating onsets as analyzed in Section 4.4.2. Comparing Figure 12(a) with Figure 11(b), or Figure 12(b) with Figure 10(e), we observe that adding components with irrelevant association cues does not harm the system, thanks to the weighting strategy in Eq. (5) over different modules. Comparing Figure 12(c) with Figure 10(c) and (f), the integrated system greatly improves the association accuracy for pieces with mixed types of instruments. The association accuracy for mixed ensembles is between that of pure string and wind ensembles.

Figure 12 

Source association accuracy of ensembles with different instrumentation using all of the three modules: onset correspondence from body motion, onset correspondence from finger motion, and vibrato correspondence from hand motion (Eq. (5)).

4.5. Discussion

The proposed source association system is designed and evaluated for the online scenario, where all the system components do not rely on the performance data after the current time instant. Note that due to the limitations of the dataset, we have not systematically evaluated the robustness of the system against camera viewpoint changes. However, we argue that this will not be a big problem for the proposed system, as all the rigid/affine motions are easy to eliminate by setting up reference points (e.g., players’ hips) after extracting the skeleton data for each player. Another challenge in a real-world application is introduced by camera shot transitions in music video post-production. One suggested strategy is to clear the accumulated association scores and re-register the players when a shot transition is detected. But further experiments need to be conducted to validate this strategy. Another limitation of the experiments is that all the players in the dataset have their front-right side facing the camera with most finger motion visible. If this is not satisfied in real scenarios, only the first computation module (correspondence between body motion and note onsets) provides useful information, making the system only work for string ensembles.