Look Once to Hear: Target Speech Hearing with Noisy Examples

The quality of the target speech extracted by the target speech hearing network, \(\mathcal {T}\), has a critical dependence on the discriminative quality of the speaker representation, ϵ₀, provided to it. In order to robustly handle various speech characteristics, we leverage the speaker representations computed by large-scale pre-trained models such as [34, 65]. In this work, we use the open-source implementation of [65] in the Resemblyzer project [49]. Given a clean speech utterance of a speaker s_i(t′), [49] uses a long short-term memory (LSTM) network, \(\mathcal {D}\), to map the utterance to a unit length 256-dimensional vector \(\mathcal {D}(s_i(t^{\prime })) = \epsilon _i\), where ϵ_i ∈ R²⁵⁶ and ||ϵ_i||₂ = 1, referred to as a d-vector embedding. During the training phase, the LSTM model computes d-vectors optimized such that embedding corresponding to an utterance of a speaker is closest to the centroid of embeddings of all other utterances of the same speaker. This is done while simultaneously maximizing the distance from the centroids of all other speakers in the large-scale speech database used as the training set. In this work, we use d-vector embeddings as reference speaker representations that the noisy enrollment network \(\mathcal {N}\) should predict using two approaches.

Noisy enrollment with beamforming. Following the notation in §3, we note that the d-vector embedding of the target speaker can be obtained with its clean speech example as \(\epsilon _0 = \mathcal {D}(s_0(t^{\prime }))\). If we could estimate the clean speech of the target speaker, provided that the target speaker is present at the azimuthal angle \(\theta _0 \sim \frac{\pi }{2}\), we could estimate the d-vector embedding corresponding to the target speaker. Essentially, this is equivalent to beamforming with direction input steered towards the azimuthal angle equal to \(\frac{\pi }{2}\). In this work, we follow the delay and process approach proposed in several beamforming works [12, 30, 66], where given a target direction and a reference microphone, inputs from other microphones are delayed according to the time it takes for the direct path from the given direction to reach them relative to the reference microphone. In this case, since we assume the direct path is equidistant from both left and right microphones, processing the raw inputs is sufficient to obtain the target speaker. Assuming the beamforming network is represented as \(\mathcal {B}\), the process of noisy enrollment with beamforming could be written as:

In this work we use the state-of-the-art speech separation architecture TFGridNet [68] as our beamforming architecture \(\mathcal {B}\). Since enrollment is a one-time operation that does not need to be performed on-device, we could use the original non-causal implementation of the TFGridNet [68] available in the ESPNet[38] framework. Following the notation in [68], we used the configuration: D = 64, B = 3, H = 64, I = 4, J = 1, L = 4 and E = 8 with short-time fourier transform (STFT) window size set to 128 and hop size set to 64.

Noisy enrollment with knowledge distillation. Conversely, we could consider this problem as the noisy enrollment network, \(\mathcal {N}\), directly computing the estimated d-vector embedding of the target speaker, \(\hat{\epsilon }_0\), given the noisy speech. This would however require us to use a resource-intensive training process like the one proposed in [65]. To do this efficiently, we train the enrollment network \(\mathcal {N}\), using knowledge distillation [5, 25], where the original d-vector model, \(\mathcal {D}\), provides d-vector embeddings computed on clean target speech as ground-truth references. We note that during the training phase, we have access to clean target enrollment speech s₀(t′), but we do not assume this during inference. Here, we train the noisy enrollment network \(\mathcal {N}\) to minimize the loss function \(\mathcal {L}(\hat{\epsilon }_0,{\epsilon }_0)\):

To make both our noisy enrollment approaches comparable, we use TFGridNet [68] with the same configuration as above, as the noisy enrollment network \(\mathcal {N}\), in this approach as well. We modify the architecture to output 256-dimensional embedding instead of an audio waveform, as shown in Fig. 2b

Now that we have the embeddings for the target speaker, which captures the desired speech traits, our goal is to design a network that can perform target speaker hearing in a way that is both real-time on an embedded CPU and that achieves an end-to-end latency of less than 20 ms. This end-to-end latency is measured as the amount of time for a single sample to pass from the microphone input buffer, through our target speech extraction framework, and then copied into the headphone speaker output buffer, as shown in Fig. 3(a). We process the input audio in chunks of 8 ms. Moreover, our system utilizes an additional 4 ms of future audio samples to predict the processed output for the current 8 ms chunk. In other words, we must wait for at least 12 ms before we can begin processing the first sample in a particular chunk. Of course, this also means that our algorithm must be designed in such a way that the first sample in the audio chunk does not take into account any information beyond 12 ms into the future, as this information will not be available in practice.

We design our target speaker hearing network by starting with a state-of-the art speech separation network, namely TFGridNet [68]. However, as this network is non-causal, we adapt the implementation of TFGridNet into a causal version with an algorithmic latency of only 12 ms. To do this, we first remove the group normalization after the first 2D convolution. We then replace the bidirectional sub-band inter-frame LSTM block with a unidirectional LSTM, and fix the unfolding kernel and stride sizes, respectively the hyper-parameters I and J, in both recurrent modules to 1. Additionally, instead of computing causal attention using causal masks, inspired by prior work in [63], we first unfold the key and value tensors into independent fixed-size chunks using a kernel size of 50 and a stride of 1. We then compute an attention matrix for every chunk between the key tensor and a single-frame query tensor corresponding to the last (rightmost) frame in the chunk, as illustrated in Fig. 3(b). This attention matrix contains the multiplicative weights of the corresponding frames in the value tensor used to obtain the final output. This ensures that when we predict the output for a single frame, we only attend to the 50 frames that arrive with or before it. Although this limits the duration of time the attention layer looks into the past, this on-device approach is necessary to allow the network to effectively process larger time sequences. We choose an STFT window length of 196 samples, or 12 ms at 16 kHz, and a hop length of 128 samples, or 8 ms at 16 kHz. When computing the ISTFT, we trim the last 4 ms of audio, as these samples will be affected by future chunks during the overlap-and-add operation of the ISTFT. Thus, our resulting output would be 4 ms shorter than the input, which means we obtain an 8 ms output from a 12 ms input. For the TSH model, we use the asteroid  [47] implementation of STFT.

Once we copy a chunk from the audio buffer into memory, we can begin processing the audio. Since we process audio 8 ms at a time, we need to ensure that each chunk is processed in at most 8 ms on an embedded device, or else incoming chunks begin to queue up, causing the processed output audio to be increasingly delayed. This constraint thus requires us to employ a very stringent processing algorithm to ensure we can keep up with the incoming audio stream. As the original TFGridNet could not meet these runtime requirements on our embedded CPU, we therefore needed to optimize the above model in various ways, we describe below, to minimize the inference time.

Caching intermediate outputs. When processing a stream of continuous chunks, there are numerous values that can be reused to avoid recomputing them. We maintain these values as a list of model state buffers that we pass as an input to the model in addition to the input signal and the target embedding at every inference. For example, we can avoid recomputing the STFT frame of prior chunks by caching these values and re-using them when computing the output of the first 2D convolution layer, as shown in Fig. 3(c). Likewise, we can store the output of the sequence of GridNet blocks from previous chunks and use them to compute the 2D deconvolution quicker. Additionally, as computing the ISTFT for the current chunk also uses information from previous chunks, we also need to maintain a buffer for the intermediate outputs of this 2D deconvolution layer. Furthermore, we maintain the hidden and cell states of the temporal unidirectional LSTM for every GridNet block we have. This allows us to truly make use of the long-term receptive field of the recurrent network. Finally, for every GridNet block, we also maintain the state buffers for previous values of the key and value tensors and concatenate them before unfolding (Fig. 3(b)). We use these buffers to avoid recomputing the linear projections of previous frames.

ONNX-specific optimizations. Our goal is to deploy network using ONNX Runtime [14]. To do this, we re-write certain parts of the network to be more suitable for this setup. First, since we set both I and J to 1, we effectively remove the need for the unfolding layers before the recurrent intra-frame and sub-band modules. Additionally, we can replace all convolution and deconvolution layers, which now have a fixed kernel size and stride of 1, with linear layers that are converted to simpler matrix multiplication kernels when converting to ONNX. We also modify the layer normalization modules to use the native PyTorch implementation, which newer ONNX converters can readily fuse into a single kernel, reducing the overhead of multiple kernel calls. Finally, we rewrite the multi-head attention layer, which was implemented as a for-loop to compute the key, query and value tensors for each head, as a single block for each of these tensors, and reshape the output appropriately. This reduces the overall number of nodes in the ONNX graph.

Reducing model size. Instead of using the hyper-parameters suggested in [68], we choose a hyper-parameter setting that produces a smaller, faster model. In this work, we use D = 64, B = 3, H = 64, I = 1, J = 1, L = 4 and E = 6. The resulting model has a total of 2.04 million parameters.

To condition the network with the speaker embedding obtained during enrollment, we use a simple linear layer followed by layer normalization to compute a common 64 × 97 conditioning vector for all time chunks, which we multiply with the latent audio representation between the first and second GridNet blocks.

We train our target speech hearing system in two steps. We first train the enrollment networks to estimate d-vector embeddings. We then separately train the target speech hearing model while conditioning it with reference d-vector vector embeddings. This approach allows us to use the same target speech hearing model with any enrollment model that can estimate d-vector embeddings. We train these models with a training dataset that considers an accurate representation of real-world use cases of a target speech hearing system. Specifically, we consider variations in speech characteristics, acoustic transformations caused by physical multipath environments, acoustic transformations caused by the human head related transfer function (HRTF) and diverse background noise. We also consider the effects caused by motion of the speaker and noise with respect to the listener as an additional finetuning step. Below we explain the dataset details followed by the training process of the enrollment and target speech hearing networks.

Synthetic dataset. Each training sample in our dataset corresponds to an acoustic scene comprised of 2-3 speech samples and background noise. To create an acoustic scene, we first sample a 5 second background noise sample and then overlay the target speech and inferring speech at random start positions. For obtaining target and interfering speech, we randomly select 2-3 speakers from the LibriSpeech dataset [46], and select a speech sample of length 2-5s for each speaker. The enrollment signals are generated using the same approach as well. We used the train-clean-360 component of LibriSpeech dataset that comprises 360 hours of clean speech with 439 and 482 different female and male speakers, respectively. We further select random noise samples from WHAMR! dataset [69] comprising a database of audio samples of real-world noisy environments. These audio samples, however, do not contain the effects of real-world indoor environments and human heads, which we found is important for extracting natural-sounding audio.

Accounting for multipath and HRTF. To account for these effects, we convolve each of the speech samples and background noise with a binaural room-impulse-response (BRIR) that captures the acoustic transformations caused by a room as well as a user’s head and torso. Let h_{r, θ, ϕ} be a BRIR corresponding to the room and subject combination r, at azimuthal angle θ and polar angle ϕ with respect to the subject’s head. Let S₀(t) ∈ R and S₁(t) ∈ R be two mono clean speech mixtures sampled from the LibriSpeech dataset, and V(t) be noise sampled from the WHAMR! dataset. Then the binaural acoustic scene x(t) ∈ R² for this source mixture could be computed as:

It is to be noted that in each acoustic scene, room and subject configuration r, remains the same for all sources, but the angles with respect to the listener are arbitrary. To improve robustness to variations in rooms and subject, we aggregate BRIRs from 4 different datasets: CIPIC [3], RRBRIR [28], ASH-Listening-Set [58] and CATTRIR [29]. Of these, CIPIC dataset only comprises of impulse responses measured in an anechoic chamber and as a result, is devoid of any room characteristics. Combined, these datasets provided us with a total of 77 different room and subject configurations.

Training. To train the enrollment networks, we first generate the component speech utterances, as described above, with the constraint that target speaker’s azimuthal angle \(\theta _{0} \sim \frac{\pi }{2}\). We train the beamformer-based enrollment network to predict target speech (Fig. 2a) with SNR loss. We train knowledge-distillation-based enrollment network to predict the d-vector embedding of the target speech (Fig. 2b) with cosine-similarity loss.

To train the target speech hearing (TSH) network, we also sample a random speech corresponding to the target speaker and convolve it with a BRIR corresponding to the same room and subject configuration. We input the TSH model with the acoustic scene and the d-vector embedding computing on the sample speech. We then optimize the TSH network, \(\mathcal {T}\), to minimize the signal-to-noise ratio [54] (SNR) loss between the estimated target speech and the ground-truth: -SNR(\(\hat{s}_0(t), s_0(t)\)).

Finetuning for motion, error in the enrollment angle and real-world noise characteristics. In the dataset setup described above, we assumed a constant azimuthal angle for each source as time progressed. This means that sources are stationary with respect to the listener’s orientation, and the enrollment angle is close to \(\frac{\pi }{2}\) and does not change with time. These assumptions, however, are not true in the real world as sources could be moving, or there might be a rotation in the listener’s head resulting in significant relative angular velocities.

We handle relative motion and time-varying error in the enrollment angle with an additional finetuning step. During finetuning, we make the azimuthal and polar angle time-varying. We simulate motion by generating an array of positions over time with a finite time step of 25 ms. For enrollment, we assume that at each time step, both the enrollment azimuth and enrollment polar angles are uniformly random and lie in the range \([\frac{\pi }{2} - \frac{\pi }{10}, \frac{\pi }{2} + \frac{\pi }{10}]\), accounting for a maximum error of 18 degrees. For the rest of the sources – interfering sources in the enrollment acoustic scene, and interfering as well as target sources in the input to the TSH model – we generate a random speaker motion by randomly triggering speaker motion events at each time step with a probability of 0.025. When a speaker motion event is triggered, we sample a pair of angular velocities along the polar and azimuthal directions with magnitudes uniformly distributed in the range \([\frac{\pi }{6}, \frac{\pi }{2}]\) rad/s. The speaker moves along this direction for a random duration uniformly sampled from [0.1, 1] s. During this time, we do not consider any other motion events. This creates trajectories where the speaker may be stationary for some time intervals and sporadically move with different velocities within the same audio clip. Assuming such time-varying trajectories, the computation of an enrollment scene could be written as: \(e(t^{\prime }) = S_{0}(t^{\prime }) * h_{r, \theta _{0}(t^{\prime }),\phi _{0}(t^{\prime })} + S_{e1}(t^{\prime }) * h_{r, \theta _{e1}(t^{\prime }),\phi _{e1}(t^{\prime })} + V_e(t^{\prime }) * h_{r, \theta _{ve}(t^{\prime }),\phi _{ve}(t^{\prime })}\), where, \(\theta _{0}(t^{\prime }), \phi _{0}(t^{\prime }) \in [\frac{\pi }{2} - \frac{\pi }{10}, \frac{\pi }{2} + \frac{\pi }{10}]\). And the input scene can be computed as:

Since the BRIR datasets typically provide an impulse response at discrete points in space, it is not possible to directly perform the computation described in the expressions above. To achieve such a trajectory simulation with available BRIR datasets, we employ a nearest neighbor approximation of BRIRs – we select BRIR in the dataset that is closest to the desired azimuth and polar angle in time. We use Steam Audio SDK [57] to perform this motion trajectory simulation. During the finetuning step, we use all four BRIR datasets we described above, but perform motion simulation only with CIPIC database [3]. We do this because CIPIC provides BRIRs that are reasonably dense across all azimuth and polar angles, while the rest of BRIR datasets sparsely vary azimuthal angle only with fixed elevation.

Finally, to allow the model to learn common noise characteristics found in the real world such as microphone thermal noise and constant noise from heating, ventilation and air conditioning systems, we also train the model with randomly scaled white, pink and brown noise components. Specifically, during training, we augment the mixture signal with a white noise signal having a standard deviation uniformly chosen from the range [0, 0.002). As for the pink and brown noise, we use the powerlaw_psd_gaussian function from the Python colorednoise library to generate these noise signals. We scale the pink and brown noise signals with different scale factors each sampled uniformly from [0, 0.05).

Hardware prototype. We design our hardware using the Sony WH-1000XM4 noise-cancelling headset. Since we need binaural acoustic data, we attach a pair of binaural microphones (Sonic Presence SP15C) to the exterior side of the earcups of the noise-canceling headset as shown in Fig. 4(a). The incoming binaural audio is processed on an embedded CPU and is played back to the user using the noise-canceling headset. Fig. 4(b) shows the attenuation achieved by the noise-canceling headsets across different frequencies, with and without active noise cancellation. In these experiments, we obtain recordings from a pair of microphones fitted into a human user’s ears as a nearby loudspeaker plays a 20 Hz-22 kHz linear frequency sweep. While wearing the headphones, we notice that the earcups cause the microphone to pick up an audio signal that synchronizes with the wearer’s blood pulse, which produce the spurious peaks at the lower frequencies (<100 Hz).

Runtime evaluation. We connect the binaural microphones to an Orange Pi 5B using a USB cable. This allows the Orange Pi to read audio chunks from the microphones, which we process on-device and play back to the headset connected via the audio jack. We deploy our neural network on the embedded device by converting the PyTorch model into an ONNX model using a nightly PyTorch version (2.1.0.dev20230713+cu118) and we use the python package onnxsim to simplify the resulting ONNX model. We run inference using the ONNX Runtime version 1.16.0. To evaluate our model runtime, we use the ONNX Runtime perftest tool to run 1000 successive inference operations with the model, and we observe that the model inference time, which is 5.47 ms on average, is lower than our chunk size of 8 ms. This means that our model can process an audio chunk before the next chunk arrives, and hence, it satisfies our real-time requirement.

In practice, since our model makes significant use of cached values, which we must provide as an input every time we run inference, we must also take into account the time it takes to copy these updated cache buffers from the model output to the model input. When we include these buffer copy operations in our runtime measurements, we see that although the overall runtime increases, it is still within the constraints for real-time operation with a runtime of around 6.24 ms, as shown in Fig. 4(c). This gives us a total end-to-end latency of 18.24 ms.

End-to-end demonstration. We demonstrate our end-to-end system and ask participants to wear and use it. We enable active noise canceling on the headphones and run our target speaker hearing pipeline after the user enrolls their target speaker. For these experiments, we also place an additional pair of microphones inside the headphone’s earcups to record the exact sound that the user will listen to during operation. This includes any residual noise from the imperfect noise cancellation, as well as the amplification of target speakers from our target speaker hearing playback signal. We show the resulting spectrogram representation for two examples in Fig. 5. For brevity, we only show one audio channel.

In the first example, the user is trying to listen to a target human speaker in the presence of airplane engine noise playing over a nearby smartphone speaker. This noise is clearly visible in Fig. 5(a), which was recorded from the outer microphone. However, when we probe the sound at the inner microphone (Fig. 5(b)), we noticed a significant attenuation of this background noise due to the headphones’ active noise-canceling abilities. And yet, because of our target speaker hearing system, the target speaker is still very clearly audible at the inner microphone. We show in the second example that this behavior can even be seen for other, more dynamic and loud background sounds. Specifically, we show in Fig. 5(c) that even when the user tries to listen to the target speaker in the presence of music, in this case a loud pop song, our end-to-end system have an impressive ability to suppress the unwanted music and the singer while preserving the sound of the target speaker (Fig. 5(d)).

We evaluate our hardware in previously unseen indoor and outdoor environments, with participants who are not in the training data. We recruited 8 individuals (5 male, 3 female) to collect data in different in-the-wild scenarios using our hardware. We ask 3 participants at a time to collect noisy enrollment signals, as well as noisy real-world mixture audio, in different acoustic environments while they read random text. Among the three participants, one of them is designated as the wearer, while the other two are the speakers. To collect a noisy enrollment for a given target, the wearer looks at the target speaker as they read a text. This mimics the "Look Once" phase in the real-world use, where the listener would look at the target speaker. As the target speaker reads the text, background sounds and, in all but one case which had significant noise, speech from the other speaker make this enrollment signal noisy.

We record multiple noisy binaural audio clips while the target speaker reads a different text in the presence of other environmental sounds and speakers. Unlike the noisy enrollment signal, there is little control over these recordings, as the wearer and target speaker are free to move around and/or rotate their head (Fig. 6 A-C). These recordings were also collected in different acoustic environments, including living spaces, busy streets, and in nature (Fig. 6 D-F). They also contained settings where the listeners were in different postures, such as standing, sitting and laying down (Fig. 6 G-I).

Evaluation procedure. Since the target speaker is speaking in the presence of interfering speakers and unknown noise, it is difficult to obtain the ground truth audio signal for our target speakers in the real world. So, we cannot rely on objective metrics to evaluate the system performance. Instead, we design a listening survey to allow human participants to rate the performance of our two enrollment methods on 15 different target speaker scenarios from the in-the-wild dataset we collected. To do this, we recruited 21 participants (13 male and 8 female with an average age of 30.4 years) to take our survey and give their opinion on our target speaker hearing system to obtain a mean opinion score (MOS). To do this, for each scenario, we first ask the users to try listening to a 5-second clean signal of the target speaker reading text collected in a quiet room. We then ask the participants to listen to the target speaker in 3 distinct audio clips: 1) the original, noisy recording of the target speech with interferers, 2) the output of our target speaker hearing network using the noisy knowledge distillation embeddings, and 3) the output of our target speaker hearing network using the noisy beamformer embeddings. The 3 clips are presented in random order.

After listening to the mixture and model output clips in a random order, we ask the participants to rate the target speech extraction quality by asking them the following questions:

Results. Fig. 7 shows that our system can greatly suppress the background sounds and interfering speakers, as evidenced by the fact that that our beamforming and knowledge distillation enrollment networks increased the mean opinion score for the noise suppression task from 1.71 to 3.05 for the beamformer enrollment method and 3.28 for the knowledge distillation method (Fig. 7(a)). Our target speaker hearing framework was also able to improve the overall mean opinion score from 2.09 to 3.18 and 3.4, respectively (Fig. 7(b)).

The results show a consistency between our objective benchmark results in §4.4 and real human evaluators, in that the embedding network trained with knowledge distillation outperforms the beamforming network in both cases. These evaluations also show that our system can generalize well to real-world environments with real human wearers. One particularly important scenario is to be able to adapt to sudden and rapid changes in the target speaker’s position due to the wearer rotating their head. Indeed, many of the participants in our study often rotated their heads, looking at different speakers and objects in their surroundings. As a result, vital spatial cues could vary widely with time. Such scenarios highlight the importance of fine-tuning our neural networks with audio samples that contain moving speakers. For example, in Fig. 8, we show a real-world example where the listener was noticeably rotating their head while recording throughout the highlighted region. This can be clearly seen by observing the level differences between the left and right channels, as the target speaker becomes more prominent in the left channel in that time frame. As a result, the target speaker appears to move relative to the listener. Without fine-tuning, our network incorrectly suppresses the target speaker during this time. However, when we fine-tune the network with our mobility technique in §3.3, we see that it can correctly pick the target speaker even when the user turns their head.

To further examine this behavior, we conduct an experiment with motion in the real world. Our goal here is to evaluate the model’s ability to adapt to the wearer’s head motion. We play 6 seconds of speech from a loudspeaker and record audio from a participant wearing our system in an office room with HVAC and other ambient noise. As the speech is played, the participant rotates their head several times per trial with different speeds, pausing briefly between successive rotations. We obtain recordings where the wearer rotates exclusively in the azimuthal direction, and recordings where they rotate exclusively the polar direction. We use a gyroscope to record the angular velocity during each trial. To evaluate the model, we use a different recording of this same speaker as an embedding signal and process the recorded audio. Since there is no other interfering speaker in this experiment (computing SI-SNR in real-world mixtures is challenging with interference), a well-trained model is expected to preserve the original speech regardless of the user rotation. As we see in Fig. 9a 9b). This is likely because when the wearer moves their head vertically, the position of the microphones do not change considerably. This analysis suggests that the model is leveraging inter-channel differences. Therefore, it is important to use moving sources during training for the system to better adapt to real-world use cases.

We investigate two main questions: 1) What interface should the user interact with when they want to enroll the target speaker?, and 2) what enrollment duration do users find acceptable for such a system? As shown in Fig. 10, we integrate into our prototype with three different interfaces through which users can communicate their intention of enrolling a target speaker: 1) a virtual button on a smartphone application, 2) a push button on the headphone, and 3) a touch pad on the headphone. We evaluate four different possible enrollment durations: 2.5 s, 5 s, 7.5 s and 10 s.

Comparing user interfaces. We conducted a user study with 9 participants, where each participant first wore our device and sat on a chair as shown in Fig. 10(d). We placed a loudspeaker to the side of the wearer, which played a mixture of human speech from the LibriSpeech dataset and generic vacuum noise. A person sitting in front of the participant would then read a random text. We then asked the participants to use each of the three interfaces to signal to the system their intention to enroll the target speaker in front of them, while suppressing the interfering sounds emitted by the loudspeaker. When the users correctly interacted with the device to start enrollment, the headset would play a voice saying "Enrollment start". While enrolling the target speaker, the participants are asked to keep their head facing the target speaker, until the enrollment duration has passed and the enrollment is completely recorded, at which point a voice is played over the headphones, saying "Finished". The enrollment duration for all three interfaces was set to 5 seconds. After interacting with the three interfaces, we asked each participant to rate the three interfaces from 1-5 based on how likely they are to use that interface for this interaction. The results are shown in Fig. 11(a). Most participants showed a strong preference for the push button because of its good haptic feedback. All the participants showed the least preference for the smartphone since it required the extra complexity of looking at the smartphone screen while simultaneously trying to face the target speaker.

Evaluating the enrollment duration. We then asked the participants to use their favorite interface from the previous study to explore their perspectives on a reasonable enrollment duration. Specifically, each participant performed enrollments with 4 different durations: 2.5s, 5s, 7.5s and 10s. The results in Fig. 11(b) show that 89% of the participants thought that 5 seconds was an acceptable duration for the enrollment period.

Qualitative Results. Next, we asked each of the participants to fill out a System Usability Scale (SUS) questionnaire, which was developed in [10]. The overall score was 80.8 ± 16.7, which suggests generally positive feedback on usability. The SUS results for each question are shown in Fig. 11(c), which correlates with being highly usable and acceptable by users, according to Bangor’s empirical evaluation [6]. Finally, in addition to the previous studies, we also ask the subjective question: "Where do you see yourself using such a system?". Five of the participants mentioned using them in crowded scenarios and expressed similar applications to the response of one of the participants: "I’d like to use it in large social gatherings like conferences and lectures. I want to just talk with a specific people without being distracted by others or loud background noise". Five of the participants also mentioned that they were willing to use it in common public locations such as cafes, restaurants, on the street, karaoke and in large parties. Furthermore, one participant proposed that this technology might be useful for hearing aids. While all participants gave positive feedback and proposed useful potential applications, two of them also raised some limitations. One participant said, "In the real-world, I would also want to focus on a group of people instead of only one person". Another participant said "I think the headphone form factor is a bit obtrusive. A wireless earbud form-factor would be more socially acceptable."

We quantitatively compare our two noisy enrollment methods described in §3.1. We also compare the quality of target speech with that extracted with embeddings computed on clean enrollments.

For the real-time target speech-hearing aspect of the system, we consider three models: our embedded real-time implementation of TFGridNet [68], Waveformer [63] and DCCRN [26]. These have been developed for speech separation, target sound extraction and speech enhancement, respectively. Waveformer and DCCRN are causal real-time models suitable for this task. We use network variants with the default algorithmic latency proposed in the respective architectures of 13 ms and 37.5 ms for Waveformer and DCCRN respectively, while for our causal TFGridNet implementation, we use an algorithmic latency of 12 ms.

This results in a total of nine combinations – 3 enrollment methods with 3 TSH backbones. We evaluate all nine combinations with 10000 binaural mixture/noisy enrollment pairs generated as described in §3.3. Table 1 reports the average target speaker’s signal quality improvement with respect to the mixture in terms of scale-invariant signal-to-noise ratio improvement (SI-SNRi). Assuming speaker embeddings computed from clean enrollment signals as a reference, we also report the cosine similarity between reference embeddings and embedding computed by both noisy enrollment methods in the second column. The last two columns show the size of the architecture in terms of parameter count as well as multiply-and-accumulate per second (MACs/s). Among these architectures, we find that TFGridNet offers the best performance while also consuming sufficiently low operations and latency.

In addition, since the original TFGridNet is not designed for streaming applications, we re-implemented it as a cached, streaming model without any other optimizations and compare our final model’s inference time with this streaming TFGridNet implementation running on PyTorch and ONNX Runtime. Table. 2 shows that executing inference on the Orange Pi is significantly faster if we use ONNX Runtime, and when we further optimize the model as described in  §3.2, we can further achieve a 41.1% reduction in the inference time with only a 1.5% reduction in the SI-SNRi.

In Fig. 12a

To understand this disparity, in Fig. 12b 12c x = y line resulted in a positive improvement. Further, in Fig. 12d

Finally, in Fig. 13a