One Track per Person: the Kezek Prototype
Kezek takes a mono recording of exactly two people talking and returns one track per person: same length, same timing, overlaps split, and short replies like "yeah", "mm-hm" and "ага" kept on the track of the person who said them. This report covers the prototype. On clean conversations it meets every separation threshold we set in advance, and on the podcasts and recordings we listened to it does what we wanted. It also has three clear limits: noisy audio, very similar voices and speed. The prototype leans on two pretrained models we did not build, so it is not open source. It is now the reference that our next version, built only from our own components, has to beat.
| Separator | Overlaps, clean (SI-SDRi) | Short replies kept, clean | Leak onto the other track |
|---|---|---|---|
| 19.6M | +10.18 dB | 90.2% | −20.1 dB |
Summary
Full-duplex speech models such as Dauys learn to listen and speak at the same time, so they need conversations recorded with one channel per person. Most recorded speech is not like that: podcasts, interviews and calls are usually one mixed channel. Kezek exists to turn that mono audio into two-channel training data.
What we ask of that data, in order of importance:
- the two people are never swapped inside a recording;
- short replies stay on the track of the person who said them;
- one person's voice does not leak onto the other's track;
- solo speech is left untouched;
- overlaps are separated as cleanly as possible;
- it is fast enough for about 120,000 hours of audio.
| Requirement | Test | Prototype | Reading |
|---|---|---|---|
| No swaps | Open Yap test, stage 1 | 97.5% of recordings | Just below the 98% bar |
| No swaps over hours | 8 recordings, 1.6–2.9 h each | 1 bad minute of ~600 | Holds |
| No swaps, related voices | Two brothers, 22.6 s | swapped | Fails (section 06) |
| Short replies kept | Open Yap test, clean / as recorded | 90.2% / 87.3% | Met clean |
| No leak | Open Yap test, clean / as recorded | −20.1 / −17.1 dB | Met clean |
| Solo untouched | Open Yap test, clean / as recorded | 21.7 / 19.6 dB | Met clean |
| Overlaps separated | Open Yap test, clean / as recorded | +10.18 / +8.66 dB | Met clean |
| Speed | Whole pipeline, RTX PRO 6000 | 107 h / GPU-hour | Missed (target 300) |
All Open Yap test numbers come from one pass over 121 held-out conversations with one model. "As recorded" means the same conversations passed through a simulated recording chain: room reverberation, background noise, music, codecs and phone bandwidth.
Why it is a prototype. Two parts of stage 1 are pretrained models from others: WavLM Base+ for speech features and ECAPA-TDNN for voice fingerprints. The one real failure we found, on two related voices, traces back to ECAPA. The next version replaces both with components trained from scratch on our own data, and will be accepted only if it is no worse than the prototype on Open Yap test and by ear, and better on similar voices, noisy audio and speed.
01 · Listen
The Open Yap license does not allow us to publish its audio, so the test conversations behind the numbers below cannot be played here. The one recording we can share is our own: two brothers with related voices, the failure case in section 06. Partners in early access can hear Kezek on their own recordings.
What to listen for. Track A plus track B equals the input exactly, so nothing is thrown away and nothing is invented. Any background in the recording stays on the tracks, split by who is speaking. Short replies should sit on the track of whoever said them, even when they land in the middle of the other person's sentence.
02 · Architecture
Kezek has two stages. The first decides who speaks when and who is who, using the whole recording. The second only carries out that decision: it separates the voices and routes everything to the right track.
Figure 1. Stage 1 runs on 16 s windows and links them across the whole recording. Stage 2 runs on 8 s chunks with the decision of stage 1 as input.
We split the work this way after earlier single models failed in the same way. Given a voice sample and a short window, they also had to decide whether the target was speaking at all, and on real audio they silenced people or collapsed into silence. Taking that decision out of the separator removed the failure.
Stage 1: who speaks when
| Part | Configuration | Parameters |
|---|---|---|
| Speech features | WavLM Base+, frozen, softmax-weighted sum of 12 layers, 20 ms frames | ~95M, frozen |
| Encoder | Conv1d 768 → 256, Conformer × 4 (width 256, 4 heads, FFN 1,024, kernel 31) | 6.7M |
| Output | Powerset over {silence, slot 0, slot 1, both}, permutation-invariant per window | — |
| Window linking | Viterbi over all 16 s windows (hop 8 s): activity agreement in overlaps + similarity of slot embeddings to two recording-level centroids | — |
| Voice fingerprints | ECAPA-TDNN trained on VoxCeleb, frozen | ~20M, frozen |
| Profile heads (ens4) | Average of 4 heads in the style of TS-VAD: FiLM on profiles + Conformer, cosine to the profiles at several scales | — |
The output is the activity of A and of B over time, a 192-dimensional profile for each person, and a confidence for each window. Activity above 0.4 counts as speech.
Stage 2: separation
The separator, KezekTSE-L, is our own Conv-TasNet-style network in the time domain, enlarged and conditioned on what stage 1 decided. It carries over to the next version.
| Part | Configuration |
|---|---|
| Encoder / decoder | 512 filters, kernel 32, stride 16 |
| Separator | Bottleneck 256, hidden 768, 8 blocks × 3 repeats, kernel 3, dilations 1…128, gLN |
| Speaker condition | Profiles A and B, concatenated → 256, FiLM in every block |
| Activity hint | a_A and a_B as two channels, 1×1 conv into the bottleneck |
| Output | Two masks with a softmax across sources, shared decoder, then y_i += (x − y_A − y_B) / 2 |
| Size | 19.61M parameters |
The last step makes the two tracks sum exactly to the input. That is a property of the network, not a loss term, and it means Kezek cannot invent sound or drop it: whatever is in the recording ends up on one track or the other.
The separator runs over the whole recording in 8 s chunks with a 6 s hop. We also tried running it only where both people are active, plus one second on each side, and routing the rest by stage 1. That kept fewer short replies (85% against 90% on clean recordings), so we kept the full pass.
Training targets. The target for each track is that person's speech after the same recording chain as the input, plus the background weighted by who is active: w_i = a_i / (a_A + a_B), and 0.5 when neither is. During solo speech by A, the whole input is A's target. This is what stops the collapse into silence we saw before.
Loss. −SI-SDR on the active part of each track (capped at 30 dB), a level term, a separate −SI-SDR term on overlaps (weight 2), a multi-resolution STFT term normalized by the input, a level term on short replies, and a penalty for leaking onto the other person's solo (floor −30 dB).
03 · Data
| Set | What | Size | Used for |
|---|---|---|---|
| Open Yap 1K, The Agentic Data Company | Real two-person conversations, one headset channel per person | 1,580 conversations: 1,346 train, 113 val, 121 test | Stages 1 and 2 |
| Libri-Light medium | Read speech, 1,596 speakers (1,544 train, 40 val) | 173 h of 2–8 s pieces | Stage 2 pre-training |
| Libri-Light windows | 4 s windows: turns with 1–2.5 s overlap (60%), 1–2 short replies of 0.3–1 s (20%), one speaker (20%) | 200,000 | Stage 2 pre-training |
| Open Yap synthetic | The same generator on solo pieces of 157 Open Yap training speakers | 100,000 windows, 66 h of source | Stage 2 |
| Open Yap natural | Real windows from Open Yap conversations with word-level activity | 60,000 windows | Stage 2 |
Open Yap splits do not share people, and no test speaker contributes training pieces. Open Yap test was used once, for the final check.
A labeling fix that made stage 1 work. Open Yap gives each person a separate headset channel, and the first activity labels came from energy on each channel. A headset also picks up the other person, so the labels said the listener was talking when they were not: 29% of "active" frames had no words of their own. Gating the energy with each person's own word timings (±0.15 s, gaps under 0.3 s merged) fixed it. Given correct activity, stage 1 made no swaps at all, which told us the swaps had come from the labels.
The recording chain. Training inputs pass through a random chain of room reverberation, background noise, music, codecs and phone bandwidth, with some inputs left clean. The "as recorded" test uses the same kind of chain.
04 · Training the Separator
Each step below was one experiment with thresholds set before it started. The separator took five of them to reach its final form.
Figure 2. The separator as it changed, measured with the real stage 1 in front of it. Two attempts to improve on KZK-11 made it slightly worse.
| Step | What changed | Result |
|---|---|---|
| Pre-training (KZK-07, 08) | Medium model (13.3M) from scratch on Libri-Light overlaps, 50,000 steps | +9.7 dB on synthetic overlaps; +5.3 dB on Open Yap without seeing it |
| Open Yap (KZK-09) | Fine-tuned on Open Yap synthetic and natural windows, with Libri-Light replay | Pipeline +9.49 clean / +7.01 as recorded |
| Large from scratch (KZK-10) | 19.6M trained from scratch on the full mix | Far slower; stopped at 12,500 steps |
| Widened (KZK-11) | The medium model widened to 19.6M, then fine-tuned | +9.86 / +7.73, kept |
| Reply weight (KZK-12) | Short-reply loss × 3 | Replies and overlaps both slightly worse |
| Harder chain (KZK-18) | Training windows with heavy noise and reverb | No gain on noisy audio |
What we learned about training the separator:
- Pre-training on clean read speech transfers. After Libri-Light alone, the separator already reached +5.3 dB on real conversations it had never seen. We had planned a conversation simulator between the two stages and skipped it.
- Size helps only if you start from the smaller model. The large model from scratch learned much more slowly than the medium one. Widened from the medium weights by function-preserving expansion, it gained 0.4–0.7 dB.
- Short replies are not lost by the separator. Given the true activity instead of stage 1's, the separator keeps 91–92% of short replies. Weighting them more in the loss did not help, because the replies it misses are ones stage 1 never marked.
An earlier separator, a conditioned TF-GridNet, never started separating natural Open Yap overlaps (at most +0.4 dB). Natural overlaps are short and rare, so the learning signal was dominated by routing solo speech. Its full-resolution design was also far too slow for 120,000 hours.
05 · Results
The final check ran the whole pipeline once on the 121 held-out conversations of Open Yap test, which we had not used to choose anything.
Figure 3. On clean audio every separation threshold is met. With the recording chain, every one is missed, by 0.4 to 3 dB.
| Measure | Clean | As recorded | Threshold |
|---|---|---|---|
| Overlaps, SI-SDR improvement | +10.18 dB | +8.66 dB | ≥ +10 dB |
| Solo, level change | 0.00 dB | −0.08 dB | within ±1 dB |
| Solo, SI-SDR | 21.7 dB | 19.6 dB | ≥ 20 dB |
| Leak onto the other person's solo | −20.1 dB | −17.1 dB | ≤ −20 dB |
| Short replies (< 1 s) on own track | 0.902 | 0.873 | ≥ 0.90 |
Stage 1 on its own, on the same test: activity F1 0.943, short-reply recall 0.902, and 97.5% of recordings with no swap (the threshold is 98%). In a simulated room, 94.2%. On real two-person podcasts its diarization is at the level of pyannote.
Long recordings. On 8 conversations of 99 to 173 minutes, we found A and B swapped in 1 minute out of about 600.
By ear. We listened to podcast clips in English and Kazakh, 10 two-person recordings we made for testing, and Open Yap test conversations. Separation sounded as intended, with the recording background audible on both tracks. Two errors stood out, both on similar voices: one short reply on the wrong track in a Kazakh podcast, and the swap in section 06.
Not done yet. A comparison with DialogueSidon (DuplexChat: pyannote followed by a separator) on the same recordings.
06 · Similar Voices
We recorded two brothers on a USB microphone for 22.6 seconds. Their voices are related and close. The prototype swapped them: the first seconds of one brother went onto the other's track.
Where the swap comes from. We first suspected the window linking, since this recording is short and the brothers' turns are brief. We tried six ways of running stage 1 without retraining: 16 s and 8 s windows, the whole recording as one window, and one or two passes of the profile heads. In every one, the same span from 2.5 to 4.5 s went to the wrong brother.
The voice fingerprints explain it. ECAPA scores that span as closer to the other brother than to the speaker himself:
| Span 2.5–4.5 s compared with | Cosine similarity |
|---|---|
| The other brother, 7–10.5 s and 14–18 s | 0.24 |
| The same speaker, 5.5–7 s | 0.16 |
| The same speaker, 11–12 s | 0.11 |
| For reference: the other brother's pieces with each other | 0.61 |
Linking can only be as good as the fingerprints it links. ECAPA was trained on celebrity interviews and has not learned to separate two related voices from 2-second pieces. That makes the fingerprint the first part we replace.
07 · Where Short Replies Go
Short replies are the hardest requirement and the one Dauys cares most about: they are how a listener shows they are still there. We traced every reply in 40 validation conversations through stage 1 under each recording effect.
Figure 4. A reply is almost never given to the wrong person. When it is lost, it is missed, and noise and reverberation cause most of the misses.
We tried three fixes. None closed the gap:
| Attempt | Replies, as recorded | Reading |
|---|---|---|
| Stage 1 as trained, threshold 0.5 | 0.838 | Baseline |
| Fine-tune the 4 profile heads on a noisy chain (KZK-16) | unchanged to two decimals | The heads are not the bottleneck |
| Lower the activity threshold to 0.4 (KZK-17) | 0.857 | Small gain, leak 0.4 dB worse; kept |
| Heavier recording chain for the separator (KZK-18) | 0.825 | Worse |
What remains is the frozen WavLM features themselves in noise. A stage 1 trained from scratch on audio that has been through the recording chain is the way past this, and it is part of the next version.
08 · Speed
The whole pipeline processes 107 hours of audio per GPU-hour on an RTX PRO 6000. Our target is 300. At the current speed, 120,000 hours take about 1,120 GPU-hours.
Figure 5. No single part dominates. Reaching 300 means cutting the total from 34 s to 12 s per hour of audio.
The separator alone runs at 483 hours per GPU-hour. Stage 1 costs more than the separator: ECAPA embeddings of segments and the four profile heads, each with its own pass of the encoder, take 18 of the 34 seconds. Caching the frozen WavLM features shared by the heads raised speed from 94 to 107. Larger batches and several processes on one GPU did not help, because the GPU was already busy.
09 · What We Could Not Establish
- Noisy recordings. All four separation measures miss their thresholds with the recording chain.
- Similar voices at scale. We have two errors on similar voices and no measurement of how errors grow as voices get closer.
- Kazakh with references. We have listened to Kazakh podcasts and our own Kazakh recordings, but have no Kazakh measurements against reference tracks.
- More than two people, streaming, a user-supplied voice sample. The prototype does none of these by design.
- Variance. Every result is from one training run.
10 · What Comes Next
The next version keeps the separator and rebuilds stage 1 from our own parts:
- Our own voice fingerprint on raw audio, trained from scratch on about 7,400 Libri-Light speakers and the Open Yap training speakers, on 1–3 s pieces that have been through the recording chain, with extra weight on pairs of similar voices. This is the first step. It has to beat ECAPA on similar pairs and put the brothers' disputed span with the right brother.
- Our own "who speaks when" model on raw audio, trained from scratch, instead of WavLM features and four heads. It is meant to fix replies in noise and most of the speed gap at once.
- The separator fine-tuned to the new profiles.
The prototype stays as the bar. The new version is accepted only if it is no worse on Open Yap test and by ear, and better on similar voices, recorded audio and speed. Kezek is in early access with selected partners. Neither the prototype nor the new version is open source yet.
Appendix · Evidence and Reproducibility
- Experiments are numbered KZK-01 to KZK-26 in our internal research log. Each records its hypothesis, thresholds set before the run, data revisions and the commit it ran on.
- Thresholds were set before the experiments and did not change after them. SI-SDR is in dB; "short replies" are utterances under 1 s.
- Open Yap test was used once, for the final check, and never to choose a model or a threshold.
- Audio excerpt in section 06 is the first-pass output of the prototype on our own recording. Nothing was edited after separation.