It Listens, Not Yet for Meaning: Dauys Stage 1

Dauys is a compact speech model that listens and speaks at the same time, without ever seeing text. Stage 1 trained a 61-million-parameter hybrid from scratch on two-channel conversations and asked a simple question: does it learn to listen? It does, but mostly for timing, not yet for meaning. This is the record of how we got there, including the negative runs.

Parameters	Training speech	Held-out conversations	Input dependence
60.9M	61h	507	507 / 507

---

Summary

What Stage 1 established:

• The full pipeline works end to end. It covers two-stream causal training, streaming inference with continuous listening, and a frozen neural codec. The model reproduced ten complete training conversations exactly, matching all 43,312 audio tokens.
• The model uses the incoming audio. On 507 held-out conversations, replacing the user stream with silence made the recorded answer less likely in 507 of 507 cases (+1.72 nats/frame on average). Replacing it with another conversation in the same voice also made it less likely in 507 of 507 cases (+0.77).
• Free-running speech is partly intelligible. In informal listening, the owner judged roughly 45–70% of the words understandable, with a robotic texture. The model gets the timing of its speech broadly right: free-running speech-activity F1 is about 0.8.

What Stage 1 did not establish:

• Relevant answers. On held-out conversations the model's replies are generic or about a different topic. The dependence on the input is mostly about timing and activity, not content.
• Generalization of the coarse audio code. The first Mimi codebook (CB0) is largely memorized: 95% accuracy on training data versus 70% on validation. The finer acoustic codebooks generalize well.
• Live conversation. Every behavioural test so far replays a pre-recorded user. Such a user cannot react to what the model actually says.

Stage 1 closes with Hybrid v0.1 as a measured baseline (checkpoint R0) and a concrete diagnosis that shapes the Hybrid v0.2 plan.

01 · Model and Data

Hybrid v0.1

Component	Configuration
Audio codec	Pretrained Mimi, frozen. 24 kHz mono per channel, 8 codebooks × 2,048 values, 12.5 frames/s (80 ms per frame)
Channels	A = user, B = model. Both share one continuous timeline, including silence and overlap
Frame embedding	One table per codebook (2,048 × 256), shared by both channels. Eight embeddings summed per frame, plus a role embedding
Listening tower	4 causal Mamba-2 layers, width 384, reads the user stream
Dialogue tower	10 blocks, width 640: 6 Mamba-2 and 4 Transformer blocks with causal self-attention and cross-attention to the listening tower (512-frame window)
Output	CB0 head, plus a 2-layer depth Transformer for CB1–CB7 within the frame and a binary speech-activity head
Size	60,873,210 trainable parameters (cap: 65M)

At frame t the model has heard user frames up to t and its own frames up to t − 1, and predicts its frame t. Recurrent state persists across the whole conversation, so listening never stops while the model speaks.

Data

All dialogues are synthetic English two-party conversations from the hohere pipeline. An LLM writes each script, and Kokoro TTS renders it with timing that includes pauses, backchannels, interruptions and overlap. Each conversation has a single fixed assistant voice and one of several user voices.

Experiments	Corpus	Size
S1-E001 – S1-E004	hohere-dataset v0.1	131 conversations, 3.1 h
S1-E005	hohere-1m	3,908 conversations, ≈ 85 h before selection

For S1-E005, train, validation and test were separated by scenario family: a whole topic is held out, not just a conversation. The final split was 2,788 / 330 / 507 conversations, or 60.8 / 7.1 / 11.2 hours, across 47 / 5 / 8 families. The test set was opened only once, after the checkpoint had been chosen.

02 · The Path Through Stage 1

Stage 1 moved from "can it memorize one clip?" to "does it generalize?". Several runs along the way were negative. They are kept here because they explain the final design.

Run	Question	Result
E001	Memorize one 20 s excerpt	Low teacher-forced loss, but free-running output was silent (RMS 0.0002 vs 0.025)
E002	Train on the model's own generated history	Speech timing matched (activity F1 1.0). A generator bug later found here masked the exact result (see below)
E003	Does the output depend on the user input?	On the training clip: silence raised NLL from 0.05 to 2.28, an unrelated user from 0.05 to 2.53
E004, E004B	16 conversations (25 min)	Activity looked fine (F1 ≈ 0.95), but listening found no intelligible speech. Negative
E005 (first try)	Continue E004B on 100 min	Stopped at step 222: no point scaling without an audible overfit
E002R / E002S	Re-audit with a fixed generator	2,048 / 2,048 tokens exact, decoded audio byte-identical to the codec reference
E003-10	Memorize ten complete conversations (7.2 min)	43,312 / 43,312 tokens exact across all ten
E003-11	Same model on an unseen conversation	Audible but irrelevant: a farmers-market question got an electricity-bill answer from training
E004-GEN	69 conversations (100 min), held-out families	Validation NLL rose after step 1,000 (overfitting). Input controls positive on 8 / 8 test cases
E005	61 h, family-disjoint, early stopping	Baseline R0, sections 3–5

The generator bug. The early generator replaced the predicted codes with a fixed "silence" code whenever the activity head predicted silence. Only 50 of 256 frames were then exact, although every speech frame was correct. Without this override, the same weights reproduced all 256 frames. The fix removed a false negative. It also set a rule for later runs: silence is represented by real codec tokens, never by a substituted index.

Lessons that shaped E005:

1. Teacher-forced loss alone is not evidence of speech. Every run since E002 checks free-running output and listens to it.
2. Training on the model's own history matters. E005 adds short own-history rollouts from the third epoch onward.
3. Memorizing a few conversations is easy. The open question is generalization, so E005 used about 36 times more data than E004-GEN, split by topic, with early stopping.

03 · S1-E005: The Stage 1 Baseline

E005 trained Hybrid v0.1 from scratch on 61 hours, one complete conversation at a time, with about 2,048 valid frames per optimizer batch. The run stopped by rule at step 7,292 (about 5.5 epochs), after three validation checks without improvement. The selected checkpoint is step 5,306, the one with the lowest validation NLL (14.98 nats/frame). This checkpoint is R0, the reference for everything that follows.

Figure 1. Left: training loss per step and its 100-step mean. The visible drops coincide with epoch boundaries (≈ 1,330 steps each); own-history rollouts begin at step 2,668. Right: validation NLL at each evaluation. It flattens after step 4,000 and is lowest at step 5,306.

Total validation NLL hides a split between the codebooks. The finer acoustic codebooks kept improving until the end, while the coarse code CB0 was best at step 2,667 and then got steadily worse:

Figure 2. Validation NLL per codebook group. CB1–CB7, the acoustic detail, improves throughout. CB0, the coarse content-bearing code, starts overfitting after step 2,667.

04 · Does the Model Listen?

For every one of the 507 test conversations, we measured how likely the recorded assistant answer is under three versions of the user stream. The model's output history stays fixed, so only the input changes:

• the original user audio;
• silence;
• a different conversation in the same user voice and of similar length.

Test set, 507 conversations	Original	Silent	Same-voice shuffled
NLL of the recorded answer (nats/frame)	14.90	16.62	15.66
Increase over original	—	+1.72	+0.77
Conversations where original is best	—	507 / 507	507 / 507

Figure 3. Per-conversation increase in NLL when the user stream is replaced. Every conversation is above zero in both controls.

Two more checks point the same way. Removing the cross-attention path from listening to dialogue raised validation NLL by 0.19. It changed about half of the free-running output tokens and made the decoded output nearly silent (RMS ≈ 0.0002). The model does not merely tolerate the input: it needs the input to decide when to speak.

Caution. The silent control costs more than twice as much as the same-voice shuffle. A shuffled conversation keeps speech present but changes its timing and content together. The comparison therefore shows that the model uses the incoming stream, mostly for when to speak. It does not show that it understands what was said.

05 · What Limits the Baseline

The coarse code is memorized

Teacher-forced, diagnostic subset	Train	Validation	Gap
CB0 accuracy, R0	95.2%	69.9%	25 points
CB0 NLL, R0	0.23	1.47	1.24
Mean CB1–CB7 NLL, R0	1.94	2.09	0.15

Figure 4. The train–validation gap is concentrated at CB0. Training further, from step 5,306 to 7,292, improves CB0 on train (0.23 → 0.12) and worsens it on validation (1.47 → 1.64).

A separate probe gave the model the correct CB0 token for each frame and let it generate the rest. Output energy then rose from 0.013 to 0.023 RMS, close to the reference 0.025. The acoustic decoder is not the bottleneck. Choosing the right coarse code is.

The codec is not the ceiling

Re-encoding the target audio with Mimi reproduces the cached tokens exactly, with a waveform correlation of 0.81 and 92% of the source RMS. The codec preserves the speech well enough. Remaining quality problems come from the model.

Decoding matters

On one held-out conversation, we compared three decoding strategies without changing any weights:

Figure 5. Greedy decoding is quiet (RMS 0.012 against the Mimi reconstruction's 0.034). Sampling only the acoustic codebooks CB1–CB7 nearly doubles the energy with almost no loss in timing (F1 0.763 → 0.752). Also sampling CB0 brings energy close to the reference but quadruples the frames falsely decoded as speech (47 → 186).

Sampling the acoustic codebooks is therefore a cheap improvement. Sampling CB0 needs to be gated by the activity decision.

Turn-taking and relevance

The full 75-second held-out conversation was transcribed by the dataset owner. It shows the main open problem. The user asks concrete gardening questions about planting distance, sunlight and drainage. The model replies with short generic phrases such as "Nice and confirm first. Take your time".

This recording also exposes a limit of offline testing. The pre-recorded user later says "Got it, 18 and 6" although the model never said those numbers. A replayed user cannot react to the model, so conversational behaviour can only be judged in a closed loop.

06 · Conclusions

Stage 1 question	Answer
Can a 61M audio-only hybrid be trained end to end with a frozen codec?	Yes. Stable training, exact memorization, and streaming inference that keeps listening
Is the generated speech audible and partly intelligible?	Partly. About 45–70% of words understood in informal listening; robotic texture
Does the response depend on the incoming audio?	Yes, on 507 / 507 held-out conversations. The dependence is mostly timing and activity
Are the answers relevant on unseen topics?	No. Replies are generic or carried over from training
Is the model ready for live conversation?	Not yet tested. No closed-loop evaluation exists

The baseline is useful precisely because its failure is specific. The model hears that someone is speaking and when they stop. It does not yet act on what they said, and it memorizes the coarse audio code instead of generalizing it.

07 · Next: Hybrid v0.2

The next stage keeps the architecture and changes what the model is trained to do:

• Listen for content. Add a listening-prediction head that predicts the user's next frame. Add a contrastive loss that compares the user's real question with a matched alternative in the same voice and timing but with different words.
• Stop memorizing CB0. Add dropout and label smoothing. Randomly blank spans of the model's own history so it must consult the user. Randomly drop acoustic codebooks on the user side for robustness to new voices.
• Better turn-taking. Weight the loss around turn ends, interruptions and backchannels. Add a head that predicts speech activity up to 400 ms ahead. Decode activity with hysteresis instead of a single threshold.
• Better decoding by default. Sample the acoustic codebooks, and sample CB0 only inside active speech.
• Pre-training. Add a pre-training stage on large single-speaker and real two-party speech before dialogue training. A small proxy model will test first whether it is worth the cost.
• Honest evaluation. Fixed held-out suites for single turns, minimal pairs, memory, interruptions and long turns, all with unseen voices. Conversational acceptance will come only from a simulated user that reacts to what the model actually says.

Every v0.2 result will be compared with R0 under the same decoding settings.

Appendix · Evidence and Reproducibility

• Every run records its code revision, dataset revision, manifest checksum, frozen split hash, seed, hardware and checkpoint SHA-256. Runs used single NVIDIA RTX A4500 / A5000 GPUs.
• Baseline R0: S1-E005 step 5,306, checkpoint SHA-256 69cdf3a4…2ac, dataset hohere-1m revision 2e3880f, test manifest SHA-256 6a1a9967…2d54.
• E005 was resumed once, at step 2,667, after a change that only made rollout preparation faster. GPU parity tests passed before resuming. The run therefore spans two code revisions, and both are recorded.
• Human listening scores so far are informal and come from a single listener, the dataset owner. Blind 0–5 intelligibility and relevance ratings for the E004-GEN and E005 test audio are still pending.
• NLL values are in nats per frame and sum CB0–CB7 and the activity head unless stated otherwise. Activity F1 measures speech timing, not intelligibility.