It Picks the Answer but Misses the Conversation: Dauys v0.2

Dauys v0.2 can often pick the right spoken answer to a short question, using only audio at its input and output. In conversation, though, what it says barely depends on what the other person said. It missed three of the six thresholds we set in advance, so we closed it as a baseline rather than train it further. This report records what it can do, what it cannot, and what we still do not know.

Parameters	Right answer, Kokoro (CB0)	Right answer, Piper (CB0)	Listening gap, Dauys vs Moshi
105.2M	78.8%	65.1%	+0.009 vs +0.337

---

Summary

Dauys asks how compact a speech model can be, and how little data it needs, to understand speech and respond to it. Abilities that are usually blended together can come apart: knowing the right answer is not the same as saying it clearly, and fluent speech does not prove the model heard anyone. So we measure each one separately.

Ability	Test	Dauys v0.2	Reading
Chooses the right spoken answer	Ranking, unfamiliar voices (CB0)	65.1–78.8%	Works on short questions
Depends on what was said	Timed swap of the other speaker	+0.009	Almost absent
Follows coherent meaning	tStoryCloze	48.8%	At chance
Knows words	sWUGGY	58.4%	Weak, close to Moshi
Knows grammar	sBLIMP	53.0%	Weak, close to Moshi
Avoids copying its partner	Excess echo, voice copy	+0.04%, +0.001	No copying
Speaks intelligibly	External speech recognition	not run	Unknown
Holds a conversation over time	Pauses, turn changes, interruptions	not run	Unknown

All speech in the dialogue tests is synthetic, and every result comes from one seed.

A follow-up test ruled out one explanation for the listening result: the model's memory of its own speech is not hiding its partner. Dauys notices that its partner spoke, but barely registers what was said (section 07).

Thresholds set before the final training stage. Scores cover the whole answer and all 8 codebooks.

Test	Result	Threshold	Verdict
Kokoro v1, top-1 answer	63.8%	≥ 75%	Missed
Piper v1, top-1 answer	48.9%	≥ 60%	Missed
Minimal pairs, word order	98.3%	≥ 95%	Met
Minimal pairs, want / don't want	73.3%	≥ 75%	Missed
No question (silence), Kokoro and Piper	13.4%, 14.7%	≤ 20%	Met
Benefit of the model's own past	+0.033 [0.024, 0.045]	> 0	Met

We did not specify in advance whether to score all codebooks or CB0 alone. On CB0, Kokoro and Piper would have passed, while want / don't want would not. All codebooks stay primary, as in the project's earlier rules, and the CB0 figures do not change the verdict.

This is not a strictly text-free result. No text enters or leaves the model, but six blocks of its dialogue tower started from the weights of a text model (section 01). Later models will start without them.

This report continues Dauys Stage 1, which closed a 61M model (v0.1). Its listening numbers were summed over all codebooks and the activity head on a different test set, so they cannot be compared with the CB0 gaps here.

01 · Architecture

A frozen "ear" hears the other speaker, and a dialogue tower predicts the model's next frame from its own past and the ear's states. All audio passes through the Mimi codec: 8 codebooks of 2,048 values at 12.5 frames per second, or 80 ms per frame. Its first codebook, CB0, is designed to carry semantic information. CB1–CB7 carry the acoustic detail.

Figure 1. Ear states reach the tower only through four cross-attention blocks. "◂ layer N" marks a Mamba-2 block initialized from that layer of a text model.

Each cross-attention block runs causal self-attention, causal cross-attention to the ear states and a SwiGLU layer in turn, with outputs initialized to zero.

The model has 105,185,502 parameters: 90,935,998 trainable and 14,249,504 frozen.

Component	Configuration	Parameters	Trained
Ear embeddings and projection	8 × 2,048 × 256, summed, then 256 → 512	4,325,888	No
Ear	6 Mamba-2 blocks: width 512, d_state 64, d_conv 4, expand 2	9,923,616	No
Model embeddings	8 × 2,048 × 256, shared with the depth decoder, plus role	4,194,816	Yes
Tower projection	256 → 768	197,376	Yes
Tower, Mamba-2	6 blocks: width 768, d_state 128	22,591,920	Yes
Tower, cross-attention	4 blocks: 12 heads, 512-frame window (41 s), SwiGLU 2,048	36,229,120	Yes
CB0 head	768 → 2,048	1,574,912	Yes
Depth decoder	4 causal blocks, width 512, 8 heads, 7 outputs for CB1–CB7	24,691,712	Yes
Activity head	Now and 5 frames ahead	9,228	Yes
Listening heads	Next frame of channel A: CB0 and activity	1,052,162	Yes
Other	RMSNorm, projection for predicting the other speaker	394,752	Yes

An intent head (1,312,256) and an EMA copy of the ear (14,249,760) are used only in training and are not counted above.

Initialization. Tower blocks 0, 1, 3, 4, 6 and 8 started from the first six layers of state-spaces/mamba2-130m, a text model, without its embeddings or output. We chose this from an A/B test on a small configuration (held-out CB0 NLL 3.44 against 3.61) and did not test it at full scale. Everything else started from scratch.

02 · Data

Corpus	Size	Speech	Stages
LibriSpeech (train.100, .360, .other.500)	799 h, 207k recordings	Human	0, 1, 2
Libri-Light medium	2,832 h, 563k recordings	Human	0, 1
Child-level dialogues, pools v1 and v2	2,024 h, 965k recordings, 906 concepts	Synthetic	1, 2
Minimal pairs, word order and negation	≈ 15 h	Synthetic	1, 2
hohere-1m, train split	3,139 dialogues, 17,947 assistant turns	Synthetic	2

Child-level dialogues are simple question-and-answer pairs on topics such as weather, feelings and numbers. They, the minimal pairs and hohere were all written by an LLM and rendered with TTS.

Held out from training:

• LibriSpeech dev-clean and dev-other;
• child-dialogue tests in Kokoro (unfamiliar phrasings, 4 voices) and Piper (6 voices from another synthesizer);
• the minimal-pair test;
• the new concepts of pool v2.

In the child pool we thinned addition problems so that the sums 2–10 appear equally often. In the second pool "ten" had reached about 20% of the pairs, and models were answering "ten" to questions about numbers.

03 · Training

Training had three stages, on one seed, one A40 GPU and one model configuration.

Stage 0, the ear. The ear trained alone from scratch on all 8 codebooks, on 100,000 examples. It predicted the next frame and also the HuBERT units of past frames, up to 32 frames back, which asks it to remember. HuBERT is self-supervised on audio, so no text enters this target. We measure memory with linear probes that read those units from the ear's states, fitted on training speakers and scored on held-out speakers.

Figure 2. Longer training mainly extends how far back the ear remembers. The 36,000-step ear was used. As a control, the score on future frames stays at 0.01.

Delay, frames	4,000 steps	12,000 steps	36,000 steps
0	0.61	0.70	0.72
8	0.60	0.72	0.75
16	0.21	0.66	0.72
24	0.06	0.38	0.66
32	0.03	0.12	0.48

Stage 1, listening. The tower trained for 40,000 steps, then a further 20,000 in the same run. Each batch held 16 groups of 4 examples: 10 from books, 5 from child dialogues and 1 from minimal pairs. Stateful streams, which carry model state across chunks of one recording, ran only up to step 20,000.

The extension went against a stopping rule we had set in advance, and the rule was right:

Held-out books	40,000 steps	60,000 steps
CB0 loss	3.253	3.226
CB0 accuracy	27.9%	27.9%
Meaning gap, CB0	+0.0049 [0.0033, 0.0066]	+0.0049 [0.0033, 0.0066]

The meaning gap compares the input as spoken with the same input where everything before the last second is cut into 1 s chunks and shuffled. Voice, words and ending stay, and only the order is lost.

Stage 2, dialogue. The model trained for 19,000 more steps from the 60,000-step checkpoint, with a fresh warmup and cosine schedule and no stateful streams. Each batch held 12 child-dialogue groups, 1 minimal-pair group, 2 hohere-1m groups and 1 book group. Book performance dropped and settled: loss rose from 3.226 to 3.398, and accuracy fell from 27.9% to 26.4%.

04 · Choosing an Answer

After each question, the model scores every candidate spoken reply by teacher-forced NLL, and the cheapest reply is its answer. Candidates are all distinct reply clips in the topic group. As a control, the question is replaced by silence of the same length, so a model that ignores the question should fall to chance.

Figure 3. On CB0, Kokoro and Piper clear their thresholds. On all codebooks, they do not. The white marker shows the all-codebook score with no question.

Test	N	Top-1, all 8	Top-1, CB0	No question, all 8 / CB0	Chance
Kokoro v1	552	63.8%	78.8%	13.4 / 16.5%	13.0%
Piper v1	570	48.9%	65.1%	14.7 / 13.0%	14.2%
Kokoro, new concepts v2	744	61.8%	—	20.0% / —	16.6%
Piper, new concepts v2	756	48.8%	—	18.8% / —	17.9%
Minimal pairs, word order	60	98.3%	98.3%	48.3 / 48.3%	50.0%
Minimal pairs, want / don't want	60	73.3%	73.3%	6.7 / 6.7%	3.1%

CB0 was not computed for the v2 concepts. They have a different number of candidates, so they should not be compared directly with v1. At step 12,000, Piper v1 on all codebooks had already reached 48.9%, so the last 7,000 steps added nothing.

CB0 beats all codebooks by 15–16 points on unfamiliar voices. Content appears to live in CB0, and adding the acoustic codebooks hurts the ranking. A likely reason is that they learned to prefer reply recordings heard often, but we did not test this on this model.

Scores vary widely by topic, and numbers are the weakest area:

Topic, Piper v1	All 8	CB0	Chance
Weather	90%	100%	10%
Needs	69%	86%	23%
Feelings	72%	82%	27%
Baby animals	50%	83%	6%
Categories	50%	77%	17%
Uses of things	70%	73%	27%
Animal homes	23%	70%	5%
Opposites	27%	67%	3%
Places	43%	63%	8%
Body parts	60%	57%	9%
Color	40%	50%	9%
Animal sounds	27%	43%	4%
Next number	17%	37%	5%
Previous number	7%	13%	5%
Addition	7%	10%	11%

Addition does not beat chance. An earlier model trained on books and then child dialogues reached 47% on it. The causes could be the thinned sums or a concept pool 2.4 times larger, and we tested neither.

The same task on earlier models:

Model	Kokoro, all / CB0	Piper, all / CB0
Child dialogues v1 only, 377 concepts, 4,000 steps	78.6 / 74.6	60.7 / 57.4†
Child dialogues only, 500 h, 4,000 steps	36–39 / 71–72	48–51 / 68–69†
Books, then child dialogues, 8,000 steps	40.8 / —	46.7 / —
v0.2 full run, 906 concepts	63.8 / 78.8	48.9 / 65.1

† Marked figures, except the first model's all-codebook score, come from a Piper split that reads about 5 points higher than the current one, so differences of a few points cannot be compared.

05 · Listening in Conversation

We ran Dauys and Moshi (moshiko) on one listening bench of 267 synthetic two-channel dialogues, with identical audio codes. The bench uses audio only. It compares the NLL of the model's own CB0 tokens with the real partner against a replaced one. A larger gap means the model's speech depends more on what it hears. The key test is the timed swap: another partner's speech placed exactly in the real partner's speaking frames, so timing stays and words and voice change.

Figure 4. Lines show 95% bootstrap intervals over dialogues. Dauys stays near zero under every replacement, while Moshi's gap is large in all of them.

CB0 NLL gap	Dauys v0.2	Moshi
Other conversation's partner, all speech	+0.013 [0.002, 0.025]	+0.796 [0.772, 0.822]
Silence, all speech	−0.007 [−0.028, +0.016]	+0.320 [0.309, 0.332]
Timed swap, all speech	+0.009 [0.001, 0.017]	+0.337 [0.324, 0.350]
Other conversation's partner, first 2 s of a turn	+0.077 [0.065, 0.090]	+0.827 [0.796, 0.858]
Silence, first 2 s	+0.044 [0.026, 0.063]	+0.523 [0.502, 0.545]
Timed swap, first 2 s	+0.019 [0.010, 0.028]	+0.475 [0.454, 0.497]

"First 2 s" means the first 25 frames of a model turn after a pause of at least 0.5 s in its channel. We had suspected the model listens only at the start of an answer. There the gap does double, but it remains 25 times smaller than Moshi's.

Other measures	Dauys v0.2	Moshi
CB0 loss on own speech, real partner	2.618	2.026
sWUGGY	58.4% [57.9, 58.9]	60.9% [60.4, 61.3]
sBLIMP	53.0% [52.3, 53.6]	57.0% [56.4, 57.6]
tStoryCloze	48.8% [46.5, 51.1]	65.4% [63.3, 67.5]
Speech within 60 s of generation	45.0 s	17.5 s
Excess echo	+0.04% [−0.02, +0.10]	+0.34% [0.23, 0.46]
Voice copy	+0.001 [−0.027, +0.029]	+0.013 [−0.015, +0.040]
Voice cosine to the data's assistant	0.979	0.366

How to read the comparison with Moshi:

• The timed swap is not pure content. It changes both the words and the voice of the partner. Separating them needs a same-voice swap, which this bench does not have.
• Moshi has an advantage we cannot remove here. It also writes a text stream, filled during scoring by its own greedy guess. That stream changes with the partner and may inflate its gap.
• Do not compare these language scores with the literature. sWUGGY and sBLIMP and tStoryCloze are spoken on the model's own channel here, so the numbers are valid only within this bench. With mean-normalized scores instead of summed ones, tStoryCloze is 52.2% for Dauys and 79.3% for Moshi.
• Echo has no human reference. It is measured against a swapped partner.

An effect we did not expect. Dauys's acoustic codebooks react strongly to digital silence. Mean CB1–CB7 NLL rises from 4.28 with the real partner to 4.99 with silence. It rises almost as much, to 4.95, when the real partner is kept but their non-speech frames are replaced with silence. Words therefore do not explain this cost, and the timed-swap gap on these codebooks is −0.017 [−0.052, +0.018]. In Moshi the same changes cost at most 0.09. We have not investigated why.

Own past and books after stage 2. Carrying state across chunks of a book lowers NLL by 0.033 [0.024, 0.045] from the third chunk on, compared with resetting it. On the same scale, state from a different speaker's recording gives −0.079, and shuffled state gives −0.019, with an interval that includes zero. The meaning gap on books is +0.0045 [0.0027, 0.0069], against a noise control of −0.0011 [−0.0026, +0.0004]. It is +0.0071 [0.0029, 0.0107] at the start of the answer. These gaps are small, though they survived the dialogue stage.

06 · What We Learned

1. Understanding a short question exists and holds across voices. On the v1 tests, the no-question control sits at or near chance on all codebooks and the gap to the real score is large. CB0 on Kokoro is the highest we have seen so far, though it comes from one run.
2. The dialogue stage raised understanding at the expense of books. Book loss rose and did not recover, while the meaning gap on books remained.
3. Longer ear training strengthened its memory. At delays of 1.3–2.6 s it is 3–16 times stronger than after 4,000 steps. Without a control run using a short-memory ear, we cannot say this helped the tower.
4. Stateful streams are not needed for all of training. After we turned them off at step 20,000, the benefit from the past rose from +0.026 to +0.033 by step 24,000, and the loss began to fall faster. We had predicted the opposite.
5. Extending stage 1 did nothing on this schedule and data mix.
6. The model's own history does not explain the missing listening. Erasing it left the gap unchanged (section 07).

07 · Why the Model May Not Listen

We started with seven hypotheses. We have tested the cheapest one, and it was refuted.

Tested: its own history drowns out the partner (H1)

The ranking tests contain almost none of the model's past speech, while on the bench it sees all of its earlier turns. That could explain why dependence appears in ranking but not on the bench. If so, erasing the model's past should bring the dependence back.

We replaced the model's speech before a chosen turn with silence and measured the gaps on the first 2 s of that turn. The rule was set before the run: H1 holds if the timed-swap gap with history erased reaches +0.10, its lower bound is above zero, and it is at least 3 times the gap with history kept. H1 is refuted if the upper bound is below +0.10.

CB0 NLL gap, first 2 s of a turn	Dauys, history kept	Dauys, history erased	Moshi, history kept	Moshi, history erased
NLL with the real partner	2.548	3.146	2.484	2.855
Timed swap	+0.010 [−0.003, +0.023]	+0.009 [−0.013, +0.029]	+0.430 [0.402, 0.459]	+0.447 [0.418, 0.478]
Silence	+0.072 [0.057, 0.089]	+0.111 [0.081, 0.142]	+0.518 [0.490, 0.547]	+0.507 [0.478, 0.536]

H1 is refuted. The upper bound of +0.029 is well below +0.10, and erasing history changes the Dauys gap by −0.002 [−0.018, +0.015]. Both models use their own history, since NLL rises clearly without it, but in neither case does it hide the partner.

The table also shows a pattern we did not set out to test. Silence produces a clear gap for Dauys, while other words at the right moments produce almost none. The model appears to register that its partner spoke, but not what was said. We have not tested why.

This has limits. It is one seed and only the first 2 s of a turn. The silence control also changes timing, and the timed swap also changes the voice. Only the past before the turn is erased. This test scores the first and last eligible turn of each dialogue, so its numbers differ from section 05, where the same Dauys gap is +0.019 and Moshi's is +0.475.

Not yet tested

Unless noted, each of these can be checked on the existing checkpoint.

H2. Too little dialogue. hohere made up only 2 of 16 groups, for 19,000 steps, and all of it was synthetic. Test: a new stage 2 with a larger share of dialogue.

H3. Answers are retrieved, not built. Reply recordings repeat, so a question may only select one. In an earlier experiment, when the first frames were swapped, the model followed the swap. Test: repeat this on the final model, and measure the gap on short child dialogues.

H4. These are different skills. Choosing among memorized replies does not require holding meaning over time. tStoryCloze is at chance, and an earlier memory probe did not hold a word for 20 s. Test: a task whose answer depends on something said earlier.

H5. The path from ear to tower is weak. The ear is frozen, was trained on books and connects through only 4 blocks. Test: switch off those blocks one by one, and train a new run with an unfrozen ear.

H6. Part of Moshi's gap comes through its text stream. Test: run Moshi with a fixed text stream.

H7. Assistant speech is predictable without a partner. Training uses one assistant voice, which also appears in the test, as the voice cosine of 0.979 shows. Test: split the bench by turn type.

08 · What We Could Not Establish

• Free speech quality. The spoken-answer test with Whisper was not run. We only know that the model talks for 45 s of every 60, so it rarely yields the floor. Turn changes were not measured.
• Live speech. This checkpoint was not tested on it. Two earlier models scored about 39% over 28 attempts with two speakers.
• The robotic voice. It remains unsolved and unmeasured.
• Variance. With one seed, differences of a few points are not confirmed.
• The value of three design choices: the ear's long memory, the text-model initialization at full scale, and child data in stage 1. None has a control run.
• Intent selection. Choosing the true continuation among 8 same-chapter candidates scores 23.8%, against 12.5% chance. A separate dual-encoder control reached only 17–22% on held-out data, so we do not read this as a failure or treat our earlier 70% target as supported.

09 · What Comes Next

v0.2 stays as the reference point for v1.0, a separate line that moves to real human speech. Its goal is full-duplex conversation without text, and that is a goal, not a result.

• The bar for listening is set: a timed-swap gap of +0.009 for v0.2 against +0.337 for Moshi.
• The problem is now narrower. Hiding the model's own past does not help, so the next explanations to test are how little dialogue it saw and how weakly the ear connects to the tower.
• Ranking will be scored on CB0 and on all codebooks separately, with the primary score named in advance.
• Later models will not start from text-model weights.

Appendix · Evidence and Reproducibility

• Model code: research/hybrid-v0.2 at 7dea864, configuration dauys-hybrid-v0.2-stage1-memear, seed 0. Stage 1 ran on b5fb778 and fe07c37.
• Listening bench: bench/, with per-dialogue results for both models. The history test in section 07 ran on dcb00b3.
• Not public: the checkpoints, the child dialogues, the bench dialogues and the raw history-test results. The model is not publicly available.
• Intervals are 95% bootstrap intervals. NLL is in nats per frame, on CB0 unless stated otherwise.
• Pre-set rule: if the thresholds were missed, v0.2 would close as a baseline with no further training. We followed it. The one departure from a pre-set rule is the stage 1 extension in section 03.