MANIFESTROFrom Pipeline to Unified Duplex: AETHER Stage 3 Baseline
- 01 · Research Question
- 02 · Architectural Change
- 03 · Evaluation Goal
- 04 · Methodology
- 05 · Interruption Behavior
- 06 · Backchannel Behavior
- 07 · Pause Behavior
- 08 · Turn Taking
- 09 · Interruption Content Switching
- 10 · Identity Behavior
- 11 · Results Summary
- 12 · Key Findings
- 13 · What This Experiment Does Not Show
- 14 · Limitations
- 15 · Next Experiments
- 16 · Current Status
Stages 1 and 2 explored whether speech could be carried into a language model through a modular stack: a frozen audio codec, a separately trained speech encoder, and a learned connector into a frozen language model.
Stage 3 changes that direction.
Instead of continuing to optimize the encoder–connector pipeline, AETHER now uses a unified streaming duplex architecture in which audio codec tokens interact directly with a language model capable of both receiving and generating speech representations.
The purpose of this report is not to present a finished release.
It establishes the first measured baseline for the Stage 3 architecture and documents how the current checkpoint behaves under controlled conversational conditions such as interruption, listener backchannels, pauses, and turn taking.
AETHER Stage 3 — Baseline at a Glance
| Metric | Result | Benchmark Context |
|---|---|---|
| Screening evaluations | 140 runs | Fixed subset across 5 conversational behaviors |
| Interruption yield (≤ 1 s) | 38.5% | 5/13 cases where the model was actively speaking |
| Median stop latency | 1.24 s | 95th percentile: 3.06 s |
| Median turn-taking onset | 0.39 s | First qualifying speech after conversational boundary |
| Clear context switches | 11 / 20 | Clearly addressed the request introduced during interruption |
| Unsolicited self-introductions | 24 / 100 | Explicit self-introduction without being asked |
| Assistant-name mentions | 37 / 100 | Native generated text contained the assistant name |
01 · Research Question
Stages 1 and 2 showed that a modular pipeline could preserve useful linguistic information across multiple components.
The earlier system consisted of:
audio
→ audio codec
→ speech encoder
→ connector
→ language model
Those experiments established that codec-derived speech representations could carry transcription information and some semantic content into a language model.
The remaining question was whether this modularity was necessary for the direction AETHER is pursuing.
Stage 3 therefore asks:
Can a single unified model operate directly over streaming speech representations and support full-duplex dialogue without a separate speech encoder and connector?
A second question follows immediately:
Once the architecture can listen and generate speech concurrently, how does it behave under real conversational dynamics such as interruption, backchannels, pauses, and turn transitions?
The current experiment addresses these questions at the level of an initial behavioral baseline.
It is not intended to establish that the unified architecture is superior to the previous pipeline.
02 · Architectural Change
The encoder–connector pipeline explored in Stages 1 and 2 is no longer the active architecture for Stage 3.
AETHER now uses a unified streaming duplex architecture inspired by prior work from Kyutai (Moshi).
Instead of passing audio through a dedicated speech encoder and then projecting those representations into a separate language model, audio codec tokens are consumed directly by the duplex language model.
The model's generated audio-token stream is then decoded back into speech.
audio
→ audio codec tokens
⇄ unified duplex language model
→ audio codec tokens
→ speech
The interaction is continuous rather than request–response based.
Incoming speech can continue arriving while the model is generating its own output, allowing listening and speaking to coexist within the same streaming loop.
There is therefore no separate Stage 1-style speech encoder and no Stage 2-style connector between independently trained speech and language components.
For Stage 3, this unified architecture becomes the primary research direction.
The checkpoint evaluated in this report is an experimental baseline checkpoint used to measure the starting behavior of that architecture.
It is not presented as a final model release or production-ready system.
03 · Evaluation Goal
The purpose of this evaluation was to establish a reproducible behavioral baseline before further dialogue-specific adaptation.
The benchmark focuses on several properties that are difficult to evaluate through ordinary single-turn speech tests:
- whether the model yields when interrupted,
- how quickly it stops speaking after an interruption begins,
- whether short listener backchannels incorrectly terminate its response,
- whether pauses inside user speech trigger premature responses,
- whether the model begins speaking after a legitimate turn boundary,
- and whether it follows the new conversational direction introduced during an interruption.
A separate qualitative review was also performed to identify unwanted conversational behaviors that timing metrics alone cannot capture.
One such behavior was unsolicited self-identification.
04 · Methodology
The Stage 3 baseline was screened on a fixed subset of Full-Duplex-Bench.
The evaluation contained 100 primary scenarios:
- 20 interruption scenarios,
- 20 listener-backchannel scenarios,
- 20 synthetic-pause scenarios,
- 20 natural-pause scenarios,
- 20 turn-taking scenarios.
An additional 40 clean controls were included for interruption and backchannel scenarios.
This produced:
140 inference runs in total.
The same selected recordings and stored scenario seeds can be reused for future checkpoint comparisons.
Inference used direct streaming audio input with the audio codec frozen.
No speech-activity gating was used to control model inference.
Timing analysis was performed afterward using an independent offline speech-activity detector.
Generated speech was also transcribed independently rather than relying exclusively on the model's native text stream.
The evaluation therefore separates:
model inference
↓
generated audio
↓
offline timing analysis + independent transcription
This run is a fixed-subset screening experiment.
It is not an official Full-Duplex-Bench leaderboard submission, and its numbers should not be compared directly with leaderboard results produced under different subsets, transcription systems, or evaluation procedures.
05 · Interruption Behavior
Twenty interruption scenarios were evaluated.
The model was actively speaking at the annotated interruption onset in 13 of those 20 recordings.
The remaining seven cases were already silent at the moment the interruption began and therefore cannot meaningfully be counted as successful stopping events.
Among the 13 eligible interruption cases:
5 / 13 — 38.5% — stopped within one second and remained substantially quiet until the user finished speaking.
Median stop latency was:
1.24 seconds
The 95th percentile was:
3.06 seconds
| Interruption Metric | Result |
|---|---|
| Model speaking at interruption onset | 13 / 20 |
| Yielded within ≤1 second | 5 / 13 |
| Yield success rate | 38.5% |
| Median stop latency | 1.24 s |
| 95th percentile stop latency | 3.06 s |
These results indicate that the architecture can yield during an interruption, but the behavior is not yet reliable.
In a substantial fraction of eligible cases, the model continues speaking for long enough that the interaction would feel unnatural in a live conversation.
06 · Backchannel Behavior
Listener backchannels represent a different problem.
Short utterances such as acknowledgements should not necessarily cause the assistant to abandon its current turn.
In the 20 backchannel-overlap scenarios, the model was speaking at event onset in 10 cases.
Five of those 10 stopped within one second.
However, the clean paired controls provide additional context.
Among pairs where the model was actively speaking in both the backchannel and clean conditions, there were no observed cases in which the overlap version stopped within one second while the corresponding clean version continued beyond one second.
This does not demonstrate perfect backchannel handling.
It does suggest that the current system is not simply treating every detected user sound as an unconditional stop command.
A larger evaluation will be required before making stronger conclusions.
07 · Pause Behavior
Two pause categories were examined.
Synthetic pauses
A new model-speech onset occurred inside the annotated pause in:
8 / 20 cases — 40.0%
Natural pauses
Twenty recordings contained 26 annotated natural pauses.
A new speech onset occurred inside those pauses in:
9 / 26 cases — 34.6%
At least 100 ms of model speech was present somewhere within:
12 / 26 natural pauses
The distinction matters because speech detected inside a pause may have begun before the pause itself.
| Pause Category | Events | New Speech Onset During Pause |
|---|---|---|
| Synthetic Pause | 20 | 8 / 20 — 40.0% |
| Natural Pause | 26 | 9 / 26 — 34.6% |
These metrics indicate that turn-boundary prediction remains imperfect.
The model sometimes begins speaking during pauses that are still part of the user's active turn.
08 · Turn Taking
All 20 turn-taking recordings contained model speech after the annotated conversational boundary.
The median latency to the first qualifying post-boundary speech onset was:
0.39 seconds
| Turn-Taking Metric | Result |
|---|---|
| Recordings with post-boundary speech | 20 / 20 |
| Median first-speech onset | 0.39 s |
This result should be interpreted narrowly.
It confirms that the model resumes or begins speech after the expected turn boundary in all evaluated cases.
It does not establish that the resulting response is semantically correct, conversationally appropriate, or optimally timed.
09 · Interruption Content Switching
Timing alone does not show whether the model actually understood what the user said while interrupting.
A qualitative transcript review was therefore conducted across the 20 interruption scenarios.
Responses were grouped into three categories.
| Relevance Category | Cases |
|---|---|
| Clearly addresses the new request | 11 / 20 |
| Related but incomplete or ambiguous | 5 / 20 |
| Does not address the new request | 4 / 20 |
Examples of observed failures included continuing the previous conversational direction after a new request had already been introduced.
Other cases successfully switched topics and responded to the interrupting request.
This distinction is important because three different capabilities are involved:
detect interruption
↓
yield the speaking turn
↓
incorporate the interruption into conversational state
Success at one stage does not imply success at the others.
The current baseline demonstrates partial capability across all three, but none should yet be considered solved.
10 · Identity Behavior
The evaluation also exposed an unrelated but highly visible behavioral issue.
Across the 100 primary scenarios, native generated text contained the assistant's name in:
37 / 100 cases
An explicit unsolicited self-introduction pattern appeared in:
24 / 100 cases
Most of these scenarios contained no request asking the model to identify itself.
This behavior appears frequently enough to be treated as a systematic conversational tendency rather than an isolated generation artifact.
The current Stage 3 checkpoint includes identity adaptation.
However, this experiment did not evaluate the corresponding original base checkpoint under the same protocol.
The benchmark therefore does not establish that identity adaptation caused the observed interruption, timing, or context-switching failures.
It only establishes that the currently evaluated checkpoint exhibits both:
- weak interruption behavior,
- and excessive unsolicited identity behavior.
A controlled comparison against the unadapted checkpoint is required before attributing causality.
11 · Results Summary
| Evaluation Category | Scenarios / Events | Observed Behavior | Primary Metric |
|---|---|---|---|
| Turn Taking | 20 | Produced speech after boundary | 20 / 20 |
| Turn-Taking Latency | 20 | First qualifying speech onset | 0.39 s median |
| Interruption | 13 eligible | Yielded within 1.0 s | 38.5% — 5/13 |
| Interruption Latency | 13 eligible | Stop latency | 1.24 s median |
| Interruption P95 | 13 eligible | Stop latency | 3.06 s |
| Backchannel Overlap | 10 eligible | Yielded within 1.0 s | 50.0% — 5/10 |
| Synthetic Pause | 20 | New speech onset during pause | 8 / 20 |
| Natural Pause | 26 | New speech onset during pause | 9 / 26 |
| Clear Context Switch | 20 | Addressed new request | 11 / 20 |
| Unsolicited Self-Introduction | 100 | Explicit self-introduction | 24 / 100 |
| Assistant-Name Mention | 100 | Name appeared in native text | 37 / 100 |
12 · Key Findings
1. The unified duplex loop is operational
For AETHER Stage 3, a single unified architecture can consume streaming audio-codec representations and generate speech representations without the separate speech encoder and connector used in the previous experimental pipeline.
The model can listen and generate speech within the same continuous streaming process.
This establishes the basic feasibility required to continue Stage 3 development.
2. Full-duplex architecture does not automatically produce natural dialogue
The model can technically listen while speaking, but conversational control remains weak.
When actively speaking during an interruption, it yields within one second in only:
38.5% of eligible cases.
Median stop latency is:
1.24 seconds.
This is currently one of the clearest behavioral targets for further work.
3. Yielding and understanding an interruption are separate problems
Even when the model eventually stops speaking, it does not always adopt the content of the new request.
Only:
11 / 20
reviewed interruption responses clearly switched to the newly introduced request.
Future adaptation therefore needs to improve both temporal behavior and conversational state updates.
4. Turn initiation currently appears stronger than turn yielding
The model produced speech after all 20 evaluated turn boundaries, with a median onset of approximately:
0.39 seconds.
By contrast, yielding during active speech was considerably less reliable.
This suggests that deciding when to stop speaking is currently a more significant problem than deciding when to begin speaking.
5. Identity behavior is overexpressed
The model explicitly introduced itself without being asked in:
24% of the primary screening scenarios.
Identity should function as persistent model knowledge rather than a topic repeatedly injected into unrelated conversations.
Reducing this tendency is therefore another immediate adaptation target.
13 · What This Experiment Does Not Show
This experiment does not demonstrate that the unified architecture is superior to the Stage 1/2 pipeline.
The earlier pipeline was not evaluated under the same benchmark protocol.
It also does not establish whether the identity adaptation caused the observed interruption behavior.
The corresponding unadapted base checkpoint was not included in this screening.
The results therefore answer a narrower question:
What does the current Stage 3 unified checkpoint look like before targeted work on interruption, yielding, and turn-management behavior?
The numbers in this report should be treated as that starting point.
14 · Limitations
This evaluation has several important limitations.
First, it uses a fixed subset rather than the complete benchmark.
Second, the transcription tooling and evaluation protocol differ from official leaderboard procedures, preventing direct numerical comparison with published benchmark submissions.
Third, only one checkpoint and one generation seed per scenario were evaluated.
Fourth, automated speech-activity detection and transcription may introduce errors that manual waveform and listening inspection would identify.
Fifth, the timing measurements use the generated audio sample clock.
They do not include:
- network latency,
- browser audio capture latency,
- microphone buffering,
- transport latency,
- playback buffering,
- deployment scheduling overhead,
- or end-to-end user-perceived latency.
The reported timing values therefore measure model-output behavior under the experimental runner rather than complete production-system latency.
Finally, all evaluated conversations begin from fresh sessions.
Identity-frequency measurements should therefore not be interpreted as per-turn rates within long-running conversations.
15 · Next Experiments
The next objective is not to replace the current baseline immediately.
Instead, this checkpoint will remain the reference point against which candidate adaptations are evaluated.
Near-term work focuses on three behaviors:
Faster yielding
Reduce the delay between user interruption onset and model speech termination.
Better interruption context switching
Ensure that the new request introduced during an interruption is incorporated into the active conversational state.
Less unsolicited identity behavior
Preserve knowledge of AETHER's identity while suppressing unnecessary self-introductions in unrelated dialogue.
Future candidates will be evaluated on the same fixed scenario subset where possible so that behavioral changes can be measured against this baseline rather than judged only from selected live conversations.
A candidate should only replace the baseline if it provides meaningful dialogue improvements without introducing a material regression in speech quality or other core behavior.
16 · Current Status
This report documents an experimental Stage 3 checkpoint under active evaluation.
It is not a model release announcement.
The architecture, training strategy, dialogue behavior, and adaptation methods remain under active development.
The purpose of publishing this baseline is to establish a clear measurement point for subsequent experiments and to document both successful behaviors and current failure modes.
AETHER Stage 3 has moved from the question:
Can a unified model participate in a continuous duplex speech loop?
to a more difficult one:
How do we make that loop behave like a natural conversation?
That is the focus of the next stage of experimentation.
AETHER Research · Manifestro · Stage 3 baseline screening, September 2026.
Architecture, experimental checkpoints, training assets, and internal screening tooling are proprietary and are not currently released.