MANIFESTROCan a Compressed Audio Signal Understand Speech?
We built a speech encoder that never touches a raw waveform — only the compact token stream a neural audio codec already produces. This is the record of what happened when we tried to read those tokens back into words: a wall we hit at 77% word error, the silent bug that caused it, and the fix that got us to 16.35%.
| Word Error Rate | Character Error Rate | Training speech | Training steps |
|---|---|---|---|
| 16.35% | 5.88% | 460h | 200k |
01 · The Question
Neural audio codecs compress speech into a short stream of discrete tokens — dozens of times smaller than the raw waveform, and cheap enough to send over a slow connection or store at scale. They exist to solve a transmission problem: get the audio from one place to another for as few bits as possible, then reconstruct something that sounds like the original. Nobody designed them to preserve meaning — they were never graded on whether a downstream system could recover what was said, only on how faithfully the waveform came back out.
That makes the question genuinely open, not rhetorical. It's entirely possible for a codec to sound perfect to a human ear while its underlying tokens carry almost nothing a language model could use — all the information needed to reconstruct the waveform, and comparatively little that captures words, grammar, or intent. Stage 1 of Aether was designed to settle that, as a narrow, load-bearing hypothesis, before we invested in anything built on top of it: if we freeze a neural codec exactly as published — no fine-tuning, no retraining — and train a new encoder on top of its token stream alone, does that encoder learn enough of the underlying language to recover what was actually said?
To measure that, we attached a lightweight text-decoding head to the encoder purely as a diagnostic instrument — a way to put a number on "did it learn language," not a component we ever intended to ship. If the number came back bad, it would mean the compressed representation itself was the problem, and no amount of clever modeling downstream could fix it.
If the answer were no, the rest of the Aether roadmap — connecting speech directly to a language model without ever writing out a transcript in between — would need a different foundation entirely.
02 · The Shape of the System
At a conceptual level, the pipeline looks like this:
What this report doesn't include. AetherSpeech's exact architecture, dimensions, training hyperparameters, and trained weights are Manifestro's own work and are not disclosed here. They're core to products we intend to build and license commercially. What follows is the scientific story — the question, what went wrong, how we found out why, and what the result means — not a blueprint.
We deliberately started from a codec we didn't build, rather than training one from scratch alongside the encoder. That's a real constraint, not a shortcut: it means every bit of linguistic signal we recover has to already be sitting in someone else's compression scheme, with no opportunity to shape that scheme around what a language model would find useful. If Stage 1 worked despite that constraint, it works considerably more easily with a representation designed for it — which is where later stages of this roadmap are headed.
03 · First Attempt, First Wall
The first full run trained for 200,000 steps on roughly 460 hours of English speech, evaluated throughout against the same held-out set we'd use for every run after it. For the first ~70,000 steps, word error rate fell the way you'd hope. Then it stopped — and sat at 76–78% for the remaining 130,000+ steps, no matter how much longer we ran it. That alone ruled out the simplest explanation: this wasn't a model that just needed more time.
Two other symptoms stood out. Transcriptions were systematically cut short — a 113-character reference sentence would come back as roughly 67 characters, as if the model kept running out of room before it finished the thought, not as if it were guessing randomly throughout. And validation loss, after an early minimum, started climbing while training loss kept falling — the textbook signature of a model memorizing a narrow slice of its data rather than learning to generalize from it.
Our first instinct was capacity: maybe the encoder was simply too small, or the optimizer settings too aggressive, for a task this hard. Neither held up under testing — the shape of the failure (truncation, not incoherence; overfitting, not underfitting) pointed somewhere more specific, and the "not enough model" explanation doesn't produce a training-loss-down, validation-loss-up curve this clean.
Word error rate across the first training run. Values are read from logged evaluation checkpoints; the line between checkpoints is smoothed for readability, not a per-step measurement.
04 · Diagnosis: A Silent Bottleneck
The train/validation divergence told us the model was overfitting a subset of its data, not underfitting all of it — so the next question was what, exactly, was shrinking that subset. Our training pipeline has a built-in safety check, standard for this kind of diagnostic decoding: it quietly discards any training example the text head has no realistic mathematical way to align, rather than forcing it through and feeding the model a broken gradient. We went back and measured what that check had actually been doing to the training set — not what we assumed it was doing, what it was doing — and it was severe: it had thrown out 84.3% of the training examples.
The reason traces back to the codec, not the encoder. It compresses audio down to roughly a dozen tokens per second of speech — a rate tuned for cheap storage and transmission, not for how densely spoken English packs information. Do the arithmetic on our training set and the two numbers are uncomfortably close: the average utterance ran about 12–13 seconds, giving the codec on the order of 150–160 tokens to work with, against roughly 150–225 characters of transcript for typical English speech at that length. For a large share of ordinary sentences — not unusual outliers, the average case — the codec was handing the diagnostic head a timeline with barely enough room, or not enough room at all, to lay out the words that had actually been spoken.
What survived that filter was a small, skewed slice of the corpus — disproportionately short sentences, the only ones with enough spare room in the compressed timeline to fit. The model hadn't failed to learn. It had been shown a biased 15.7% of the curriculum and had, quite reasonably, learned that well — which is exactly what a falling training loss and a rising validation loss look like from the outside.
05 · The Fix
We built an internal component that expands the codec's compressed timeline before the diagnostic head ever sees it, and corrected the safety check to reason about that expanded timeline instead of the raw one — so it stops discarding sentences that are, in fact, perfectly alignable. The core AetherSpeech representation used by later stages of the roadmap was left completely untouched; only the diagnostic branch changed. That distinction mattered to us: we wanted to fix the ruler, not quietly loosen what counts as a passing grade.
Rebuilding the training cache with the fix in place recovered training data almost entirely: from 15.7% kept to 99.99% kept, out of the same 460 hours — practically the whole corpus, not a marginal improvement. Before spending another 200,000 steps of compute on the strength of a hypothesis, we wanted to see it start working on a small budget first.
06 · Second Attempt, Different Curve
Before committing the full compute budget again, we ran a short validation pass on the corrected data: word error fell from 51.2% at step 1,000 to 37.2% at step 4,000 to 29.6% at step 8,000 — falling the entire time, with validation loss falling alongside it. That's the opposite shape of the first run, and it was enough to commit to the full run.
Training continued out to the full 200,000-step budget (the schedule was extended right after step 8,000, which produced a brief, expected uptick in error rate as the learning-rate schedule reset — it fell back below its pre-extension level within about 10,000 steps, exactly as that kind of restart predicts). The run settled into a genuine plateau over its final ~30,000 steps, oscillating in a tight 16.3–16.9% band rather than still visibly improving — this is a real result for the current design, not an artifact of stopping early.
Same 460 hours of speech, same evaluation set. The "after fix" line is the exact evaluation trajectory logged every 2,000 steps (104 checkpoints); the "before fix" line is a reconstruction from the first run's own report, since that run's raw logs weren't carried forward.
What the model actually gets right
REF: HE WOULD HAVE TO PAY HER THE MONEY WHICH SHE WOULD NOW REGULARLY
DEMAND OR THERE WOULD BE TROUBLE IT DID NOT MATTER WHAT HE DID
HYP: HE WOULD HAVE TO PAY HER THE MONEY WHICH SHE WOULD NOW REGULARLY
DEMAND OR THERE WOULD BE TROUBLE IT DID NOT MATTER WHAT HE DID
Many utterances now come back verbatim. Most remaining errors are phonetically close substitutions — ARRANGING read as AVANGING, MANAGERIAL as MAAGEIAL — rather than incoherent output.
07 · What the Result Means
Before this experiment, we set out score bands for what different result ranges would mean, so we couldn't talk ourselves into a good story after the fact. 16.35% lands in the band we'd defined as "practically complete" for this stage.
Combined with the qualitative read — verbatim transcriptions, coherent near-miss errors, no more truncated output — this tells us the encoder genuinely extracts linguistic structure from the codec's compressed tokens alone. That was the entire question Stage 1 was built to answer, and the answer is yes.
We didn't reach the "very good" band, and we're treating that as an open question for the next round of work rather than a shortfall of this one — the plateau we measured tells us specifically where to look next, instead of leaving us to guess.
Was it worth the detour?
The bug we found wasn't a footnote to the result — it was hiding inside every future experiment that would have reused the same training cache, silently capping quality well below what the design was actually capable of, on every run built on top of it from here forward. Finding and fixing it before it propagated further is, on its own, worth more than the headline number: it's a permanent correction to how we build training data for this whole line of work, not a one-off patch to make one run's chart look better.
08 · Limitations
A results section without its limitations isn't a report, it's an advertisement, so here's what this result doesn't show.
16.35% sits just above the "very good" 10–15% band we'd defined going in — we don't yet know whether that band is reachable from this exact setup, or whether it needs a genuinely different design decision to unlock. The diagnostic decoding used to measure all of this is deliberately simple: greedy, one symbol at a time, no external language model rescoring anything after the fact. That means every number in this report is closer to a lower bound on what the underlying representation carries than an upper one — a more capable decoder would very likely read a better number off the same encoder, for free.
The training and evaluation data is English only, and drawn from clean, studio-quality audiobook readings — not noisy rooms, overlapping speakers, or accents underrepresented in that corpus. And the codec was run in its single most conservative configuration; richer configurations of the same published codec exist and were deliberately left untested here, to keep this experiment isolating one variable at a time instead of several at once. Widening any of these is real, unfinished work — not a detail we expect to resolve itself.
09 · What's Next
Stage 1 asked whether a frozen codec's tokens carry enough of the underlying language. Stage 2 asks a harder question: can a large language model understand that same speech representation directly — without a transcript ever being written out in between? That work is underway; we'll report on it separately once it's concluded.
Aether Research · Manifestro · Stage 1 closed, 2026 — Architecture, hyperparameters and trained weights referenced in this report are proprietary and not released.