A music model learned to hear feeling in a voice from sound alone. I have spent four months trying to carry that into a language model's own geometry, and I can now say precisely why it has not worked yet. I'm looking for collaborators to help solve the part I can't solve alone.
Three serious attempts at the bridge, plus a week of auditing my own instrument. Each one failed in a way I can explain.
| Attempt | What happened | Why |
|---|---|---|
| Word bridges (July) | Paired music readings with words and searched for the words they lit up. | Words are coarse. Gradation, micro-movement and timing are exactly what didn't survive. |
| A multimodal model (Qwen3-Omni) | Fed the audio in through a model built to hear sound. | Its audio side is trained toward speech. It squeezed the gesture down to words and basic prosody before the language model saw it. |
| Graded maps and a projector, through a lexical atlas | Graded Gemma on 13,546 written phrases, one set per gesture, and matched each music reading to its counterpart in Gemma, then pushed Gemma's states toward the target. | Pushes registered but didn't land. Straight pushes cross the voids. A linear fit of the readers doesn't predict the readers themselves. And see the audit below. |
| Audit of my own readers | Read random state nudges and ordinary Wikipedia sentences through the same graded readers. | They show the same leading patterns as the gesture phrases. "Bright against lament", essentially valence, and "pastoral against martial" are the instrument's grain. Every one of Gemma's 2,560 coordinates is counted by at least 21 of the 141 readers, and single coordinates are superposed. The readers were built on the wrong basis. |
| Readers rebuilt on sparse features | Rebuilt the readers on Gemma Scope features so each would own its signal. | Unfinished. Eight designs so far; giving each reader private features costs separation. Paused, not concluded. |
The written atlas itself is on trial. Gemma organises it first by form (scene, phrase, adjective) and by writer, and shared vocabulary explains much of the family signal. That's the "lexical leakage" worry, measured rather than assumed.
These are the open problems. Each comes with baselines and measurements already built.
Graded readers whose evidence isn't shared across every coordinate. Sparse features, directions, or the curved surfaces themselves. The test is ready: random nudges and ordinary text must stop reproducing the readers' patterns.
Natural pairs where music and language already happen together in time: art songs and opera with their texts, film scenes with dialogue and score, and speech with its own prosody. The music's course and the words' course at the same moment, compared course to course, with no translation step.
Pushes that move a model's state along the net of real states, so they land instead of falling into voids. The machinery exists, tested on fixtures. It needs a valid instrument to aim with.
Take a musical gesture, carry it across, and see whether the model's expression follows, with no emotional cue in the prompt. Controls: random and permuted projectors, a non-affect domain, and valence and arousal regressed out. Finer distinctions surviving (tension against anxiety, nostalgia against sadness) is the convincing result. Shared culture remains a possible explanation, and I'd state it as such.
Rasa and tala as places on the same map. Wonder, disgust, laughter and terror are places the Western gesture map lacks. Tala would give the "when" of the forecast.
Most of what I learned this month came from a small modification of known interpretability tooling, put together to watch the language model while it works. It records Gemma at every token and every layer: what goes into each block, what the block computes, what the model is attending to, and the exact state after any change I make. It keeps one complete recording and reads it three ways:
The parts are existing work: DeepMind's Gemma Scope, a lens in the line of the logit and tuned lenses, and an activation trace like those in TransformerLens-style tools. My modification is small. It ports them to Rust on a laptop GPU, has all three read one exact recording that replays without re-running the model, and makes it fast enough to sit inside the listening chain (a full passage across all 34 layers in under half a minute).
How it helps:
Demo: modscope.ffctos.com, recorded examples of Gemma reasoning, step through them token by token and layer by layer.
I'm adding a shape evaluator beside it. It finds the low-dimensional shapes that concepts form inside the model (for example the days of the week forming a ring, a known finding from Engels et al., 2024, which shows up on Gemma in the early layers) and asks where the courses of feeling pass through or around them. It's early and still being validated against random and shuffled controls.
Not public yet; results will be added here once they hold up against the controls.
The goal from the start has been a model whose own course moves with the feeling in a voice, and which we can watch and forecast: where its state goes next, layer by layer, before it gets there. The other half is expression: a singing voice whose affect comes from that course, not from imitation. The ear is done, the measurements of why the bridge is hard are done, and what's left is a well-defined open problem.