Open research problem · music, language and feeling

Affect Geometry

A music model learned to hear feeling in a voice from sound alone. I have spent four months trying to carry that into a language model's own geometry, and I can now say precisely why it has not worked yet. I'm looking for collaborators to help solve the part I can't solve alone.

Mithun Dalal · independent researcher · since late June 2026 · provisional patents filed on the core methods

The question: is a language model's emotional language only word statistics, or does it carry a structure that a model trained on a completely different modality also arrives at? If a geometry of feeling learned from audio alone can be mapped into a text model, keep its shape, and drive what the model says, then the text model's affect is grounded in more than co-occurrence. That doesn't show the model feels anything. It would show its affect expression rests on structure that doesn't depend on language.

What works

works
An ear for feeling, from music. Inside a frozen music model (MERT-v1-330M, 25 layers), I mapped 82 families of musical gesture and 66 cues: the sarabande lament, the Mannheim rocket, the ombra scene, rhetorical figures like suspiratio and aposiopesis. Every family and cue is graded at every layer, with no averaging. The model was never trained toward emotion. The idea validated almost immediately, and it has held.
works
Personal and live. Each voice is read against its own baseline, so the reading is personal. The chain runs end to end in my app on real recordings: voice, the graded map, then a frozen language model (Gemma 3 4B) with every token and layer captured.
works
Language carries the gestures, modestly. In Gemma's raw states, phrases written for the same musical gesture by different writers, in the same form, find each other about 4× more often than chance at the middle layers. Part of that is shared vocabulary. The signal grows with depth and settles on the last words of a phrase. Read as a course through the tokens, a phrase carries its gesture better than its final state does. With shared words removed, adjectives still sit about 4.5× above chance.
works
The geometry is curved, in both models. A straight line between distant concepts crosses voids where real states don't live; paths through real states don't. In Gemma at layer 29, 1,890 of 1,890 straight midpoints between distant families fall in a void, against 0 along the net of real states. MERT shows the same at most of its layers. I also mapped how MERT holds pitch. It's laid out by octave height, not as a helix, and microtonal steps (shrutis, quarter-tones) merge into the nearest semitone. So fine intonation has to come from a pitch tracker, not from MERT.

What failed, and why

Three serious attempts at the bridge, plus a week of auditing my own instrument. Each one failed in a way I can explain.

AttemptWhat happenedWhy
Word bridges (July)Paired music readings with words and searched for the words they lit up.Words are coarse. Gradation, micro-movement and timing are exactly what didn't survive.
A multimodal model (Qwen3-Omni)Fed the audio in through a model built to hear sound.Its audio side is trained toward speech. It squeezed the gesture down to words and basic prosody before the language model saw it.
Graded maps and a projector, through a lexical atlasGraded Gemma on 13,546 written phrases, one set per gesture, and matched each music reading to its counterpart in Gemma, then pushed Gemma's states toward the target.Pushes registered but didn't land. Straight pushes cross the voids. A linear fit of the readers doesn't predict the readers themselves. And see the audit below.
Audit of my own readersRead random state nudges and ordinary Wikipedia sentences through the same graded readers.They show the same leading patterns as the gesture phrases. "Bright against lament", essentially valence, and "pastoral against martial" are the instrument's grain. Every one of Gemma's 2,560 coordinates is counted by at least 21 of the 141 readers, and single coordinates are superposed. The readers were built on the wrong basis.
Readers rebuilt on sparse featuresRebuilt the readers on Gemma Scope features so each would own its signal.Unfinished. Eight designs so far; giving each reader private features costs separation. Paused, not concluded.

The written atlas itself is on trial. Gemma organises it first by form (scene, phrase, adjective) and by writer, and shared vocabulary explains much of the family signal. That's the "lexical leakage" worry, measured rather than assumed.

Where I'd like help

These are the open problems. Each comes with baselines and measurements already built.

  1. A reader that owns its signal

    Graded readers whose evidence isn't shared across every coordinate. Sparse features, directions, or the curved surfaces themselves. The test is ready: random nudges and ordinary text must stop reproducing the readers' patterns.

  2. An anchor that isn't words written to order

    Natural pairs where music and language already happen together in time: art songs and opera with their texts, film scenes with dialogue and score, and speech with its own prosody. The music's course and the words' course at the same moment, compared course to course, with no translation step.

  3. Steering along the surface

    Pushes that move a model's state along the net of real states, so they land instead of falling into voids. The machinery exists, tested on fixtures. It needs a valid instrument to aim with.

  4. The causal test

    Take a musical gesture, carry it across, and see whether the model's expression follows, with no emotional cue in the prompt. Controls: random and permuted projectors, a non-affect domain, and valence and arousal regressed out. Finer distinctions surviving (tension against anxiety, nostalgia against sadness) is the convincing result. Shared culture remains a possible explanation, and I'd state it as such.

  5. Beyond the Western map

    Rasa and tala as places on the same map. Wonder, disgust, laughter and terror are places the Western gesture map lacks. Tala would give the "when" of the forecast.

The instrument: Gemma Scope modded

Most of what I learned this month came from a small modification of known interpretability tooling, put together to watch the language model while it works. It records Gemma at every token and every layer: what goes into each block, what the block computes, what the model is attending to, and the exact state after any change I make. It keeps one complete recording and reads it three ways:

The parts are existing work: DeepMind's Gemma Scope, a lens in the line of the logit and tuned lenses, and an activation trace like those in TransformerLens-style tools. My modification is small. It ports them to Rust on a laptop GPU, has all three read one exact recording that replays without re-running the model, and makes it fast enough to sit inside the listening chain (a full passage across all 34 layers in under half a minute).

How it helps:

Demo: modscope.ffctos.com, recorded examples of Gemma reasoning, step through them token by token and layer by layer.

Adding a shape evaluator

I'm adding a shape evaluator beside it. It finds the low-dimensional shapes that concepts form inside the model (for example the days of the week forming a ring, a known finding from Engels et al., 2024, which shows up on Gemma in the early layers) and asks where the courses of feeling pass through or around them. It's early and still being validated against random and shuffled controls.

Not public yet; results will be added here once they hold up against the controls.

Why it matters past the bridge

The goal from the start has been a model whose own course moves with the feeling in a voice, and which we can watch and forecast: where its state goes next, layer by layer, before it gets there. The other half is expression: a singing voice whose affect comes from that course, not from imitation. The ear is done, the measurements of why the bridge is hard are done, and what's left is a well-defined open problem.