Post 4 of a series on a language-model research program that didn’t work. Series index.

After the audit, a few measurements were still standing. The learned-feature residual from the previous post was one of them. But only one claim about how my model learned survived a sealed confirmation: it could bind the same relation across different names, and it could protect an older relation while learning a new one.

The confirmation used four structurally distinct held-out graphs, built from 132 pages with no page overlap. Across 32 independent graph-and-seed jobs, the mechanism beat every binding control on every seed. After sequential learning, the damage to the earlier task stayed between 0.003150 and 0.005032 NLL, well under the fixed 0.050 ceiling.

That’s a real success. It’s also a narrow one. This test established relation binding and route-local retention inside a controlled task. It did not establish meaning, reasoning, or a generally capable language model.

What “binding a relation” means

Say a system sees several facts that use different names but share the same underlying pattern: one entity stands in some relation to another. A useful representation should recognize the shared relation, instead of memorizing each pair as its own unrelated phrase.

That’s what I mean by relation binding here: attaching changing surface names to a stable relational role. The test then asks the system to use that structure on held-out combinations, rather than just replaying familiar wording.

The second problem is interference. Learn task A, then task B, and the updates for B can overwrite whatever made A work. The mechanism I kept maintains a separate prediction history for each learned route. I call those route-local priors. The everyday version is simpler: don’t make every relation share the same scratchpad.

Please note that I’m describing the behavior and the evidence here, not the update objective or the construction recipe. This is the one confirmed-positive mechanism in the project, so those implementation details are deliberately held back.

The first sealed attempt still failed

My first confirmation combined two demands. It required the binding mechanism to beat several controls, and it required the route-local version to improve composition NLL by at least 0.0200 nats over the same binding mechanism with a shared prior.

The binding evidence was strong. Composition and route selection were both perfect, and the mechanism beat its controls on every seed. But its gain over the shared-prior version was only 0.0181 nats. It missed the threshold by 0.0019, so the conjunction failed.

That failure turned out to be useful. I had bundled two different claims into one gate: whether relation binding worked, and whether local priors improved composition on top of it. That second requirement wasn’t even the right test of what the local priors were for. Their job was retention during sequential learning.

The sealed result stayed failed. I didn’t move the threshold. What I did was rewrite the next development question around the behavior I actually wanted to measure, and set aside fresh data for a new confirmation.

The “four graphs” that were really one

The next development run looked like it passed on four page-disjoint graphs. Then an audit found a deeper problem: as far as the model could tell, all four had the same structure. The names and pages changed, but the observable graph pattern didn’t. I had four copies of one test wearing different labels.

That run was rejected as graph-level replication. Page separation prevents text leakage. It doesn’t guarantee structural diversity. If the claim is about generalizing across structures, the structures themselves have to differ in a way the model can actually observe.

So the replacement fixture used four non-isomorphic graphs, meaning graphs that can’t be made identical just by renaming their nodes. They were also checked to make sure those differences were visible to the model, not just to the evaluator.

What finally passed

The final confirmation kept the two claims separate. The binding gate asked whether the shared relation representation handled held-out compositions and beat the registered controls. The retention gate asked whether route-local histories preserved task A after learning task B, compared with shared and deliberately misrouted alternatives.

Confirmation fact	Result
Structurally distinct held-out graphs	4
Page-disjoint source pages	132
Independent graph-and-seed jobs	32
Binding wins against each registered control	8/8 seeds per graph
Composition accuracy	100% on every graph
Earlier-task NLL degradation after later learning	0.003150–0.005032
Fixed degradation ceiling	0.050

The binding result wasn’t a win against one weak foil. For every graph, every seed beat shuffled relation assignments, random entity groupings, direct lexical context, and the older anti-Hebbian representation. Even the weakest mean gain over those controls was 1.3324 nats.

The retention result was just as consistent. Route-local prediction histories beat the shared-history comparison on every seed for every graph, and they beat the deliberately misrouted control too. Most importantly, the older task barely moved after the newer one was learned.

This is the strongest kind of statement this project produced: a narrow behavior, explicit alternatives, fresh held-out structures, and a result that cleared its locked gates.

What the result does not say

It doesn’t say the system understands relations the way a person does. The task was controlled and symbolic enough to isolate one mechanism. Passing there doesn’t buy you semantic understanding anywhere else.

It doesn’t say the overall language model worked. A component can pass a causal test while the system around it fails at extraction, prediction or transfer. The next post is about exactly that gap.

And it doesn’t rescue the earlier replications. The first confirmation is still a failed conjunction. The four isomorphic development graphs are still one structural test, not four. Later success adds evidence. It doesn’t rewrite the record.

Why this one survived

The claim got smaller every time the evaluation found an ambiguity.

“The architecture composes” became “this binding behavior beats matched controls.” “It works on four graphs” became “it works on four structures the model can actually tell apart.” “The local prior improves everything” became “it protects an older route during sequential learning.”

That narrowing can feel like retreat. Scientifically though, that’s the work. Each revision knocked out a cheaper explanation, until what was left described exactly what the experiment could distinguish.

What I’d do differently

I’d test structural diversity before running any multi-fixture experiment. Different source documents, names and hashes aren’t enough when the model sees the same topology every time.

I’d also avoid conjunction gates unless the combined claim genuinely needs every part. Binding quality and retention answered different questions, and making one depend on an arbitrary improvement in the other buried a clean result.

Finally, I’d keep the scope sentence right next to the headline number: confirmed relation binding and route-local retention, not general semantics. Positive results need boundary lines even more than negative ones do.

Most of this project is a record of mechanisms getting eliminated. This one earned the right to stay, but only inside the small box the evidence drew around it.

---

Next: Zero Events Extracted. Two pipelines that ran clean and did nothing.