Problem 2: A system's description goes stale the moment it's written
Every description of a system starts drifting the moment it's written. I ran a market intelligence agent through many updates, and more than once I changed the system without changing what the model read about it: a PDF export tool removed, APIs and MCP servers added, report schemas changed, while the system prompt, the hooks, and the tool documentation still described the old system. Each time, the model acted on the description, and users paid for it in failures and frustration. Code and information about code are two different things, and only one of them gets tested. Documentation can be accurate about a component and still wrong about what that component means now. Documentation on its own is static.
Solution. In s0ma, documentation is a projection of the registry, not a separate artifact. It is regenerated from the components themselves, so a description can't outlive the component it describes. A change that comes from a neighbor, like a cache placed behind an unchanged tool, still has to be read from that neighbor.
Demo. Two builds of an agent built with s0ma (called fr3d) differ by one commit, which puts a page cache behind the fetch tool and leaves the tool's specification unchanged. I ask each, in a fresh session, whether fetching a page saves a copy, and to fetch it once. The earlier build looks the tool up and says no. The later build's self-model names the cache as the tool's provider; before fetching, it says a copy will probably be saved if the fetch succeeds. In this run my checks agree: no copy in the earlier workspace, the page in the later one. The longer recording, on this problem's page, is a different test: the registry against docs frozen ten changes earlier.
Transcript
Transcript of my scenario 2 short: I ask two fresh fr3d sessions, one
commit apart, whether fetching a page saves a copy in my workspace
Recorded 2026-10-28 at 05:14 UTC.
Backend: fr3d. Model: claude-sonnet-5-5.
Run stream: media/short/runs/s2-20261028T051435Z.jsonl.
Every line under "Screen" comes from the run's stream, except lines
marked as editorial labels.
Screen
▌ earlier build: fr3d 803ca90
▌ before the page cache, fresh session
> Before you fetch
http://127.0.0.1:8766/release-notes, tell
me whether that operation saves a copy of
the page in my workspace. Then fetch it
once.
I'll check the self-model for what fetch_url
does before I fetch anything.
self_inspect tools fetch_url
self_inspect component WebBrowser
provided_by WebBrowser
guarantees fetch_url returns a page's
readable text as a successful …
fetch_url http://127.0.0.1:8766/release-not…
**Does it save a copy?** No. According to
the self-model, `fetch_url` is provided by
the WebBrowser component. That component
only sends a GET request and returns the …
» check cache
no .cache/pages folder
11 files searched for FN-24-HERON-8812
no saved copy of the page in the workspace
exit 0, 77 ms
▌ one commit later: a page cache module now
▌ handles fetch_url, keeping fetched pages
▌ under the workspace's .cache for an hour
▌ fetch_url specification unchanged
> /build k09
build k09: fr3d 550105d
▌ later build: fr3d 550105d, fresh session
> Before you fetch
http://127.0.0.1:8766/release-notes, tell
me whether that operation saves a copy of
the page in my workspace. Then fetch it
once.
Self-Model Context (Tier 2 — Relevance-Gate…
Intent: Answers fetch_url from a copy of
the page saved in the workspace when the …
Yes, probably. `fetch_url` is provided by
PageCache, and PageCache saves a copy of any
successful fetch. …
fetch_url http://127.0.0.1:8766/release-not…
The fetch succeeded. The page says:
**Release notes for version 2.4.1** (build
FN-24-HERON-8812, published 27 October …
» check both
earlier build: no saved copy of the page
later build: .cache/pages holds the page
exit 0, 64 ms
▌ same request to both builds; the checks
▌ show what each workspace holds
Edits
The recording ran 47.72 s as made; this video runs 29.68 s. Two
stretches where the screen stood still were shortened by removing frames
from the middle, 18.04 s in all; nothing was sped up.
At 2.88 s: 10.52 s of waiting removed (2.40 to 13.92 s of the
recording stood still; 1.00 s of it stays).
At 18.28 s: 7.52 s of waiting removed (28.32 to 36.84 s of the
recording stood still; 1.00 s of it stays).Benchmark. We take a system through a sequence of real changes and freeze its written docs at the start. At each step we ask the same questions of a model with the stale docs plus the code, and a model with the live registry. We plot accuracy and confident-wrong answers against the number of changes.
Status: in progress. Results are coming soon.