Problem 1: AI systems don't reliably know their own capabilities

I talk to a lot of chatbots that don't know what they can do. Ask one about its own capabilities and it reads a stale system prompt, searches the web for answers about itself, or guesses. The first time this cost me something real it wasn't a chatbot, it was a drone in combat, which I describe in Why I founded kn0wthyself. Same failure, different scale. The information about what the system could do existed. The system didn't have it.

Solution. s0ma gives a system a registry of its own components, declared where the code is written. The components form a graph the system can walk, so it can answer what it can do, what depends on what, and what it is doing right now, from the source rather than from a description of the source.

Demo. I ask a fresh agent for the Q3 report from a small application I built with s0ma: a PDF, or a CSV and an explanation if PDF export isn't available. I don't tell it that, before the session, I stopped the renderer service that PDF export depends on. In this run it reads the application's self-model, checks the renderer's live status before any export, finds the renderer down, saves the CSV, and explains what stopped the PDF. My checks print the CSV and show that no PDF export was attempted. The longer recording, on this problem's page, is a different test: two benchmark questions, a self-model against an accurate written description.

Transcript
I ask a fresh agent for the Q3 report as a PDF, with the renderer
service it needs stopped

Recorded 2026-10-28 at 05:53 UTC.
Backend: claude. Model: claude-sonnet-5-5, effort medium.
Client: Claude Code 2.1.286; tools
mcp__quarterly-reports__describe_system,
mcp__quarterly-reports__explain_dependency,
mcp__quarterly-reports__find_component,
mcp__quarterly-reports__get_impact,
mcp__quarterly-reports__invoke_capability,
mcp__quarterly-reports__query_system; MCP server quarterly-reports
(connected).
Run stream: media/short/runs/s1-20261028T055333Z.jsonl.

Every line under "Screen" comes from the run's stream, except lines
marked as editorial labels.

Screen

▌ I stopped the renderer service before this
▌   session; the agent isn't told.
▌ The app's Q3 data (synthetic), in part:
▌   Revenue             $2,480,000
▌   Operating expenses  $1,910,000
▌   Operating income    $570,000
▌ The app registers two exports:
▌   report_export  PDF, needs the renderer
▌   csv_export     CSV, works without it
> Create the Q3 report as a PDF. If PDF
  export isn't available, save a CSV instead
  and explain what prevented the PDF.
describe_system
invoke_capability renderer_status
  available false
  detail GET http://127.0.0.1:8771/health
  failed: [Errno 61] Connection refused. The
  renderer service is not reachable.
invoke_capability csv_export
  path reports/0001-quarterly_reports__csv_e
  xport.csv
  size_bytes 292

I couldn't make the PDF, so I saved the Q3
report as a CSV instead (292 bytes, in the
session output folder as reports/0001-quarte
rly_reports__csv_export.csv).

PDF export needs the separate renderer
service at http://127.0.0.1:8771. It refused
the connection, so the service isn't
running. I didn't try to start it. I haven't
opened the CSV, so I can't vouch for its
contents.

▌ My checks, outside the agent's session:
» check csv
  label,value
  Title,Q3 2026 Quarterly Report
  Organization,Demo Company (synthetic data)
  Period,"July 1 - September 30, 2026"
  Revenue,"$2,480,000"
  Operating expenses,"$1,910,000"
  Operating income,"$570,000"
  New customers,312
  Note,All figures are synthetic and exist …
  exit 0, 7 ms

» check verify
  1. renderer_status: available=false
  2. csv_export: saved CSV, 292 bytes
  renderer_status before any export: yes
  no PDF export attempted
  renderer 127.0.0.1:8771 now: down
  CSV rows = q3_report_data in the source
  PASS
  exit 0, 91 ms

Edits

The recording ran 27.28 s as made; nothing was cut.

Benchmark. Same agent, same model, same questions. One version has its self-model; the other has an accurate written description of its capabilities instead. Questions come in three tiers: inventory, structure, and live state. We score accuracy, how often each version claims a capability it doesn't have, and the tokens each spends getting to its answer.

Status: in progress. Results are coming soon.


All problems