Aurora shipped with everything an AI agent is supposed to need: a rulebook, generated documentation, a component catalog, even a bundle for agents with no network access. We had never sat down and watched an agent use any of it. When we finally did, some of what we found had less to do with the models than with what we had been telling them.
Design-system teams have settled on a fairly standard shelf of artifacts for machines: a rulebook at the root of the repository, a generated documentation index, a component catalog, a drop-in bundle for agents working offline. We had built all of it and were reasonably proud of it, and we had never checked whether any of it changed a line of what a model wrote.
As far as we can tell that is the normal position. Teams put real effort into teaching agents their system and then measure the effort rather than the result. So we borrowed two instruments and pointed them at ourselves.
The first is a static audit. Seven checks read the repository and ask what a machine would ask on arrival: is the public API reachable, are the docs findable, is the catalog any good, what enablement exists, do our names match the names models expect, can a tool read our tokens, are deprecations legible. The result is one score out of a hundred, and it takes seconds to produce.
The second is behavioral, and less comfortable to read. A fixed set of ordinary interface tasks runs through six model families at two levels of context: nothing at all, and the same tasks with our rulebook in the prompt. A grader then asks two questions of every result. Did it use our system, and does what it wrote compile against the package a consumer would install? We ran the whole matrix twice, once against our own source tree and once against the published packages, because consumers build against what we ship rather than against what we happen to have checked out.
The audit came back at 64.8, in the tier the tool calls Invested: a repository that reads well. The behavioral composite was 44.9 against our source and 51.2 against the published packages.
Aurora read about twenty points better than it behaved, and the detail explained why. Nearly everything we had built described the system to a reader who was already looking for it. Very little of it gave an agent a reason to look.
The repo read better than it behaved
surface score (static audit) vs behavioral composite (six models) · 0 to 100
Four findings accounted for most of the gap.
One check failed outright. A folder of shared types was meant to be public and was not reachable at all: nothing pointed at it from the front door, and it was missing from the list of entry points the package exposes. Nobody had noticed, which is the sort of thing a static check is for.
Across both runs, all 113 unguided attempts used zero Aurora components. With the rulebook in context, roughly three-quarters of attempts still hand-rolled their own buttons and dialogs. Read from outside, our rulebook mostly warned models off other people's components without ever telling them to use ours. The correction turned out to be almost free, and we found it by accident: one sentence saying the package was already installed took hand-rolling down to about a fifth.
Then we compiled the attempts that did use the system, against the published types rather than our own source. Ten of fifty-five passed. The couple of hundred errors clustered into five naming habits, two of which accounted for four fifths of the total. Models write the name their training expects, and Aurora calls a few of those things something else.
This one was uncomfortable. While writing up the naming crosswalk we noticed that the rulebook, the design guide and the agent skill files all advertised two conveniences that do not exist in the library. The benchmark had already flagged both as things models commonly invent, so our own documentation was confirming a guess we should have been correcting.
Did the output use the system at all?
graded attempts · filled = used our components
Where the compile errors came from
55 engaged attempts · type-checked against the published package
The plan that fell out of the findings had eleven items across three phases. We shipped nine of them, under a rule we took from the plan itself: nothing here games a number without fixing the thing the number measures. The nine sorted into four moves.
The shared types are now a published entry point of their own. The obvious fix was one line: re-export the folder from the front door and move on. It would also have been wrong. Two unrelated types in the codebase share a name, and the compiler resolves that kind of collision by exporting neither of them, so the one-line version would have passed the audit while quietly removing a type our consumers depend on. Giving the shared types their own address avoids the collision and leaves both types reachable.
The documentation index already existed. It is now also written at the root of the repository, where the ecosystem has settled on looking, and it names every export of every multi-export family, so a model can find any of the eighty-nine by searching one file. One component had no documentation reach at all; it was missing from the generator entirely, so we wrote a description at the source and let it flow through the catalog into every surface downstream, which is where a fix has to go when the docs are generated. A handful of pass-through components documented nothing, honestly enough, because they forward whatever they are handed; they now document the one prop everybody passes. That last one took a failed attempt to learn, because the tidy way of writing it made the generator credit the prop to React and filter it out.
The largest new asset is a machine-readable token export, generated by reading the stylesheet rather than editing it: a couple of hundred variables with their aliases intact, dark-mode and responsive overrides preserved, and the comments people had left in the CSS carried across as descriptions. On top of that sits a derived layer for the roles that live as utility classes instead of variables, which would otherwise be invisible to any tool that only looks at variables. The first line of the file says that web code still styles through the classes. We did not want it read as permission to start pasting hex values.
The rulebook got the sentence it had been missing: every interactive element comes from the package, and a hand-rolled button, dialog, toggle or toast is a defect. Two lines of yes-and-no example underneath it. Then a short crosswalk of what we call things and what the published types will reject, covering the habits the benchmark had caught us on. Condensed copies are generated into the consumer-facing surfaces rather than pasted into them, so they cannot drift away from the original. A deprecation policy and two small editor entry files pointing back at the rulebook finished the move.
The two conveniences that do not exist are gone from the seven files that had been advertising them. Both are now rows in the crosswalk, so a model that guesses at them finds the real name sitting next to the wrong one.
The audit is free and takes seconds, so we re-ran it after every phase, with the tool pinned to its baseline version so the before and after stay comparable.
Every check, before and after
baseline audit → re-run after the changes · tool pinned · grey = before, green = after
| Check | Before | After | What moved it |
|---|---|---|---|
| Overall surface score | 64.8 | 82.0 | Everything below, in aggregate. |
| Export hygiene | 65 + fail | 100 | The shared types got their own entry point; the hard failure is cleared. |
| Docs greppability | 69.3 | 100 | An index at the root, every export named, the last undocumented component documented. |
| Enablement surface | 55 | 90 | The root index, agent bundles, editor rules, credit for fresh docs. One hint still missing. |
| Token machine-readability | 66.6 | 81.6 | The machine-readable token export. |
| Catalog quality | 86.4 | 88.1 | Exports with no documented props, nine down to three, all described. |
| Vocabulary | 61.4 | 61.4 | Did not move, by design (see below). |
| Deprecation legibility | 40 | 40 | Did not move, by design (see below). |
Two checks did not move. Vocabulary compares our API names against a lexicon mined from nearly nine hundred graded generations, and the only thing that moves it is renaming the API: a breaking change for every consumer, chasing a target that shifts with the next model release. We wrote the mapping down instead and left the score alone. Deprecation legibility has a floor and we are sitting on it, because static analysis cannot tell nothing is deprecated from deprecations are undocumented. In our case nothing is deprecated, and the checker has no way to know that, so we adopted the annotation as policy and the score will move the first time a real deprecation ships with one.
The behavioral side has not been re-measured yet. The harness is landing a fix that changes the prompts, and changing a prompt moves engagement by itself, so the six-model matrix waits for that. When it runs we will read it model by model rather than pooled: across the two baseline runs the same rulebook was worth +35.9 to one model and −2.1 to another, and an average would have hidden the regression.
The score is the part you can see; the change underneath it is that every surface an agent can read is now generated from the same sources the humans edit. One build command regenerates the catalog, the documentation index, the token export and the icon registry. One layer is still written by hand, the rulebook, and even its consumer-facing copies come out of a generator rather than out of somebody's clipboard. That is what keeps the surfaces from drifting apart the next time the team gets busy.
A respectable surface score was sitting on top of 113 out of 113 attempts ignoring the system. A static audit finds what is missing; a behavioral run tells you whether what exists changes anything. Do both, and grade against what you publish rather than what you have checked out, because that is what surfaced the type-level traps our source run could not see.
An index inside the installed package can be read offline and always matches the version the consumer has. Tools without repository access will only ever see a hosted copy, and audits and crawlers go looking at the repository root, so all three addresses need the same file. Tokens want the same treatment. Hand-written copies drift inside a week; we know, because ours did.
Models arrive with a prior about what things are called. A short "what we call things" table, kept next to the mandate and regenerated into every consumer surface, costs a fraction of a breaking rename. It also catches your own documentation teaching the wrong names, which is how we found ours.
Two checks did not move and one of them is capped, on purpose. A score moved by inventing props, or by annotating nothing, would tell the next reader something false about the system, and the next reader is probably an agent.
We still haven't given agents a way to query the catalog from an editor. Search the components, ask what props one takes. We already generate that information; we just never put a small read-only service in front of it. The audit still warns about the gap. It's also the leftover that would help people who aren't sitting in our harness.
The six-model re-run is waiting on a harness change that rewrites the prompts. Changing the prompts moves engagement by itself, so running now would mix two things. Opening the stylesheet so roles are real variables, not something we parse out of class bodies, is a bigger call than this round was. It would pick up the leftover token score. That can wait until someone wants to open that file.
One baseline audit and two full benchmark runs, with the published packages as the primary result and our own source tree as a cross-check, plus an earlier two-model pilot. The after-scores are audit re-runs against the working tree after those changes, with the benchmark pinned to its baseline version. Every number on this page comes out of those runs and regenerates from the same commands.