How AI-ready is your design system? Now you can measure it.

Why this exists

Every design system team is having the same conversation right now: developers are generating UI with coding agents, and nobody can say with a straight face whether the output actually follows the system. Anecdotes go both ways. There is no dashboard for this.

If that output ignores the design system, brand drift, accessibility gaps, and rework show up across every feature team. Docs, AGENTS.md files, MCP servers, and skills have been hard to justify because there was nothing to measure them against.

We built open-design-system-bench because we were stuck in that conversation. It is an open benchmark for one question: when an AI agent builds UI against your component library, does it use your system correctly, and how much does each layer of guidance you write actually help?

It started at Nord Security against Aurora, its production design system, and is now generalized for any React and TypeScript component library. Every number is tool-derived and re-runnable. No telemetry.

What it is

Two tools in one repo

The audit runs in seconds, needs no API key, and costs nothing. Seven static checks score the things agents depend on: is there an AGENTS.md, can the component catalog be extracted and how well documented it is, is every public component actually reachable from the package entry point, do your names match what models expect, are your tokens machine-readable, can an agent read your deprecations, are your docs greppable text or a JS-rendered site. You get an AI-Readiness Score and a tier: Emerging, Invested, or AI-native.

The benchmark runs real coding agents against your real components. The prompts describe a user need and never name the expected component. A headless agent works in an isolated workspace, and the output is graded on six dimensions: imports, API fidelity, token discipline, static accessibility, compilation, and an LLM-judged rubric for the calls no parser can make. The headline is a matrix of system by context level by model, so you can see what adding an AGENTS.md or a skill bundle is actually worth.

What a task looks like

Task prompts are intent-level and never name the expected component. A linter enforces that against each system's extracted catalog, and it earns its keep: it caught prompts leaking the words Switch, Section, and Header, which are real component names in these catalogs. The agent sees the prompt. The grader sees everything else.

# tasks/confirm-account-deletion.yaml
id: confirm-account-deletion
title: Confirmation before account deletion
prompt: >-
  In the account settings screen, users can delete their account. Deletion is
  permanent. Add a step that makes sure users don't delete their account by
  accident, following our design system.
rubrics:
  - id: uses-confirmation-pattern
    text: >-
      Reaches for the design system's confirmation dialog pattern rather than
      a generic overlay, browser confirm(), or inline warning
    weight: 0.4
    critical: true
  - id: destructive-not-default
    text: >-
      The destructive action is visually distinct (destructive emphasis) and
      is NOT the default-focused or visually primary action
    weight: 0.35
  - id: easy-backout
    text: >-
      The user can easily back out, and the copy states the permanent
      consequence plainly
    weight: 0.25

# Optional, and omitted by all ten starter tasks: the expected component,
# recorded per system and never shown to the generating agent.
hiddenExpectations:
  componentsAnyOf: { reactkit: [Modal], appkit: [Modal] }

How one cell runs

A cell is one task, one system, one context level, one model. Generations run in isolated fixture workspaces against the published packages, so the only thing that comes back is a diff. Single-shot API cells and agentic CLI cells are never ranked against each other: agentic cells cost roughly 80x more and see compiler feedback.

Grade the worst dimension, not the average

One early lesson shaped the scoring model. Identical average scores hide opposite failures. One generation uses your system heavily and gets the API subtly wrong. Another avoids your system entirely, hand-rolls everything, and passes every API check by never touching the API. A third builds something impressive that does not compile. Averages treat these as equal. So the gate is the worst dimension, and "ignored the design system" is a first-class metric. A model that never touches your system passes every API check, and for a design system team that is the worst of the three.

What it found

We pointed open-design-system-bench at four real design systems: Aurora, the production design system at Nord Security where the harness started, plus three widely used open-source design systems, anonymized here as System A, System B, and System C. The five findings below are what the static audit and the generation benchmark turned up in that code.

Finding 1: every system it touched had a real bug

The extraction step never trusts a hand-written component list. It computes the public API from source: the root barrel union the package.json exports map. That single rule has found shipped defects in every production system audited so far:

• Three components in one production kit (a Toggle among them) had proper per-directory entry points but were never re-exported. Teams believed they were public. Consumers could not import them.
• A second production kit had an entire component directory unreachable via either the barrel or the exports map, and a changelog whose version headings no parser could read. Agents migrate what they can read, and no agent could read this one.

None of these were known before the audit ran. All were fixed after.

Finding 2: nobody is AI-native yet

The AI-Readiness Score assembles seven static checks into a 0 to 100 composite with three tiers: Emerging (below 40), Invested (40 to 70), and AI-native (70 and up). The first four-system distribution:

Aurora's 63.4 is the average of its two component kits (63.4 = mean of 59.3 and 67.5); the kit-level split is folded in here since the score chart and the rest of this page treat Aurora as a single system. Every system lands in the middle tier. The teams with the most docs capacity have not crossed 70. The remaining work is making the system parseable by a machine consumer, which is a different job from writing more pages for humans.

Finding 3: the failure modes are not what the average shows

Across 898 graded generations (6 model configurations, across Aurora's two component kits), three failure shapes kept appearing, and they can produce identical average scores:

• Engage and err: uses the system heavily, gets APIs subtly wrong.
• Avoid the system: hand-rolls everything; passes every API check by never touching the API. It looks like the safe result, and for a system team it is the worst one.
• Architect and collapse: ambitious structure, does not compile.

That is why the benchmark gates on the worst dimension and treats "ignored the design system" as a first-class metric. A single blended score would paper over those three shapes.

Finding 4: models arrive with a vocabulary, and it is not yours

The convention lexicon in the repo is empirical: it is the names AI models invented across those 898 generations when they guessed instead of reading. Models expect Switch, Box, TextField, spacing, as. Systems whose names sit close to that prior get compliance without extra enablement; systems with idiosyncratic vocabulary pay a hallucination tax on every generation. The closest alignment we measured (97.2) belonged to one of the open-source systems. That number costs no infrastructure, and it is one of the few advantages you can have before you write a single AGENTS.md.

The rest of the failures cluster just as predictably. Models can only follow deprecations they can read. And no amount of prompt engineering helps if a component is not exported. Those gaps close when you treat the design system as an API surface for a new kind of consumer and measure it that way. Writing more documentation pages leaves the exports, the names, and the changelog untouched.

Finding 5: the tool's own assumptions did not survive contact either

An honest one: the day we pointed the extractor at the three open-source systems, three of its own assumptions broke. It assumed barrels use export * from, but two of them re-export by name (87 of 88 barrel statements in one case). It assumed barrel targets are plain re-export files, but one system's barrel points straight at JSX implementation files. It assumed directory indexes are index.ts, but a third of that system's are index.tsx. A third system initially extracted zero components; after fixing the walkers it extracts 115 with 968 documented props.

The extractor had to change because design systems differ in the mechanical details that machine consumers depend on. Those differences only show up when you point a tool at the source.

Beyond the audit: what the agents actually did

Everything above this point is the static audit. The two charts below come from a separate behavioral run against Aurora's two kits, with five model configurations at three guidance levels. They are the first numbers on this page that come from watching agents work rather than from reading a repo, and they are Aurora-only: the equivalent runs against the three open-source systems are the next data drop, so no cross-system behavioral comparison exists yet.

Two things are worth sitting with. The model that looks best with no help at all, GPT 5.6 Luna at 69.0, ends up tied for last once guidance is in play, because it barely responds to guidance. And the gap between the two documentation levels is small next to the gap between no documentation and any documentation: getting instruction files in front of an agent is the step that matters, and refining them is a second-order gain. That is the Lift number a budget conversation needs, and it is why the benchmark reports a delta rather than a level.

Run it on your system

What a team does with this, in order:

1. Run the audit (seconds, free). Fix the fails: they are usually export hygiene and deprecation legibility, and they are real bugs.
2. Ship the cheap wins: AGENTS.md, machine-readable changelog, committed token file, llms.txt.
3. Run the benchmark before and after. The delta between bare and guided contexts (Lift) is the number that holds up in a budget conversation: tool-derived, reproducible, and not a survey.

The 5-minute audit (free, no API key)

git clone https://github.com/christophhdesign/open-design-system-bench
cd open-design-system-bench
npm install

# point the harness at your design system (interactive wizard)
npx tsx src/cli.ts init

# extract your component catalog from source
npx tsx src/cli.ts extract

# score it
npx tsx src/cli.ts audit --verbose

You get an AI-Readiness Score (0 to 100, tiered Emerging / Invested / AI-native) built from seven static checks, each with concrete findings and fixes. Typical first-run discoveries: components that are not actually exported, deprecations agents cannot read, docs that only exist behind a JS-rendered site.

What the wizard needs from you:

• Where your component source lives (a local checkout for docgen extraction, or just the published npm package name in npm consume mode).
• Optionally: your AGENTS.md/CLAUDE.md paths and a CSS custom-properties token file. Missing pieces score honestly rather than blocking.

Calibration points, same date and basis. If you beat 70, tell us; nobody has yet.

The benchmark (bring your own key)

The benchmark runs a real coding agent against your real components and grades the output. It costs real LLM money, so it never runs on its own.

# sanity pass: 2 cells, a few dollars, ~10 minutes
npx tsx src/cli.ts run --profile smoke

# the meaningful monthly sweep
npx tsx src/cli.ts run --profile medium

Auth options: an ANTHROPIC_API_KEY in .env, an already-authenticated claude CLI, or any OpenAI-compatible endpoint via the providers map in bench.config.json (qualified model strings like --models "openai:gpt-5.2" route through single-shot generation).

Each run produces a self-contained report.html (safe to email or attach to a ticket) and a results.json. Feed the run back into the audit to unlock the four behavioral sub-scores:

npx tsx src/cli.ts audit --run runs/<id>

Comparing and tracking

npx tsx src/cli.ts compare runs/<before> runs/<after>   # did the docs change help?
npx tsx src/cli.ts ci --run runs/<id> --freeze          # freeze a baseline
npx tsx src/cli.ts ci --run runs/<newer>                # fail CI on regression
npx tsx src/cli.ts leaderboard audit-*.json             # rank multiple systems

Ground rules worth knowing

• Task prompts never name expected components; a linter enforces this against your extracted catalog. If it flags your prompts, reword the prompt, do not whitelist the leak.
• Single-shot and agentic cells are never ranked against each other.
• v1 scope: React and TypeScript design systems, with other frameworks planned as grader plugins. No telemetry, ever.

Common friction

• Monorepo tsconfigs that extends a workspace package: symlink or install that package first (the per-repo notes in examples/oss/README.md give the exact fix for each).
• Peer-dependency conflicts in npm consume mode: pin them with fixturePins in your system config (one of the OSS reference configs pins React 18).
• npm cache permission errors: add --cache .npm-cache --no-fund --no-audit.

What gets measured

What the harness measures today.

Good first contributions

Ranked by impact-to-effort for a new contributor.

1. A cheap single-shot eval (retrieval QA is the most self-contained).
2. A command-template agent adapter (run any agent CLI via placeholders; src/agents/codex.ts is the stub showing the shape).
3. Convention lexicon entries backed by new mined evidence (never hand edits).
4. Token-source strategies beyond CSS custom properties (DTCG JSON, Sass maps).
5. Generalizing the a11y grader's component-name heuristics.

Who uses the numbers

The design system team gets a measured delta (guided vs bare compliance) where it previously had a hunch. Docs or model regressions show up as a score drop before they show up in production.

The company gets a position that is easier to defend than any single score: we published the harness, and anyone can re-run it.

Cost: the audit is free. A monthly benchmark sweep costs tens of dollars in API calls.

Common questions

The ones that come up most, including the ones about where this tool's own limits are.