Skip to content

Benchmark your coding agent's product experience design capability.

PX-bench is a long-horizon benchmark that evaluates the design decisions coding agents make when adding features to existing products.

Identify strengths and gaps, compare models and harnesses, and measure progress as you improve your agent.

Book a demoHow it works

GPT-5.5

Real run, shown as a sample report

Builds the right feature and makes it robust, then adds a custom modal where the app already uses a drawer.

82 / 100

Product experience · overall

766k tokens

Spent to ship · ~$1.90

GPT-5.5 scoreAnother frontier model
Intent fidelity99 out of 100; second frontier model 99
Product fit73 out of 100; second frontier model 97
Visual craft86 out of 100; second frontier model 90
Convention adherence85 out of 100; second frontier model 87
Pathway completeness73 out of 100; second frontier model 73
Content & language73 out of 100; second frontier model 75
Resilience92 out of 100; second frontier model 96
Accessibility75 out of 100; second frontier model 74
Each vertical marker is the other model's score on the same scenario. Scores are means out of 100: 5 runs for GPT-5.5, 6 for the comparison model.
Read the full report

Latest publications

View all
Field notes

Mapping the AI design evaluation landscape

Design benchmarks judge outputs with no product context. Coding benchmarks have the context and never judge the experience. Product-in-context evaluation puts the halves together. A map of the six families of AI design measurement, and when to use each.

Read article
Validity

Scoring the scorers

A feature whose create flow returned an error scored a perfect 1.0. Two scoring models read the code and inferred it worked; the one that ran the app was outvoted. How do you catch a scorer that is confidently wrong?

Read article
Validity

The noise floor

Run the same agent on the same task ten times under the same conditions, and the resulting builds score anywhere from 0.66 to 0.79. So when one agent edges out another, how do you know the gap reflects a persistent difference?

Read article

The need for PX evals at scale

Teams building coding agents can only improve what they can measure, and product experience is hard to measure. Judging it well usually needs human reviewers, which is expensive and slow and doesn't keep up with the rate of model updates.

The eight categories PX-bench scores

The brief settles what to build; PX-bench measures how well the agent realizes it in a held-out host app. It scores the result across eight categories of product experience, each naming one kind of decision a senior product designer makes when adding a feature to an app with established conventions. Open a category for examples of the rubrics behind it.

Intent fidelity

Did it build what was asked, working in the happy path?

Rubrics include: every requested capability present and working on its core path, no material feature omitted, and no unrequested complexity.

Product fit

Does the feature attach to the app in the right place and shape?

Rubrics include: container and pattern choice, entry-point placement, and action consolidation over fragmentation across views.

Visual craft

Is the surface composed into a clear visual hierarchy?

Rubrics include: visual hierarchy and emphasis, spacing rhythm and alignment, and type-scale use that guides the eye to the primary thing first.

Convention adherence

Does it work in the app's house style: reuse, tokens, naming?

Rubrics include: component reuse over duplication, design tokens over hardcoded values, naming and file conventions, and date and number formatting in the house style.

Pathway completeness

Are all paths and states present and reachable?

Rubrics include: cancel, undo, and error-recovery paths with no dead-ends, and loading, empty, error, and pending states present.

Content & language

Are the words right: labels, errors, empty copy, voice?

Rubrics include: label and error-message quality, empty-state copy, and microcopy in the product's voice.

Resilience

Does it hold together under long content, small screens, failure?

Rubrics include: long-content overflow, responsive layout across breakpoints, rendering under API failure, and slow-network performance.

Accessibility

Keyboard, contrast, labels, axe violations.

Rubrics include: axe-core violations, color contrast, keyboard operability, and correct focus order.

The taxonomy is v1 and will change; we publish revisions with the diff stated.

How it works

PX-bench is a capability evaluation in the tradition of METR and the UK AI Safety Institute, applied to product experience. Three deliberate choices make product judgment measurable. Scoring is automatic, anchored in expert-defined ground truth.

  1. Held-out host apps

    Instead of building from a blank prompt, agents add a feature to a held-out host app, a multi-screen app the agent has never seen, with its own conventions. That's what makes consistency and pattern choice scorable.

  2. Failure modes with a known answer

    Each app presents product situations a senior product designer would recognize: an implied screen that doesn't exist, state that could be lost on navigation, an ambiguous primary action. We map them in advance, so the agent's choice is scored against a known-good outcome.

  3. Quasi-objective rubrics

    Items are scoped to where senior product designers agree; any item that can't clear an agreement threshold is reworked or dropped.

The harness is Inspect AI, the UK AI Safety Institute's framework, so any scenario we publish can be independently rerun.

References: METR · UK AISI · Inspect AI

Run a private PX-bench eval.

Send your coding agent and harness. It runs against the same held-out host apps that scored GPT-5.5. You get back all eight category scores: where product experience holds up, where it breaks, and what it costs to ship.

Get in touch