PX-bench
Long-horizon product experience benchmark for coding agents.
A methodology for evaluating AI agents on product experience
What a PX-bench evaluation run actually looks like, from the feature request we hand the agent to the scores we're willing to publish: the brief, the sealed environment where the agent doesn't know it's being tested, the scorers, and the independent audit that checks them.
Agentic design benchmarks need product context
Most AI design benchmarks evaluate design capabilities in silo, on a blank canvas. Yet, outside of the lab, agents build new features on top of an existing app with existing conventions. Measuring agents in a context-free environment misses important failure modes. To address this, PX-bench uses an application as a substrate for evaluating agent design work.
A taxonomy of product experience design for AI agents
Most discussions of AI "design capability" talk past each other because "design" points at three different things. This is the capability we name instead (product experience), set out as eight categories, with the exclusions stated explicitly.
Why it matters.
Coding agents now make product design decisions as they write frontend code. Given an incomplete brief, an agent must decide where a feature belongs, which states exist, and how the interface explains itself. PX-bench evaluates those decisions inside held-out host apps because product quality fails in ways ordinary coding checks do not catch.
- 01
Product quality is hard to reward.
Models improve fastest where outcomes are easy to verify: tests pass, builds complete, errors disappear. Product decisions are harder to score. An empty state can render correctly and still be unhelpful; a feature can work and still belong in the wrong place. PX-bench makes those decisions observable.
- 02
Working demos miss edge paths.
Most agent loops treat a working implementation as done. Review passes can catch broken code, but they do not guarantee the agent checked cancel paths, error states, long content, or mobile layouts. PX-bench scores the shipped product, including the paths a demo often misses.
- 03
Consistency depends on context.
Many product decisions only make sense relative to the surrounding app. The right component, term, format, or entry point is usually the one the product already uses. PX-bench tests agents in held-out host apps so consistency is judged against local conventions.
PX-bench measures what the agent ships: the product it hands you, judged the way a senior product designer would judge it.
What it measures.
The brief settles what to build; PX-bench measures how well the agent realizes it in a held-out host app. We score the result across eight categories of product experience, each naming one kind of decision a senior product designer makes when adding a feature to an app with established conventions.
The taxonomy is v1 and will change; we publish revisions with the diff stated.
How it works.
PX-bench is a capability evaluation in the tradition of METR and the UK AI Safety Institute, applied to product experience. It rests on three deliberate choices, with scoring that runs automatically but answers to senior product designers.
- 01
Held-out host apps
Instead of building from a blank prompt, agents add a feature to a held-out, multi-screen host app with its own conventions. That's what makes consistency and pattern choice scorable.
- 02
Failure modes with a known answer
Each app presents product situations a senior product designer would recognize: an implied screen that doesn't exist, state that could be lost on navigation, an ambiguous primary action. We map them in advance, so the agent's choice is scored against a known-good outcome.
- 03
Quasi-objective rubrics
Items are scoped to where senior product designers agree; any item that can't clear an agreement threshold is reworked or dropped.
Senior product designers set the ground truth. A scorer - script or agent - earns its place on an item only by matching that judgment at a set agreement bar. Where a call still needs a trained eye, a designer makes it directly. And where even the experts disagree, we publish the disagreement instead of forcing a score.
The harness is Inspect AI, the UK AI Safety Institute's framework, so any scenario we publish can be independently rerun.
Once a benchmark is public, models train on it, and it starts measuring exposure as much as capability. PX-bench keeps its scored scenarios held out and rotating, separate from anything we publish, so a score reflects capability rather than familiarity with the test.
Run a private PX-bench eval.
Put your coding agent through the same bench and get back a complete scored report.
Get in touch