PX-bench
Long-horizon product experience benchmark for coding agents.
Builds the right feature and makes it robust, then reaches for its own components instead of the ones the app already ships.
Bar = GPT-5.5 (mean of 5 epochs). Tick = a comparative SOTA model run on the same scenario.
Run PX-bench on your own agent.
Evaluate your coding agent and custom harness and get a scored report back. Identify quality gaps, regressions, and cost-saving opportunities.
Get in touchLatest publications.
View all →The AI design evaluation landscape
Design benchmarks judge outputs with no product context. Coding benchmarks have the context and never judge the experience. The two halves of the measurement that matters already exist, in separate fields, and nothing public puts them together.
Read →Scoring the scorers
A feature whose create flow returned an error scored a perfect 1.0. Two scoring models read the code and inferred it worked; the one that ran the app was outvoted. How do you catch a scorer that is confidently wrong?
Read →The noise floor
Run the same agent on the same task ten times, change nothing, and the score swings from 0.66 to 0.79. So when one agent edges out another, how do you know the gap is real?
Read →Why it matters.
Models improve fastest where the correctness of outcomes is easy to verify at scale. Product decisions are hard to score. An empty state can render correctly and still be unhelpful; a feature can work and still belong in the wrong place.
PX-bench makes agent product decisions observable and verifiable without requiring humans to review every output. It helps agent builders find quality gaps, catch regressions, and reduce cost without sacrificing quality.
What it measures.
The brief settles what to build; PX-bench measures how well the agent realizes it in a held-out host app. We score the result across eight categories of product experience, each naming one kind of decision a senior product designer makes when adding a feature to an app with established conventions.
The taxonomy is v1 and will change; we publish revisions with the diff stated.
How it works.
PX-bench is a capability evaluation in the tradition of METR and the UK AI Safety Institute, applied to product experience. It rests on three deliberate choices, each designed to make product judgment measurable with automatic scoring anchored in expert-defined ground truth.
- 01
Held-out host apps
Instead of building from a blank prompt, agents add a feature to a held-out, multi-screen host app with its own conventions. That's what makes consistency and pattern choice scorable.
- 02
Failure modes with a known answer
Each app presents product situations a senior product designer would recognize: an implied screen that doesn't exist, state that could be lost on navigation, an ambiguous primary action. We map them in advance, so the agent's choice is scored against a known-good outcome.
- 03
Quasi-objective rubrics
Items are scoped to where senior product designers agree; any item that can't clear an agreement threshold is reworked or dropped.
The harness is Inspect AI, the UK AI Safety Institute's framework, so any scenario we publish can be independently rerun.
Run a private PX-bench eval.
Put your coding agent through the same bench and get back a complete scored report: where its product experience holds up, where it breaks, and what it costs to ship.
Get in touch