PX-bench
The long-horizon product experience benchmark for coding agents.
Evaluate your agent's product experience design capability to reduce token spend and confidently update models without introducing regressions.
Builds the right feature and makes it robust, then adds a custom modal where the app already uses a drawer.
Bar = GPT-5.5 (mean of 5 runs). Tick = a second frontier model on the same scenario.
Run PX-bench on your own agent.
Evaluate your coding agent and custom harness and get a scored report back. Identify quality gaps, regressions, and cost-saving opportunities.
Get in touchLatest publications.
View all →The AI design evaluation landscape
Design benchmarks judge outputs with no product context. Coding benchmarks have the context and never judge the experience. The two halves of the measurement that matters already exist, in separate fields, and nothing public puts them together.
Read →Scoring the scorers
A feature whose create flow returned an error scored a perfect 1.0. Two scoring models read the code and inferred it worked; the one that ran the app was outvoted. How do you catch a scorer that is confidently wrong?
Read →The noise floor
Run the same agent on the same task ten times, change nothing, and the score swings from 0.66 to 0.79. So when one agent edges out another, how do you know the gap is real?
Read →The need for PX evals at scale.
Product experience is hard to score, and judging it well usually needs human reviewers. That is expensive and slow, and it never keeps up with the rate of model updates. So the questions that decide cost and quality go unanswered. Which model is most cost-efficient for this task? Does your harness still hold after an update? Without answers, regressions creep in and good product experience stays a costly, uncertain part of every agent build.
PX-bench answers those questions without a human reviewing every output. In the demo above, GPT-5.5 builds the right feature: intent fidelity 99. Then it builds a custom modal for create and edit where the app already uses a drawer: product fit 73. The report names gaps like that, catches regressions, and prices every run.
The decisions we score.
The brief settles what to build; PX-bench measures how well the agent realizes it in a held-out host app. We score the result across eight categories of product experience, each naming one kind of decision a senior product designer makes when adding a feature to an app with established conventions.
The taxonomy is v1 and will change; we publish revisions with the diff stated.
How it works.
PX-bench is a capability evaluation in the tradition of METR and the UK AI Safety Institute, applied to product experience. Three deliberate choices make product judgment measurable. Scoring is automatic, anchored in expert-defined ground truth.
- 01
Held-out host apps
Instead of building from a blank prompt, agents add a feature to a held-out host app, a multi-screen app the agent has never seen, with its own conventions. That's what makes consistency and pattern choice scorable.
- 02
Failure modes with a known answer
Each app presents product situations a senior product designer would recognize: an implied screen that doesn't exist, state that could be lost on navigation, an ambiguous primary action. We map them in advance, so the agent's choice is scored against a known-good outcome.
- 03
Quasi-objective rubrics
Items are scoped to where senior product designers agree; any item that can't clear an agreement threshold is reworked or dropped.
The harness is Inspect AI, the UK AI Safety Institute's framework, so any scenario we publish can be independently rerun.
Run a private PX-bench eval.
Send your coding agent and harness. It runs against the same held-out host apps that scored GPT-5.5. You get back all eight category scores: where product experience holds up, where it breaks, and what it costs to ship.
Get in touch