PX-bench measures product experience: whether the feature a coding agent ships actually works and feels usable. In each test, we ask a coding agent under evaluation to add a feature to an existing application. Then we score the result across eight categories. A few scores come from scripts. Most come from scoring agents, which compare the finished work with the expected outcome written before the run.
Scoring agents help because they can judge work that simple scripts would miss. But they can also make mistakes, and their mistakes look just as official as correct scores. A scorer might give full credit to code that looks right but breaks in the app. It might miss a working feature because the agent built it in an unexpected way. Or it might say the feature is missing when the scorer simply did not know where to look. That is why each score needs a plain meaning: 1.0 means the feature works well, 0.0 means it does not, and a missing score means the evaluated agent did not make a meaningful attempt.
The sources of a wrong score
The scoring layer is built from parts, and each part encodes an assumption that can fail.
The rubric. Written too narrowly, a rubric item (a single check in the rubric) rewards one implementation where the brief permits several: it names a drawer, the agent ships a defensible dialog, and real work scores as a miss. Written too loosely, grading collapses back into opinion.
The scorer's instructions. A scoring agent told to grade from the code alone will pattern-match. Code that contains all the pieces of a working feature reads like one, whether or not it runs.
The scorer's tools. Under the judgment sits plumbing: helper scripts, regexes, DOM probes, the harness that boots the app. Each assumes a shape for work it hasn't seen.
The host app. Scorers are written against the app as it shipped; the agent is free to edit it. Rename a component a probe selects on, and the probe comes back empty on work that exists.
The model. Even with a sound rubric, clear instructions, and working tools, the language model can make the wrong call. It may mistake code that looks plausible for a working feature, or overlook evidence that a feature works. Its answer can also vary from run to run, so the same finished work may not receive exactly the same score twice.
To calibrate the scorer, we test and adjust the scoring layer until its judgments are consistent and well supported. The process has four steps: check the scoring setup, limit any one model's mistake, audit the results independently, and fix every traced disagreement.
1. Check the scoring setup
Before any agent runs, we check the rubric, the scorer's instructions and tools, and their assumptions about the host app. Each item is reviewed against the app as actually built, not as specified. Does the app genuinely carry the convention the item rewards? Does the item have exactly one home among the categories, so the same miss isn't punished twice? Does the scorer's code test what the item's words say? Each item comes out ready, usable with a footnote, or in need of a fix before it can score anything.
2. Limit any one model's mistake
Every item that calls for judgment is scored three times, by the same scoring agent on models from three different providers, and the published score is their median, not their average. This limits the effect of one model making a bad call. An average lets one scorer drag the number; a median ignores the outlier, so one provider's bad read, or its habit of going easy on work that looks like its own, cannot move the result.
The median has a failure mode of its own, and one run showed it plainly. The item graded whether saved views (a filter setup the user names and reapplies) could be created, edited, and deleted end to end. Two providers read the code and scored 1.0. The third exercised the running app, watched the create request fail with an HTTP 400, and scored it down. The median of [1.0, 1.0, 0.75] is 1.0. A median doesn't average the minority down; it discards it.
Two fixes followed. A scorer that exercised the app and landed well below the consensus now raises a flag that puts the item in front of a person; the dissent can still be wrong, so it doesn't overwrite the median, but it can no longer disappear. And the item itself now requires every flow verified against the running app. Re-scored, all three providers land on 0.75.
3. Audit what the scorers agree on
Three scorers can contain one provider's mistake. They cannot catch a mistake all three share, or a rubric item that is itself wrong. During calibration, we audit every run independently. This is how we find out whether a scorer is ready to publish scores. Once it has met the checks described below, we audit a sample of published runs instead of every one.
The auditor is an agent with read-only access to the run's artifacts: the code the agent wrote, the screens that actually rendered, and the case each scorer files alongside its number (what it expected, what it found, why). It re-grades every item from scratch and only then compares. Its independence is structural. It shares no scripts or probes with the scorers, and the audit runs several times, from different agent harnesses on different labs' models, with unanimous disagreements ranked first.
This is the same first pass a careful human reviewer would make: inspect the evidence, apply the rubric, and compare the conclusion with the original score. People still review the disagreements. But having them read every item in every run means that every new agent, task, or rerun adds another full set of scores for people to inspect by hand. An audit agent can make that first pass across the whole run and put the uncertain cases in front of them.
A disagreement is a finding, not a verdict. Each gets attributed to a cause from a closed list, and each cause is fixed in a different place. The cleanest example is the missing score, which hides three situations: the agent never attempted the feature, it wrote code but never wired it in, or it built the thing and the scorer failed to find it. Only the third is a scorer bug, and the bare number can't say which is which. The auditor can.
4. Fix the cause, then score again
Run the eval, compare scorer to auditor, attribute every disagreement, fix what the attribution names, run again. Sometimes that thing is the scorer, sometimes the rubric, sometimes the auditor itself. Sometimes it is nothing, because the call is genuinely ambiguous and belongs with human graders.
Every fix in that loop starts with a person reading the run. Attributing a disagreement means seeing what the agent built, what each scorer looked at, and what the auditor found, and digging that out of raw logs made every iteration slow. So we built Lens, a web app that lays a run out for inspection: the agent's transcript, the diff it shipped, the screens that rendered, and each scorer's evidence beside its score, down to where the three providers split. Most iterations begin there, with a person looking at a number that feels wrong.
One quick calibration check is to ask the scorer to grade the same frozen build several times. Because the work has not changed, the scores should not move much. A wide spread means the model's randomness is having too much influence on the result. But consistent scores can still be consistently wrong, so this repeatability check works alongside the independent audit rather than replacing it.
Full auditing is part of calibration. We reduce it to a sampled fraction of published runs only after meeting criteria set in advance: scorer and auditor agree on at least 95% of items where both produce a signal, repeated scores for a frozen build stay within the measured range, and auditors agree with each other at least 90% of the time (a yardstick that wobbles more than what it measures is not a yardstick). The evidence records stay for good; they are what lets a reader check our work without re-running it.
The strengths and limits of these checks
In the first end-to-end runs, scoring the same finished build twice could produce composite scores 0.2 apart. That is too much variation to trust: the apparent difference comes from the scorer, not the work. After calibrating the scoring layer, repeated scores for a frozen build differ by about 0.007. The scorer is now far more consistent, while the independent audit checks whether those consistent judgments are also defensible. None of this machinery guarantees that a scorer is right; it makes a wrong score visible, and a visible error can be attributed and fixed.
Agreement is not truth: the scorers and the auditor are all models, and a mistake every lab's models make would sail through both layers. That is why the most contested items go to people, and why each published score carries its evidence. The last line of defense is not our loop; it is a reader who can check the case we filed.
A calibrated rubric, a host app whose conventions carry real signal, scoring agents that verify what happened rather than infer it. None of it came out right on the first pass. Each piece is the product of reviews and runs that caught it being wrong. Most of the work of building the benchmark turned out to be exactly this.
Version 1.0 — June 2026. Reach us at hello@chordio.com.