This page is unfinished. Content and design are still in progress.
Uncontaminatable benchmarks
Agent benchmarks face a basic tradeoff between openness and contamination. Publish the tasks, and within months they can enter training corpora. Once that happens, a score may reflect memorization as well as capability. Keep the tasks private, and they are harder to contaminate, but the benchmark becomes difficult to inspect or reproduce.
Private benchmarks are also hard to trust. Outsiders cannot examine the full task set, audit the grader, or reproduce a published score from the same inputs. Even public benchmarks can depend on a particular harness, environment, or scoring rule that does not resemble how people use agents day to day. A short, isolated task says little about an agent's ability to explore a codebase, use tools, recover from mistakes, and keep improving over a long session.
Evaluation can be coarse even when the task is public. In benchmarks such as Terminal-Bench, an implementation is usually reduced to whether a task-specific grader passes. That grader can reject a correct solution because it expects a particular output or implementation detail. It can also accept a flawed solution because its tests miss the failure. The score then measures agreement with the grader rather than the full quality of the work.
Fixed time limits make that mismatch worse. They penalize an agent for spending more time on a hard problem, even though persistence and long-horizon work are capabilities we want to measure. A useful benchmark should record how quickly an agent improves without treating a deadline as the definition of success.
Pass/fail scoring loses information too. Imagine an 89-task benchmark ordered by difficulty. An okay model and a frontier model may both solve the first 76 tasks, while neither can solve the final 10. Their entire capability gap is then represented by tasks 77, 78, and 79. The other 86 tasks tell us almost nothing about the distance between them. A large improvement in capability appears as a difference of only three task completions.
Fixed task sets eventually saturate. Once frontier models solve nearly every task, only a few unresolved tasks remain to separate them. Scores bunch together, further progress becomes hard to measure, and the benchmark must be replaced with a harder task set.
An optimization benchmark does not end when an agent passes the task. Its limit is the mathematical floor or ceiling of the objective, and that limit may not even be known. Every verified improvement moves the known frontier. A stronger agent extends the benchmark instead of exhausting it.
Continuous, analog scoring gives every valid improvement a value. Evaluation follows directly from the task definition: exhaustive correctness is the gate, and performance under a published deterministic cost model is the score. There is no coarse rubric or sampled proxy between the implementation and its result.
The design below addresses the other problems as well: remove the answer key, publish everything, verify results deterministically, and record time rather than capping it.
With no answer key, training on the benchmark stops being cheating, because it is indistinguishable from acquiring the skill. That is why everything can be public, including complete transcripts of the best runs. A model trained on a winning transcript learns the techniques in it, but replaying them only matches the old score. Beating it requires improving past what was memorized, which is what the benchmark measures.
The spec
Six properties, each guaranteed by the structure of the task rather than by policing.
| Property | Meaning |
|---|---|
| Uncontaminatable | No answer to memorize. Specs, graders, harness, and every transcript are published, and none of it helps a model except by teaching the skill. |
| Hard to saturate | Scored on optimization depth, not task completion. The ceiling is the state of the art, and an agent that beats it extends the benchmark instead of exhausting it. |
| Deterministic | Same submission, same score, always. No sampled test set, no timing noise. |
| Quantifiable | The metric falls out of the task definition: instruction count under a published cost model. No rubric, no judge. |
| Continuous | An analog score axis instead of pass/fail, so capability differences appear as score differences at every level. Time is recorded, never capped. |
| Cheat-resistant | Correctness is verified over the entire input space, so there is no test suite to overfit. A strong reference implementation is the starting point, so looking up published algorithms only helps below the frontier, and past it there is nothing left to copy. |
jcode bench
jcode bench v1 is the first benchmark built to this spec. It is deliberately small: three tasks that are inexpensive to run. A task hands the agent a working, tested implementation of a real primitive (shortest round-trip float printing, JSON string unescaping, UTF-16 transcoding), together with its exhaustive verifier and a deterministic cost model. The task is to make it faster while staying correct on every possible input.
Exhaustive verification is what closes the overfitting loophole. A sampled test suite can be gamed: measure grep by timing it on the Linux repo, and an implementation can win by tuning to that repo specifically. Here the verifier covers the whole input space, all 232 floats in the float-printing task, so a submission is either correct everywhere or rejected. Correctness is a gate, and the only score is speed under the cost model.
Scoring
Improvements to optimized code are multiplicative, so the score is logarithmic: doublings of improvement over the given implementation.
A grade takes seconds, and the harness tracks the best score continuously, so a run produces a score-over-time curve rather than a number at a deadline. The curve shows how fast an agent climbs and how far it goes when simply left to work.
See jcode bench →jcode bench is in development. Task specs, graders, the harness, and complete run transcripts are published as they land. There is no hidden test set.