Model comparison

The same agent harness, Jcode, run with six frontier models. Identical prompts, hardware, and harness version, so the differences below are the models.

jcode bench gives an agent a working implementation of a real primitive and asks it to make it faster while staying correct on every possible input. Correctness is exhaustively verified, the cost model is deterministic, and there is no time limit, so there is nothing to memorize and nothing to overfit. The three v1 tasks are float-print, json-unescape, and utf16-transcode. Benchmark design · tasks and full leaderboards · source.

These 18 runs launched 2026-07-19 on Jcode v0.51.4 with high reasoning, 4 CPUs, 8 GiB, single-use Modal containers. Every run passed the final correctness grade.

Leaderboard

A task score counts doublings of cost reduction: +1 means twice as fast, +2 four times. The combined score is the geometric mean of the three task scores, typical doublings per task, so no task dominates on raw headroom and the number is independent of every other entry. How the aggregate works.

Score over time

The three tasks per model ran concurrently, one container each. The line is the geometric mean of each task's best-so-far score at that moment, so it is the leaderboard number as it developed. The x-axis is concurrent wall clock time, total tokens processed, or total API cost at standard list prices, summed across the three tasks. Each line starts when the model has its first passing grade on all three tasks; before that the aggregate is undefined. Smoothed draws one ease curve from that first aggregate to the final score; points shows the exact best-so-far steps.

*On the time axis the Claude Opus 4.8 line is its historical 2026-07-05/06 head-to-head run on different hardware and an older harness, since time-curve data for its 2026-07-19 matrix run is not yet published; its endpoint there (+4.24) differs from its leaderboard number. The tokens axis uses the matrix runs for all six models, reconstructed from each run's per-message usage logs (all input tokens including cached context, plus output). The cost axis prices the same usage at standard API list rates, with cached reads and cache writes at their own rates.

Per task

Full unedited transcripts of all 18 runs:
Claude Fable 5: json float utf16 · GPT-5.6 Sol: json float utf16 · Claude Opus 4.8: json float utf16 · GPT-5.5: json float utf16 · GPT-5.4: json float utf16 · Claude Sonnet 5: json float utf16.

Scope

jcode bench v1 is deliberately low cost: three tasks, one run per cell, graded in seconds, so a full six-model matrix stays cheap to run and rerun. That keeps single-run variance in the numbers (a matched GPT-5.6 Sol rerun came in 16.3% lower one day later). Later iterations will add more tasks and higher k (multiple runs per cell), so the aggregate carries error bars instead of a variance caveat. Cross-harness results are on the full leaderboards.

Finals from the audited 2026-07-19 matrix. Raw data: json. Task specs, graders, and verifiers: github.com/1jehuang/jcode-bench. Benchmark design: /bench.