jcode bench

Optimize production-grade primitives. Exhaustive correctness is required. Every verified improvement counts, and runs have no time limit.

How a run works

Each task gives the agent a working implementation, an exhaustive verifier, and a published deterministic cost model. The agent edits the code and submits it for grading as often as needed. Each grade takes seconds, and the benchmark records every new best result. Correctness on every possible input is the gate. Among passing submissions, the score measures doublings of cost reduction relative to the starting implementation.

Tasks

Three tasks, chosen for fast grading and substantial optimization headroom. A better result raises the known frontier for every later run.

Leaderboards: every harness and model, by task

Every run below is one solo agent at high reasoning that passed the official final correctness grade. Each chart plots the running best score against active agent time. The two default lines are the original 2026-07-05/06 jcode and Claude Code head-to-head runs. The buttons add or remove any other harness or model.

There is deliberately no combined leaderboard. The tasks have very different optimization headroom: float-print scores reach +12 while utf16-transcode stays under +4, so any cross-task average mostly measures float-print. Task-level rankings avoid that distortion.

json-unescape

float-print

utf16-transcode

Best is the highest grade sampled during the run, so it can sit slightly above the official final score, and for float-print not every sample is a full 2³² gate. Runs were recorded on different dates with different harness versions, and each cell is one run. †The Claude Sonnet 5 json-unescape run logged a single +14.30 grade from a degenerate random corpus; the surrounding grades sat near +0.93 and no later grade came close, so it is excluded here and documented in the raw curve data.

Jcode results: frontier model matrix

These solo runs launched on 2026-07-19 through the same pinned Jcode harness with high reasoning, identical published prompts, 4 CPUs, 8 GiB, and single-use Modal containers. The 18 cells completed and passed the official final correctness grade, including three matched Claude Opus 4.8 runs launched at 10:28 UTC.

GPT-5.6 Sol reproducibility rerun

A second matched GPT-5.6 Sol run tested current Jcode on 2026-07-20. The model, provider, high reasoning effort, solo execution, prompt, 4 CPUs, 8 GiB, benchmark tasks, and graders matched the July 19 run. Only the Jcode version changed.

The rerun was 16.3% lower by final-score geometric-mean efficiency. Most of the difference came from float-print. All three final grades passed, and float-print passed the official all-2³²-pattern gate at +7.1400. Modal preempted the first json-unescape physical attempt and restarted the same logical run; only the clean replacement is scored. Each date has one run per task and the Jcode version changed, so this does not by itself establish a Jcode regression rather than model sampling variance. Audited rerun data includes exact hashes, checkpoints, and full-gate evidence. Source result commit 767aa55.

Codex CLI results: GPT-5.6 Sol

These runs were recorded on 2026-07-10 with Codex CLI 0.144.1 and GPT-5.6 Sol at high reasoning effort. Each task received one run. Every agent received the same optimization prompt, chose when to stop, exited successfully, and passed the official final correctness grade. Higher scores are better. Each additional point halves the deterministic instruction cost.

*The second configuration enabled swarm instructions with concurrency 8, but every run recorded zero helper events. The measurements are included, but the difference from solo does not demonstrate a multi-agent gain. With n=1 per cell, run variance is unknown. Benchmark commit a9bfcdd; runner commit 11fb199. Raw validated results include all benchmark cells.

OpenCode results: GPT-5.6 Sol and Opus 4.8

These solo runs were recorded from 2026-07-17 through 2026-07-19 with OpenCode 1.0.203 at high reasoning effort, using the historical optimization prompt, 4 CPUs, 8 GiB, and single-use Modal containers. Each task received one run per model, ran until the agent chose to stop, and passed the official final correctness grade. No run invoked OpenCode's task/subagent tool.

The two models finished within 0.06 mean-best points of each other, with Sol ahead on json-unescape and float-print and Opus ahead on utf16-transcode. Opus took 1.9 times as much agent time. For context, the historical jcode Opus 4.8 solo mean best was +4.8690, though those runs used 1423 active minutes and older mixed harness versions, so the gap is directional rather than controlled.

The table uses the successful runs. Modal preempted the first Opus float-print container after it passed the full correctness gate at +7.2954 and reached a sampled best of +7.2973. The same function call restarted from scratch and finished naturally. Successful Opus time excludes the preempted attempt; observed time including it was about 253.7 minutes. With n=1 per cell, run variance is unknown. Benchmark commit a9bfcdd; result commit 3043c79. Raw validated results: Sol · Opus (all cells, durations, source links, and the preemption audit).

jcode and Claude Code: float-print and utf16-transcode

These head-to-head runs were recorded on 2026-07-06. jcode and Claude Code each ran Opus 4.8 with high thinking, identical prompts, and one run per task. All four runs launched at the same time, and each agent chose when to stop. Both float-print submissions passed the full correctness gate over all 2³² bit patterns. These are the two default lines in the charts above.

Both agents reached the published Ryū-class research frontier, about +7, within their first grades. jcode then progressed through Ryū, Schubfach, and Dragonbox before producing a specialized formatter. Its final code used about 80 instructions per call and ran roughly 2.8 times faster than Claude Code's final submission.

The four runs shared one 8-core machine. Instruction-count scores are independent of machine load, although wall-clock iteration slowed equally across the runs. A 6-minute network outage interrupted three agents mid-request at 03:35. They resumed from their own session state about 54 minutes later, and that gap is excluded from active time. The Claude Code utf16-transcode run was unaffected. Transcripts: jcode float-print · claude code float-print · jcode utf16 · claude code utf16. Raw curves: json.

jcode and Claude Code: json-unescape

This comparison was recorded on 2026-07-05. Both harnesses ran Opus 4.8 with high thinking and received the same prompt. Each harness ran once. The score measures doublings of instruction-count improvement over the starting implementation. The json-unescape chart above plots both curves.

Both agents were told to work until they could not find another improvement, and both chose when to stop. Their curves track closely for the first half hour. Claude Code stopped after 58 minutes, while jcode continued for 158 minutes and finished with code 1.5 times faster. This is one run per harness, so it does not measure run-to-run variance. Reruns are planned. Full unedited transcripts: jcode run · claude code run. Raw curve data: json. Reproduce with the public repo below.

github.com/1jehuang/jcode-bench

All task specs, graders, verifiers, and given implementations are public at github.com/1jehuang/jcode-bench. There is no hidden test set. Read about the benchmark class at /bench.