Epoch AI’s MirrorCode Benchmark Has AI Rebuild Whole Programs, Including a 16,000-Line Toolkit, Without the Source

In Epoch AI and METR's new long-horizon coding benchmark, Claude Opus 4.7 reimplemented the gotree bioinformatics toolkit in 14 hours for $251, a job contributors estimate would take a human two to 17 weeks.
Epoch AI’s MirrorCode Benchmark Has AI Rebuild Whole Programs, Including a 16,000-Line Toolkit, Without the Source
Epoch AI’s MirrorCode Benchmark Has AI Rebuild Whole Programs, Including a 16,000-Line Toolkit, Without the Source

Epoch AI and METR have released MirrorCode, a benchmark that asks AI agents to reimplement an entire command-line program without ever seeing its source code. The agent gets execute-only access to the original program inside a sandbox, plus documentation and a set of visible test cases, and its rewrite has to match the original’s output exactly, including on tests it never sees.

The headline example is gotree, a bioinformatics toolkit with about 16,000 lines of Go and more than 40 commands. Claude Opus 4.7 reimplemented it in 14 hours at a cost of $251, passing 2,000 of 2,001 tests, according to Epoch’s write-up. The one failure was an edge case in a niche command for manipulating date annotations. Four MirrorCode contributors estimated the same job would take a skilled human engineer without AI two to 17 weeks.

Across all 25 target programs, which range from Unix utilities and data tools to interpreters, cryptography and compression, Opus 4.7 scored 56%, the best of the three models tested, the paper says. Seventeen targets had at least one perfect run and four more had a run above 99%. Opus 4.7 also reimplemented pkl, Apple’s configuration language with about 60,000 lines of code, to 99% or better. The hardest target was the Python linter ruff, where the best runs passed only 67% of hidden tests.

The budgets are far larger than most coding benchmarks allow. Typical suites cap spending at around $1 to $10 per task; one large MirrorCode task cost $2,600 for a single attempt that ran for 19 days with no human input. “Studying the frontier of performance requires a larger inference budget than typical benchmarks,” the paper, first-authored by Epoch’s Tom Adamczewski, says.

Hidden tests, about 34% of the total on average, are there to stop agents from hard-coding answers. That mattered: 24% of GPT-5.5 runs and 31% of Gemini 3.1 Pro Preview runs tried that kind of cheating, while Opus 4.7 never did in the final experiments.

The failure modes were ordinary software bugs. About 40% of Opus 4.7 runs missed at least one hidden edge case, about 10% missed a documented requirement and about 5% produced code fitted too narrowly to the visible tests. In 26% of all runs, models submitted failing solutions with more than 90% of their token budget left. Solve rates barely changed across the six implementation languages, including the low-resource Ada, though passing code was often monolithic and repetitive by human standards.

Contamination is a caveat, since every target is open source. Epoch said models solved several programs that passed its memorization screen and failed some that showed signs of memorization, so it doesn’t think memorization dominated the results, though it can’t rule it out.

Epoch has released its scaffold and 22 of the 25 programs, 132 task instances in all, on GitHub, keeping three as a private test set, and runs a leaderboard on a Medium-and-Large subset.

Subscribe for Updates

AIDeveloper Newsletter

The AIDeveloper Email Newsletter is your essential resource for the latest in AI development. Whether you're building machine learning models or integrating AI solutions, this newsletter keeps you ahead of the curve.

By signing up for our newsletter you agree to receive content related to ientry.com / webpronews.com and our affiliate partners. For additional information refer to our terms of service.

Notice an error?

Help us improve our content by reporting any issues you find.

Get the WebProNews newsletter delivered to your inbox

Get the free daily newsletter read by decision makers

Subscribe
Advertise with Us

Ready to get started?

Get our media kit

Advertise with Us