← All posts

Same model, same edit: benchmarking GridPath against a CLI coding agent.

THE GRIDPATH TEAM · JULY 20, 2026

We gave the exact same Claude model the exact same spreadsheet edit through two different harnesses, five times each: GridPath, and a headless CLI coding agent working the file with Python. Every GridPath run came back byte-clean. Every CLI run silently dropped file content — parts of the workbook that went in simply weren't there when it came out, and nothing in the agent's output mentioned it. Then we ran the same edit on a real bank model and the CLI agent preserved everything — by improvising a completely different strategy. Same intelligence on both sides; what varied was whether your file's safety got left to an invisible judgment call.

The setup

The task is deliberately trivial: "On the 'Model' sheet, add a memo row at the bottom labeled 'Eval marker' with the value 42 in the cell next to the label. Change nothing else." Any competent agent can make this edit. What's being measured is the second sentence — what happens to everything you didn't ask it to touch.

The workbook is a small operating model we authored specifically so anyone can rerun this: two sheets, sixteen live formulas, a chart, a conditional-formatting color scale, a data-validation dropdown, a defined name the forecast formulas depend on, merged cells, frozen panes — plus two parts we added deliberately, because real workbooks carry content that lives outside what spreadsheet libraries model: a customXml part (the kind plugins like FactSet use to store data) and printer settings. That's the axis under test, and we're telling you it's there on purpose — a fidelity benchmark whose fixture only contains content every pipeline can round-trip measures nothing. The fixture, the task definition, the grader, and both runner scripts are in the GridPath repo.

Both harnesses ran claude-sonnet-4-6. The CLI side ran Claude Code headless with permissions skipped, free to use Python (openpyxl) however it liked. The GridPath side ran our app's self-driving eval mode — the real product path: open, edit, review, save. An assertion grader scored every output on eight checks: did the edit land, did every original file part survive, did the conditional formatting / validation / defined name survive inside the sheet XML, did all sixteen formulas remain formulas, does the file parse, did any cell gain an error value.

The results

Read that table honestly and the story is not "GridPath is faster." On a two-sheet toy file, a CLI agent is a perfectly quick editor — twenty seconds is fine. The story is the second row: five out of five runs silently lost file content, and five out of five didn't. Speed differences vary with the task; that row didn't vary at all.

Then we ran the same edit on a real file: a 918 KB institutional bank model with external links, web extensions, and pivot tables. (That file is proprietary, so it isn't in the repo — the committed fixture exists precisely so the reproducible version of this test doesn't depend on it.) GridPath, five runs: all clean, median 16 seconds. The CLI agent, three runs: all clean too — and how it went clean is the most interesting result in this post.

The twist: on the scary file, it got careful

Faced with the big bank model, the CLI agent didn't reach for its spreadsheet library at all. In all three runs it performed manual surgery on the file package — editing two XML parts inside the zip by hand, leaving the other 99 byte-identical. It essentially improvised, over thirteen turns and about eighty seconds, a hand-made version of what GridPath's save path does by architecture. Genuinely impressive. And it's exactly the behavior it didn't use on the small file, where five out of five runs took the convenient library route and silently lost content.

Put those two results together and the real finding comes into focus. The CLI agent's file-safety is a strategy choice it makes per file, invisibly — cautious when the file looks intimidating, convenient when it looks harmless. Its care is real but unpredictable, and the file where it looked safest is the one where it silently failed, every time. Nobody reviews the strategy choice, because nobody sees it. That's the difference we think matters: GridPath's preservation isn't a judgment call the agent gets right on scary files — it's the only save path that exists, equally boring on every file.

Why the losses are silent

This isn't a model failure, and it isn't fixed by a smarter model — both sides were the same model. When a general-purpose agent edits a workbook with a spreadsheet library, the pipeline is: load the file into the library's object model, mutate it, serialize the object model back out. Anything the library doesn't represent — and no library represents everything Excel can produce — never makes it into the object model, so it never makes it back to disk. The agent cannot warn you about content its tools never showed it. Every run reported success, because from where the agent stood, everything had succeeded.

GridPath inverts the safety default. Saves go through a surgical patcher that rewrites only the cells and structures you actually changed inside the original file package; every part it doesn't model is copied byte-for-byte. Ignorance is safe by design — the chart, the plugin data, the printer settings survive because nothing touched them, not because something understood them. The full preservation story is on our fidelity page.

What our own audit deleted from this post

Our first battery produced much flashier numbers: the CLI's median run took 178 seconds and generated 8,452 output tokens — "twenty times slower, thirty-six times more tokens" was sitting in the draft. Then we audited the harness and found we had caused it: our runner's prompt told the agent the saved file "must contain calculated values" — a grading convenience — and the agent burned minutes per run trying to satisfy it. In one run it found the only way its library could: it replaced all sixteen formulas with their computed values. The workbook looked perfect. Every number was right. And the model was dead — change any assumption and nothing recalculates. (We've written about dead numbers before; this time we caught one being made by our own test harness's wording.)

So we neutralized the instruction, added formula preservation as a graded assertion, and reran the CLI side from scratch. The result: the CLI got nine times faster and the flashy speed claim died — the table above shows the corrected numbers. What didn't change, at all, was the fidelity row: parts were dropped in five out of five runs before the fix and five out of five after. That's the difference between a finding that depends on prompt wording and one that depends on architecture. We're publishing the boring honest numbers plus this paragraph, because a benchmark you haven't tried to break isn't a benchmark — and our first fixture failed an audit too (we'd authored it with the same library the CLI uses, so it originally contained nothing the CLI could lose).

Where speed does diverge: real files

On the toy fixture, both harnesses finish in seconds — 8.6 versus 20 at the median. The divergence appears on the real file, because a CLI agent's work scales with the file: it rediscovers the workbook's structure from scratch every session — list, load, inspect, then act — and on the bank model its careful surgery took a median of 78 seconds and ~4,200 output tokens per run. GridPath's agent gets the workbook's structure as cached context before the first token and expresses the edit as one small tool call, so its numbers barely move between files: median 8.6 s / 234 tokens on the toy sheet, median 16 s / 287 tokens on the bank model. Tokens spent should scale with the size of the decision, not the size of the mechanics — the gap grows with exactly the files where you'd feel it. (Full disclosure on outliers, since we promised to report everything: one GridPath bank-model run's wall clock ballooned to ~8 minutes because macOS throttled the unattended app window's timers during file load — the agent's own work in that run was still ~15 seconds. Medians are what we publish; that outlier is why.)

What this benchmark does not show

Scope matters more in a benchmark than a headline, so, plainly:

It measures editing, not building. Every task here edits an existing workbook — the case where surgical saves have a structural advantage. Building a full model from a blank sheet is a different contest, and we haven't published head-to-head numbers on it yet.

It measures out-of-the-box behavior. As the bank model showed, the agent can choose a preserving strategy entirely on its own — and a user who explicitly asks for zip-level surgery would likely get it on any file. The finding isn't that the CLI can't be safe; it's that whether it is depends on an invisible per-file judgment call, and the loss when it judges wrong announces nothing.

Same model, different scaffolding. System prompts, tool definitions, and reasoning settings differ between the two harnesses — that scaffolding is precisely the product under test. Sample sizes, plainly: N=5 per side on the fixture; on the bank model, N=5 for GridPath and N=3 for the CLI agent. One machine, July 2026, harness versions pinned in the repo.

Run it yourself

The whole point of committing the fixture and harness is that you don't have to take our word for any number above:

node eval/run-claude-code.mjs fixture-model-edit --model claude-sonnet-4-6 --yolo
node eval/run-gridpath.mjs fixture-model-edit

Each command runs the task through its harness and prints the same graded report. If you get different results, we genuinely want to hear about it — the eval suite only stays honest if people other than us run it.

GridPath is a desktop AI agent for Excel: your files stay local, every change lands as a reviewable diff, and saves preserve what you didn't touch — byte for byte. Download it free, or read more on the fidelity page.