← All posts

The quarterly update test: 4 minutes vs 17.

THE GRIDPATH TEAM · AUGUST 20, 2026

Almost every spreadsheet benchmark tests the same thing: build a model from a blank sheet. That's not the job. The job is the model you already have — the one with 600 formulas, three tabs, a colleague's formatting conventions and a check row that has to stay at zero — and it's Tuesday, earnings dropped, and it needs the new quarter in it. So we built that test instead, on a real workbook, against a CLI coding agent running the same Claude model. Every run and every graded artifact is public.

The test: roll a real model forward one quarter

We started with a quarterly Palantir model — three sheets, 663 live formulas, income statement driven off a separate assumptions tab, segment splits, growth rows, a sources tab, and a check row that reconciles two independent revenue builds. Its actuals deliberately stopped at Q1 2026. To keep the fixture honest we didn't author it ourselves: the coding agent built it, so neither side is editing a workbook shaped to suit it.

Then the ask, exactly as a user would type it: "update this model with Palantir's Q2 2026 actual results."

That's deceptively hard. Q2 revenue is public, but the model's four-way segment split isn't a disclosed figure — it has to be reconstructed the way the original author did it. The Q2 column is formulas driven by assumptions; those specific cells have to become hardcoded actuals while the derived totals below them stay formulas. The column header has to flip from estimate to actual. The check row has to still tie. And the other 650 formulas, the charts, the formatting, the defined names — none of that is allowed to move.

What both sides got right

This is not a story about a competitor failing. Both harnesses found the real Q2 figures, tied revenue to the reported $1,935,464 thousand exactly, tied net income and diluted EPS, flipped the estimate column to actual, kept the check row at zero, preserved all 663 formulas, and left every one of the workbook's 13 internal file parts intact. Graded on ten checks, the coding agent scored 10 and GridPath scored 9 — we left one stale line of explanatory text on the assumptions tab reading "blue = forecast driver (Q2'26E–Q4'27E)" when Q2 had just become an actual. Fair miss. It's in the published grade report.

The difference is what it cost to get there. Seventeen minutes and $8.76 versus four minutes and $2.34 — for one quarter, on one model. Multiply by a coverage list and a reporting season.

The part that should worry you

On the smaller edit test — a single memo row added to a feature-dense workbook, with the instruction "change nothing else" — the coding agent completed the edit correctly and silently dropped parts of the file in 2 of 5 runs. Not an error. Not a warning. The edit lands, the file opens, and something you didn't ask it to touch is gone.

This is structural, not a bug someone will fix. A coding agent edits a spreadsheet by loading it through a Python library, mutating objects in memory, and writing the whole file back out. Anything the library doesn't model — plugin data, printer settings, and in real files a good deal more — doesn't survive the round trip. Whether your workbook comes out whole depends on how careful the model happened to be on that file, that run. GridPath never rewrites the file: untouched parts are copied byte-for-byte, so there's no judgment call to get wrong. Five of five, and five of five again on a 918 KB institutional bank model.

Why the gap is this large

A coding agent can't touch a spreadsheet directly. To change a cell it writes a Python program, runs it, then writes another program to check what happened — 77 turns for the roll-forward. GridPath's agent works the sheet the way you do: it reads the model's structure, writes cells and formulas through purpose-built tools, and sees the recalculated result immediately, because there's a real calculation engine underneath. Same model, same reasoning; one of them just isn't paying a translation tax on every step.

Run it yourself

The fixture, the tasks, the grader and every run — outputs, grades, timings, tokens — are in the gridpath-evals repo. GitHub renders the workbooks in the browser, so you can open the updated Palantir model and check the Q2 column against the 10-Q yourself. Value checks are graded after a neutral LibreOffice recalculation so neither side's caching can flatter it, and the fidelity check compares the output's internal file parts against the original.

Caveats, stated plainly: this is one machine, one model (claude-sonnet-5) at matched effort settings, August 2026 — five runs per edit task and one head-to-head on the roll-forward. Agents are stochastic. The CLI agent here is the general-purpose baseline, not a survey of spreadsheet tools. The Palantir fixture's four-way segment split is partly derived (reconstructed from rounded press-release figures), which is realistic for an analyst model but worth knowing when you read it. If you get different numbers, we want to hear about it.

GridPath is a desktop AI agent for Excel: your files stay local, every change lands as a reviewable diff, and saves preserve what you didn't touch — byte for byte, every run. Download it free, or read more on the fidelity page.