21×
higher error rate: the weakest setup against the best
The model you pick is the whole game.
Same pages, same instructions. The weakest setup put 30.6% of numbers in the wrong place; the best, 1.5%.
Carriage model benchmark
Before AI touches your numbers, it has to earn the job.
Carriage runs deal and portfolio-company workflows across many systems, tools, and AI models. We don’t pick models on reputation. We test them on real work and give each step to whatever does it best. This is one of those tests: reading the charts and tables in a CIM.
14 setups · 4 models · Updated September 22, 2026Share on LinkedIn
Out of every 100 numbers on a hard CIM page
1 in the wrong place98.5% placed correctly
31 in the wrong place69.4% placed correctly
Right place Wrong placeSame pages, same instructions. Only the model setup changed.
What we tested
AI reads the digits fine. Putting them in the right place is the test.
Every CIM, lender presentation, and board deck reaches us as page images. Before Carriage can build a model or draft a memo, it has to turn each chart and table into clean data. A misplaced number is worse than a missing one, because it looks right.
Illustrative example
On the page
Revenue by segment ($M)
Products Services
What the AI writes
Right
| $M | FY22 | FY23 | FY24 |
|---|---|---|---|
| Products | 42 | 48 | 55 |
| Services | 18 | 23 | 29 |
Wrong, but looks right
| $M | FY22 | FY23 | FY24 |
|---|---|---|---|
| Products | 42 | 48 | 29 |
| Services | 18 | 23 | 55 |
Every digit is correct in both tables. In the second, two numbers sit in the wrong cell, and FY24’s revenue mix is now backwards. Our test only counts a number when it lands in the right place.
- Stacked barsWhich segment is which, when the labels sit inside the bars.
- Donut chartsMatching every slice to its color in the legend.
- WaterfallsFollowing a bridge step by step from one total to the next.
- Two-tier tablesWhich column a number belongs to when headers are stacked or flipped.
What we found
Same pages. Same instructions. Very different results.
We ran 14 setups — 4 Google Gemini models, each at several “thinking” levels — through the same hard CIM pages, and checked each answer against a key verified by hand.
9.3×
higher error rate, to save $1.45 per 1,000 pages
The cheapest option is not a bargain.
Gemini 3.5 Flash-Lite · Low was the lowest-cost setup. Against the winner it saved $1.45 per 1,000 pages and misplaced numbers 9.3× as often. On a CIM, that trade is never worth it.
4.4×
the winner’s cost, for lower accuracy
Paying more did not buy accuracy.
The most expensive setup, Gemini 3 Flash · High, cost 4.4× as much as the winner and took 6.4× as long per page, yet placed fewer numbers correctly: 93.4% against 98.5%.
The winner
Gemini 3.8 Flash at low thinking
- Numbers in the right place
- 98.5%
- To read a page
- 3.3 sec
- Per 1,000 pages
- $3.14
Even the winner misplaces about 1.5 numbers in 100. That is why reading is only one step in a Carriage workflow.
Why it matters
Reading the page is step one. Carriage runs the whole workflow.
No single model is best at every step. Carriage is a custom-built system that connects your files, systems, and tools, and gives each step of a workflow to the model, tool, or rule that measurably does it best.
- 01Take in the workEmail requests, CIMs, data rooms, invoices, and exports from source systems.
- 02Read every pageCharts, tables, and scans turned into clean, structured data.This benchmark
- 03Check against the sourceNumbers matched to systems of record, contracts, and prior periods.
- 04Do the analysisData cuts, models, research, and comparisons in your formats.
- 05Route the exceptionsAnything uncertain goes to a person, with the evidence attached.
- 06Deliver finished workFormula-linked Excel, Word memos, PowerPoint decks, and dashboards.
At the fund
From a CIM to an IC draft
Forward the CIM and data room by email. Get back a first read, a formula-linked LBO in your template, and a draft IC deck.
At the portfolio company
From an invoice pile to pay, review, or dispute
Subcontractor invoices, timesheets, and backup are matched to work orders and contract terms, then routed to the right decision.
The full results
All 14 setups, side by side.
A setup is one model at one “thinking” level: how long it may reason before it answers. More thinking costs more and takes longer. Rank by accuracy, cost, or speed, and hover or tap a row for detail.
- Gemini 3.8 Flash
- Gemini 3.7 Flash
- Gemini 3 Flash
- Gemini 3.5 Flash-Lite
Every setup, ranked
Share of numbers placed in the right row and the right period. Higher is better.
Accuracy against cost
Each point is one setup. Higher is more accurate; further left is cheaper. The top-left corner is where you want to be.
Every metric, every setupFull table and definitions
| # | Model · setting | Accuracy | Pages fully right | Numbers read | Invented / page | Page # captured | Avg s / page | Slowest 5% (s) | Thinking tokens | $ / 1,000 pages |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemini 3.8 Flash · Low | 98.5% | 52% | 100% | 0.9 | 100% | 3.3 | 5.1 | 0 | $3.14 |
| 2 | Gemini 3.8 Flash · High | 98.4% | 67% | 100% | 0.8 | 100% | 12.6 | 19.5 | 2,253 | $11.50 |
| 3 | Gemini 3.8 Flash · Medium | 98.0% | 52% | 100% | 1.1 | 100% | 9.0 | 13.7 | 987 | $6.74 |
| 4 | Gemini 3.7 Flash · Low | 97.7% | 55% | 100% | 1.1 | 100% | 4.6 | 9.5 | 111 | $3.59 |
| 5 | Gemini 3.7 Flash · High | 97.3% | 64% | 100% | 0.8 | 100% | 10.9 | 14.7 | 1,632 | $9.16 |
| 6 | Gemini 3 Flash · Medium | 96.5% | 62% | 100% | 0.8 | 100% | 15.4 | 22.9 | 2,526 | $9.91 |
| 7 | Gemini 3.7 Flash · Medium | 96.0% | 52% | 100% | 1.1 | 100% | 8.8 | 12.0 | 947 | $6.64 |
| 8 | Gemini 3 Flash · High | 93.4% | 57% | 98% | 0.9 | 93% | 21.0 | 56.0 | 3,810 | $13.95 |
| 9 | Gemini 3 Flash · Minimal | 91.2% | 43% | 100% | 1.4 | 100% | 3.9 | 6.0 | 0 | $2.40 |
| 10 | Gemini 3 Flash · Low | 90.6% | 45% | 100% | 1.4 | 100% | 3.4 | 5.5 | 0 | $2.39 |
| 11 | Gemini 3.5 Flash-Lite · Medium | 89.3% | 45% | 100% | 1.0 | 100% | 3.3 | 9.4 | 135 | $2.06 |
| 12 | Gemini 3.5 Flash-Lite · Low | 86.2% | 38% | 100% | 1.0 | 100% | 2.5 | 4.0 | 0 | $1.69 |
| 13 | Gemini 3.5 Flash-Lite · High | 84.8% | 40% | 100% | 0.8 | 100% | 9.7 | 12.6 | 1,667 | $5.91 |
| 14 | Gemini 3.5 Flash-Lite · Minimal | 69.4% | 29% | 100% | 1.4 | 100% | 2.5 | 3.7 | 0 | $1.78 |
- Setting
- How long the model may “think” before it answers, from Minimal to High. More thinking costs more and takes longer.
- Accuracy
- Share of checked numbers placed in the right row or series and the right period, averaged across pages.
- Pages fully right
- Pages where every checked number landed in place and nothing was invented.
- Numbers read
- Share of numbers that appear somewhere in the output, placed correctly or not.
- Invented / page
- Numbers the model wrote that are not on the page at all.
- Slowest 5%
- 95% of pages finish within this time; the rest take longer.
How we measured
A fair test, run the way production runs.
- Hard pages on purpose
- Chart-heavy CIM pages: stacked bars, donuts, waterfalls, and dense, multi-level tables.
- Production settings
- The prompt, image preparation, temperature, thinking level, and retry behavior are the ones our live pipeline sends.
- Hand-verified answer keys
- Every number on each page and how often it appears, plus headings, legends, row names, and the printed page number.
- Accuracy means placement
- A number counts only when it sits in the right line item or series and the right period or category, whether the model wrote a table or a list.
- Invented numbers are tracked
- Any value a model wrote that is not on the page counts against it. Repeating a real value does not.
- Real cost and speed
- Cost is list price from measured usage, with thinking billed at the output rate; prices checked September 22, 2026. Speed is wall-clock time per page, retries included.
Have a workflow your team still does by hand? Let’s measure it.
Send us a deal workflow or a recurring portfolio process. We will show you which steps Carriage can run, how we would prove it works, and what it is worth.
Book a free scoping call