In Opus is cheaper than Haiku my own token-saving tip didn't survive my evals. I ended with a warning: if anyone tells you their setup "cuts agent costs by 66%," ask whether they mean tokens or dollars.
Then I went and looked at the most popular one, and found something more interesting than a gotcha. 🗿
44% fewer tokens, 0% saved
Caveman makes coding agents talk like a caveman to save tokens. It started as a joke in April 2026 and now has over 111,000 GitHub stars.1 It got a lot of developers, me included, thinking seriously about where their agent's tokens go.
In July, the maintainer of LemonCrow tested it: 20 engineering prompts, 5 repetitions each, three separate runs, 300 runs per arm, and provider-billed costs.2
| 300 runs per arm | Baseline | Caveman | Difference |
|---|---|---|---|
| Billed output | 369,825 | 206,894 | −44.1% |
| Cache reads | 6.86M | 5.85M | −14.7% |
| Initial cached context | 880K | 1.34M | +52.2% |
| Cost | $26.01 | $26.02 | +0.06% |
The output savings were real and repeatable. But the skill's instructions added about 1,500 tokens of context to every run, and an agent's bill is mostly input and cache, not output. The two cancelled out.
Fewer words. Same bill. 🥲
It's my first finding again: count dollars, not tokens.
This tested the response skill only. It doesn't tell us whether Caveman's newer proxy or router saves money. Those are different strategies, and they deserve their own evaluations.
Credit where it's due
The issue was closed on October 8, referencing a commit that removed claims nobody could reproduce. A "46%" compression figure became a reproducible 33.2%, and unbenchmarked subagent savings were deleted. It also added a test that fails if those numbers ever come back.3
Their Honest Numbers page is more candid than most:4
- Against a plain
Answer concisely.instruction, the default Caveman level cut output by 3% at the median. - One tester found that output was 0.3% of all tokens across 662 Claude Code sessions. Caveman's effect on that bill was within ±0.2%.
- "If caveman lose for your workload, this page tell you to turn it off."
Respect. 🤝 The README is honest in the same way. Its headline "54% cheaper, same score" for their new Auto router comes with the note that the run is half done, the score gap is within noise, and most of the saving comes from lowering effort on easy tasks.5 Headlines travel further than fine print, but that's true of every README, mine included.
Everyone is building one
Caveman isn't alone, and it made the whole space more visible. There are output styles, compression hooks like precc, routers, and my own beholdr-prism-mini. LemonCrow's maintainer was benchmarking their own tool in the same run.
Building one is the cheap part. Finding out whether mine helped took repeated runs on two repositories, fact lists written before the first run, and a result I didn't expect.
Who actually runs the evals?
Makers run evals, and Caveman has published plenty. I doubt most users reproduce them on their own workloads.
That matters, because the result depends on how you work: your harness, your CLAUDE.md, your other tools, your cache behavior, your tasks. One tester already asked for terse replies in their CLAUDE.md, and Caveman cut only another 8% on top.4 On GitHub Copilot, which bills per request, a shorter answer costs exactly the same.
Caveman's own advice is right: run the same task with and without it, then compare the bill.4 But a fair A/B needs many runs, because two identical runs of mine differed by about 15%. It also needs a way to check the answers stayed correct. That costs money and a weekend you'd rather spend building.
So we skip it and trust the maker's numbers, whoever the maker is. Even when the maker is honest, they measured their workload, not yours.
Evidence is expensive, so we borrow it. Trust me bro. 🤣
The feedback loop is the moat
The code isn't the moat. Honestly, it never was. I could rebuild a terse-output skill in an afternoon. What I can't rebuild in an afternoon is the distribution behind a project with over a hundred thousand stars, and the chance to learn from real-world usage at a scale I can't reach alone. That learning is just engineering: form a hypothesis, test it, improve, repeat. Without that loop, you're just copy-pasting popular features into your own tool.
Stars aren't users, though, and telemetry isn't evidence yet. Caveman collects two different kinds of data. The CLI sends aggregated usage stats by default: commands, token counts and an install ID, but not prompts or code. The Auto router is opt-in: once you sign in, each request sends your latest ask with whatever context your agent attaches, and on the free plan Caveman "may keep it to improve routing."6
Neither tells you whether a task succeeded, or what it would have cost without Caveman. Their own /caveman-stats reports savings as unknown for exactly that reason.4 There are three kinds of evidence here:
- Synthetic benchmarks show whether a technique works under controlled conditions.
- Production telemetry shows what happens across real workloads.
- Outcome-linked experiments show whether an intervention actually improves results.
The third is the valuable one, and it isn't free. It needs a real outcome measure, a fair comparison, disclosure to users, and someone who pays for the extra runs. But with enough users, it becomes possible, and Caveman, with its user base and telemetry already in place, is in a better position than almost anyone to try it. And the prize isn't an optimizer that always saves tokens. It's one that learns when not to compress, route or summarize.
As I wrote in AI is shrinking the minimum viable organization, "we have data" by itself isn't a moat. The loop is: turning real usage into better decisions than anyone else can make. Distribution feeds it, and a solo builder can't easily reproduce it.
So if you want to build a better general optimizer, how will you show it's better without comparable evidence?
Do you even want a general tool?
A general tool has to work on everyone's workload, and that's the expensive evidence. A personal tool only has to work on yours, and that evidence is affordable.
My Prism Mini rules came from a handful of tasks on two repositories. That's not enough to say anything about your setup. It was enough to tell me when to delegate in mine.
Open-source projects become a parts catalog, and Caveman is a good one. Read how it compresses tool output, borrow the idea, rebuild it for your stack, and test it on the only workload you care about. I already have tools I built myself. I know how they work, and when they break, I know where to look.
You give up reach and stars. You get a tool you understand and evidence you can trust.
Evidence beats confidence
That's the motto at Beholdr Labs, and this is why.
Software is getting cheaper to build. Proving that it works is not.
The more general your claims, the more evidence you need, and the people who can least afford to evaluate their claims may be the ones who most need to. But I don't need to prove an optimization works for everyone. I only need to know whether it works for me.
Ask the three questions: tokens or dollars, how did you check the answers, and how many runs? Then run your own evals. Yes, they cost money. Yes, you might find out your brilliant optimization makes things worse.
That's the point. 🤣🧪
Footnotes
-
JuliusBrussee/caveman, README and star count as of October 14, 2026. ↩
-
Caveman issue #727, "Benchmark results and three questions - 0 savings E2E", with the method and raw results. It tested the July 2026 skill on
claude-opus-4-8. Caveman 3 has since rewritten the rules, so treat it as evidence about the approach, not today's file. ↩ -
Commit
4ec0d9b, "docs: replace unreproducible compress and cavecrew figures", October 6, 2026. ↩ -
Caveman, "Honest Numbers". The 3% figure is from n=10 on the 3.1.0 skill text; the 662-session and
CLAUDE.mdfigures are from issue #733. ↩ ↩2 ↩3 ↩4 -
Caveman README, "Auto: 54% cheaper, same score": 85 coding tasks, one attempt each, compared with Opus 5.5 at xhigh effort. ↩
-
Caveman README, "Privacy". Telemetry can be turned off with
caveman telemetry offorDO_NOT_TRACK=1. ↩