Skip to main content
Kodelyth ECC
engineering

Tokens Saved Is Not Dollars Saved. The RTK Ledger Lies By Omission.

RTK compresses 60-90% of shell output. Independent replays of real sessions find 3-7% actual cost reduction. The gap is cache. Here is what our ledger measures, what it does not, and how to read it honestly.

· 6 min read

A bar labelled 'tokens removed' sits at 72 percent. A second bar labelled 'dollars saved' sits at 6 percent. A dashed line connects them labelled 'prompt cache ate the discount'.

The RTK dashboard says the user saved 5,107,394 tokens across 1,285 shell calls. Sixty-four percent reduction. Pretty.

Then a careful engineer replays 614 million tokens of real Claude Code sessions through RTK, headroom, and caveman end-to-end — and finds 3.7 percent spend reduction. codepointer, 2026

Both numbers are true. Both describe different things.

What the dashboard counts

Our /api/rtk endpoint reads rtk gain --format json. The summary.total_saved field is the sum of (raw bytes RTK received) - (compressed bytes RTK returned), converted to tokens. That is what shows up on the Token Savings tab.

That number is honest about what RTK does to the pipe. It is silent about what the LLM provider charges.

Where the compressed tokens were going anyway

Anthropic charges full price for input tokens on a first read, and 10 percent of that on a cache hit. Prompt caches last five minutes. In a long coding session, the same git status, ls, and cargo check output lands in cache within the first few turns.

So RTK is proudly removing something that would have been billed at 10 percent anyway. Monday.com's engineering team spelled this out:

"RTK tends to compact output that would usually be cached cheaply anyway, so double-digit token compression may only buy us a single-digit cost cut." engineering.monday.com, 2026

Quesma's benchmarks went further: on some workloads RTK was a no-op; on others it was net negative because the agent needed one extra turn to re-ask for the summary RTK had trimmed. quesma.com, 2026

What ECC's dashboard should actually show

The honest axes are:

  1. Tokens-off-the-wire (what RTK compresses) — easy, already shown
  2. Billed-token delta (what the provider actually charges) — requires reading cache-hit flags from each turn's usage field
  3. Cost-per-accepted-task (what the user cares about) — requires an "accept/reject" signal

We ship #1. We should ship #2 next. #3 needs product work, not infrastructure.

Where RTK is clearly winning

The honest cases where RTK moves the dollars:

  • First git log of a session — large, un-cached, often > 10k tokens. RTK cuts to ~500.
  • CI output streaming into a chat tool — not usually cached, every line costs full price.
  • One-shot subagent calls — subagent prompts are not shared with the parent cache. Compressing their input saves at full input rate.
  • Devil-mode parallel sweeps — 8 agents each with their own cache, each paying full price on first read. RTK saves at full input rate on every one.

If your workflow is dominated by long interactive sessions on the same files, your billed delta will trail the compression number. If your workflow is dominated by cold starts, subagent fan-out, or parallel sweeps, the two numbers will be close.

What we are changing in ECC 2.5

  • The RTK tab will show tokens-off-the-wire and billed-token delta as separate numbers, side by side. If they diverge, you will see it.
  • Terse mode already measures output tokens correctly — the LLM response is not cached by the provider at all, so the Terse savings number is honest as shipped.
  • Combined-savings math on the landing page is being revised from "55-65% total token reduction" to "55-65% token volume reduction, which translates to roughly 15-25% billed cost reduction on long interactive sessions."

That second number is less exciting. It is also real.

What you should do

  • Keep RTK on. The tokens-off-the-wire number is still useful for ratelimit avoidance and context-window headroom, not just cost.
  • Measure your own cost-per-accepted-task by exporting the provider's usage CSV monthly. The absolute dollar line is the only number that matters.
  • Treat any "60-90% token savings" claim — ours or anyone else's — as a volume number, not a dollar number, until the publisher shows a billed-token diff.

Honest numbers scale. Marketing numbers decay. We would rather update the dashboard than keep writing a check our pipe cannot cash.

Read the full breakdown in Terse Mode and RTK integration.

Sources: