Don't trust the number. Rerun it.
Every savings figure we publish comes with the exact commit it was measured against, so it can be rerun. A prospect's engineer, a due-diligence analyst, a competitor: any of them can pull the same public dataset, run the same script, and check our arithmetic. That's the real claim sitting underneath the audit chain, the diagnostic, and everything else here. You don't have to trust our number. You can recompute it. This page holds a benchmark to that same standard for the first time.
The headline result
In an independently reproducible test on public conversation data, routing alone cut inference cost by an average of 74% across four separate test runs, drawn independently of each other (range: 69–79%). Every call was a real API request, and every cost came from the token counts the provider actually returned rather than an estimate.
What this test doesn't show
This measured routing on public conversation data, not your traffic. The dataset (OpenAssistant/OASST2) skews toward short, simple, single-turn questions. It reads more like a Q&A archive than an enterprise workload with long prompts, multi-step agent chains, and RAG context. Short questions route to the cheap tier more often than typical production traffic does, so 74% is a real number for this specific test, not a forecast for any one customer's traffic mix.
Caching barely moved the needle in this test (0.46%), and that's expected. The sample was 432 distinct strangers each asking one unrelated question, so there's almost no natural repetition to catch. Real production traffic looks nothing like that: the same customer re-asking things across a session, or many users hitting the same handful of FAQs. This benchmark happened to run on the one kind of traffic where caching has the least room to help. It doesn't mean caching doesn't work. It means caching should be measured on your traffic, not a public dataset.
That's what our free diagnostic actually measures: your real cache-hit rate and your real routing split, on your own logs.
Results
| Sample | n | Routing-alone | Caching-alone | Combined |
|---|---|---|---|---|
| 1 | 110 | 78.51% | 0.0% | 78.51% |
| 2 | 110 | 69.46% | 0.91% | 69.46% |
| 3 | 108 | 74.4% | 0.93% | 74.4% |
| 4 | 104 | 74.12% | 0.0% | 74.12% |
| Mean | 432 | 74.12% | 0.46% | 74.12% |
Four separate numbers, not one lucky run. The real variance is right there instead of being collapsed into a single flattering figure.
In dollar terms, pooled across all 432 prompts: routing every request through the frontier model would have cost $4.625. With InferLayer's routing turned on, those same 432 prompts cost $1.192, a 74.22% reduction that lines up with the 74% mean above. It isn't a blanket discount. InferLayer classifies each request by complexity and sends only the complex ones to the frontier model. Simple and medium-complexity questions, which make up most of this dataset, route to a cheaper tier automatically rather than defaulting to the expensive one.
Quality check: does the cheaper model hold up?
Cost savings mean nothing if the cheaper tier's answers are worse. On the same 432 prompts, we generated a routed response and a frontier response for every prompt, then had an independent judge score each one blind and separately, with no visibility into the other response and no idea which one was "routed." The judge was Claude Sonnet 5, a different vendor from both models being compared, which rules out favoring one vendor, or one position in a side-by-side layout, from the start.
Result: across 430 usable head-to-head comparisons (2 of 432 came back unparseable, and we're not hiding that), InferLayer's routed responses tied or beat the frontier model in over 70% of cases. We checked reliability by having the judge re-score the same response three separate times. It agreed with itself 78% of the time, and that number held from a 50-prompt pilot all the way up to the full 432-prompt run. It didn't look good at 50 prompts and then quietly fall apart at scale, which is the failure mode worth watching for in any judge-based benchmark.
This measures whether responses were judged comparably good, not whether they were identical. On a finer-grained 1–10 scale, frontier still holds a small, measurable edge. The full raw scores, the two earlier judging methods we tried and abandoned because they weren't reliable, and every caveat that goes with them are documented in full, not smoothed over for this page.
How this is different from our existing "~50%" figure
This is a new, independent measurement. It isn't an update to the ~50% cost reduction figure quoted elsewhere on this site: the baseline model, the dataset, and the methodology are all different. Both numbers are real. They measure different things, and we never merge them into one claim. If the two don't match when you compare them, that's why, and it isn't an inconsistency that needs explaining away.
Methodology, in brief
432 real, distinct first-turn prompts from OpenAssistant/OASST2 (Apache 2.0, licensed for commercial use), split into four disjoint samples stratified by language. Each sample ran twice: once with every request forced through the frontier model as the baseline, once through InferLayer's real default routing and live semantic cache as the measured condition. Every cost is the real token count the API returned for that specific call, never estimated from a pricing table. The exact methodology, the commit it was measured against, and the exact prompts used are all documented in full.
Two real engineering issues came up while running this benchmark, including a config-reload bug that briefly mispriced results before we caught it. We wrote both up alongside the fix instead of quietly correcting and moving on. A benchmark's credibility rests partly on showing the mistakes that got caught, not only the numbers that survived them.
What this benchmark does not claim
The quality check above compares routed responses against the frontier model on the same prompts. That's the comparison that actually matters for a "similar quality, lower cost" claim. A weaker, secondary comparison against OASST2's own crowdsourced human answers was run under our first pairwise judging method, and routed responses did very well in it. We later found that method had a position-bias problem (see "two earlier judging methods" above) and replaced it with the independent single-answer method used in the quality check itself. We never re-ran the human-reference comparison under the better method, so we're not citing a number for it here, only flagging that an earlier, superseded result exists. It was always the weaker comparison anyway: crowdsourced answer quality varies too widely to be a real bar, and only about half of OASST2 entries have a usable human reference at all. None of this, or the cost figures above, predicts what any one customer will see on their own traffic. Both are measurements on one public dataset, and we're presenting them as exactly that.