On testing a cheap LLM gateway

I wanted to see whether a cheap LLM gateway could replace OpenRouter in an AI-assisted forecasting feature we run at work. Plenty of gateways advertise the same models for much less, and switching can look as simple as changing a base URL.

The feature asks an LLM to make several judgement calls: choose a category, score the evidence, and write a verdict. I wanted to know whether changing the gateway would change those decisions. I’ll leave the cheaper gateway unnamed here.

The first trial

I started with three cases using an older model generation and a deeper second-opinion mode. The verdicts matched our baselines. Reading the gateway’s usage page, the savings looked like 52% to 63% per case.

But one case returned a forecast about three times lower. On three scoring calls for that case, the gateway put an unexplained prefix before the dated model name. Our identifier check rejected it. We asked support about the prefix and hadn’t heard back when we made the decision.

It could have been an internal routing label. We didn’t know what it meant or which upstream provider handled those calls.

The prefix didn’t appear in the later eight-case trial, and that earlier forecast shift didn’t recur. We had changed the models and mode, though, so I can’t use that to explain what happened in the first trial. I kept the results separate.

The eight-case comparison

For the next trial, I ran eight real requests twice through each route. That gave us 32 runs of the workflow, each making several model calls. All 32 completed without failures.

We used the same build, requested model, settings, and seeds, with the runs back to back on the same day. The hashes matched for prompts, tool schemas, and response schemas on comparable requests. We replayed earlier research instead of searching again, so changing web results wouldn’t affect the comparison.

Later steps could still get different inputs because they read each route’s own outputs. The gateway also had no web-search tool, so research stayed on OpenRouter. Any savings would apply to the judgement calls.

Before running the comparison, I wrote down what would make the switch acceptable: the same verdicts, forecasts within the observed repeat spread or an explanation for the differences, and model identifiers that passed our checks. I also wanted judgement calls to cost at least 40% less and take no more than 30% longer.

The forecasts changed

The cheaper gateway was 1.7 times faster by median time per judgement call. We estimated that each judgement call cost about 80% less.

I couldn’t reconcile its charges in our ledger. The response metadata and pricing information we had weren’t enough to price the calls, so we recorded their costs as unknown. I had to read the gateway’s usage page and allocate the charge to compare it with OpenRouter’s billed ledger. The saving was an estimate, and ledger totals with those unknown calls were lower bounds.

In six of eight cases, the gap between the routes’ average forecasts exceeded the repeat spread on either route. Comparing the first run through each route, three cases landed roughly twofold to fivefold apart. The gateway’s forecast could be higher or lower.

Diamonds show the percentage gap between each route’s average forecast. Bars show the larger repeat spread of the two routes. Both use middle-scenario estimates before application limits. The chart describes the observed differences; it isn’t a statistical significance test.

I compared estimates before application limits because those limits hid some differences. In three cases, the final estimates matched even though the underlying forecasts had changed.

One case also changed verdict. OpenRouter returned NO GO on both runs. The gateway returned GO on one of its two runs, and that case had its largest repeat spread.

This doesn’t tell me which route made more accurate forecasts. Eight cases and two runs per route also give me a limited view of how much either route normally varies. But the differences were enough to fail the rules I’d set for switching.

The extra input tokens

The gateway reported 64% more input tokens across the main model calls, with roughly the same number of calls on each route.

On matched workflow requests, it reported around 4,400 extra input tokens per call. The offset appeared with and without tools. Calls that stayed on OpenRouter didn’t have it.

I also tried two minimal prompts. Those produced a different offset, but the extra count was the same for both:

These are reported usage counts, with one observation per probe configuration. For “hi,” the gateway reported 4,096 tokens as cached.

A hidden instruction block could explain the extra tokens. So could an accounting difference. We didn’t inspect such a block, and the counts don’t establish that one existed or changed the forecasts.

Every call in the eight-case trial passed our model-identifier check. That check accepted the requested identifier or its dated version. OpenRouter reported a dated snapshot; the gateway reported a bare model name. I still couldn’t verify that both served identical versions.

We tried changing reasoning effort too, but with one observation per setting, I wouldn’t conclude that the gateway ignored it. We also hadn’t kept complete raw request and response bodies in our benchmark logs, which limited what we could check afterward.

The gateway also stored prompts

The gateway stored prompt content in trace logs. I had to consider that alongside its forecasts and cost.

Our prompts can carry business context, research, and tool results. I would want to know which companies process that content, how long the gateway and its upstream providers keep it, and who can read the logs. Deletion and zero-retention controls would need to cover the full route.

The benchmark didn’t answer those questions. It also didn’t establish whether the gateway used our data for training. Logging a prompt doesn’t tell us how long it’s kept or whether it’s used for training.

A provider can keep prompts without using them for training. OpenRouter’s documentation treats those as separate policies and describes zero-retention controls for specific endpoints, with limits for tools and plugins. I need to check those details on our existing route too.

The cached-token figure doesn’t tell me whose content was cached, how long it stayed there, or whether the gateway separately logged prompts and responses. The extra input tokens also don’t prove that the gateway added unseen instructions or disclosed confidential data to another party.

Next time, I would settle the data-handling terms before sending confidential inputs. Synthetic or suitably redacted prompts are enough for the initial probes.

What I would check before switching

1. Write the acceptance rules before looking at results. Decide which verdict changes, forecast differences, latency, and costs the application can tolerate.
2. Replay fixed inputs and compare request hashes. Record where later steps legitimately receive different inputs.
3. Repeat requests within each route before comparing routes. Use more runs when the decision warrants it.
4. Log reported input, cached and reasoning tokens, model identifiers, and the upstream provider where available. Investigate offsets you can’t explain.
5. Check which parameters the selected provider supports, then test their behaviour. A seed helps control a comparison but doesn’t guarantee identical output.
6. Check who processes the data, what they log, how long they keep it, who has access, and what the terms say about training and deletion. Do this before sending confidential inputs.
7. Compare costs using the same kind of accounting records where possible. Judge the saving against the application’s acceptance rules.

We kept the gateway in a clearly labelled test mode. We don’t use its runs as decision evidence, and it isn’t our replacement or backup for this feature. The forecasts failed our adoption rules, and we still had questions about the charges and prompt logs.

I want to understand those differences and where our data goes before reconsidering the switch.

References:

• OpenRouter’s zero data retention documentation: retention and training policies, endpoint controls, and exceptions for tools and plugins.
• OpenRouter’s seed documentation: limits on deterministic output.