When a Successful Request Is Still a Failure
Imagine you have been talking to the same character for weeks. Their voice is familiar. Their language is consistent. Their replies feel sharp and alive.
Then, without changing the character or the model, something feels wrong.
The next reply takes ten seconds to begin. A feature that depends on structured output suddenly breaks. The character starts rambling, inserts fragments from another language, or responds in a style that seems to belong to a much weaker model.
From the server's point of view, nothing failed. The API returned 200 OK. Tokens arrived. The request was billable.
From the user's point of view, the model just got worse.
That gap is why we built a provider canary and routing quarantine system for Reverie.
One Model Name, Many Serving Environments
OpenRouter is our primary model gateway. One of its strengths is that a single model can be served by many independent inference providers. If one provider is unavailable, another can take the request. That gives us more capacity, competitive pricing, and better resilience than depending on a single endpoint.
It also creates a hidden variable.
The model name may stay the same, but the infrastructure serving it can change from one request to the next. Each provider may use a different inference engine, hardware configuration, quantization level, parser, chat template, queue, or additional policy layer.
OpenRouter's provider-routing documentation explains that its default routing considers recent availability and price, with other providers available as fallbacks. Those are important signals—but an endpoint can be online, inexpensive, and still produce an unacceptable result for a particular product.
OpenRouter has also written publicly about measurable variance between providers serving the same model. In theory, identical weights at identical precision should behave alike. In production, serving a large model is complex, and differences emerge.
What We Mean by “Model Degradation”
“Model degradation” is not one standardized diagnosis, and it does not automatically mean a provider intentionally substituted a smaller model.
We use the term operationally: a serving endpoint has degraded when it no longer preserves the behavior, capabilities, or interactive performance we expect from the requested model under representative prompts—even though the request may technically succeed.
For Reverie, the clearest forms are:
- Latency degradation — the connection succeeds, but time to first token becomes long enough to break the feeling of a live conversation.
- Capability degradation — an endpoint that is expected to support structured output returns malformed text or stops satisfying the schema.
- Behavioral degradation — replies become corrupted, incoherent, unexpectedly repetitive, or visibly drift into another language.
- Policy mismatch — an upstream host adds its own filtering layer, turning a scene supported by Reverie into a refusal, an empty response, or a truncated reply.
There can be many causes. Lower-precision quantization can affect difficult prompts, as both OpenRouter's documentation and published quantization research note. But quantization is not a universal explanation: inference-engine bugs, tool parsers, tokenizer or chat-template mistakes, overloaded queues, and provider-side middleware can be equally important. OpenRouter has observed that, for tool calls in particular, parser implementations are often a bigger source of variance than precision alone.
The important question is not “Which cause sounds most suspicious?” It is “Can we isolate the endpoint and reproduce the failure?”
Two Cases That Made the Problem Concrete
This was not a theoretical concern for us.
In one pinned comparison, a provider corrupted the beginning of 19 out of 20 replies by inserting punctuation or a fragment of an unrelated token. Across fourteen other hosts serving the same model, the issue appeared 0 times in 49 replies. In roleplay, where the first character is often a Markdown emphasis marker, one stray byte could also break the formatting of the entire response.
In another test, one endpoint produced severe Portuguese-Spanish mixing in 3 out of 3 generations. The same prompt stayed in Portuguese on the model's official host and on the other tested hosts using the same precision class. That made “the model is just inconsistent” a poor explanation; the defect followed the serving endpoint.
These failures are subtle because they do not necessarily throw exceptions. Traditional uptime monitoring sees a healthy API. The user sees a character who suddenly cannot speak properly.
Our Answer: Test Every Route, Not Just the Model
Every thirty minutes, our canary discovers the active providers serving Reverie's default chat model. It then sends a small set of synthetic requests to each provider, pinned to that provider alone with fallbacks disabled.
Pinning matters. If a probe is allowed to fall back, a healthy second provider can hide the failure of the first. We need to know exactly which serving environment produced the result.
The system tests four narrow, machine-judgeable contracts.
1. Time to First Token
For a streaming conversation, throughput is only half the experience. The first token determines how long the user stares at a spinner.
We measure it directly from the stream. A provider that produces no text within ten seconds fails the latency probe. The threshold is intentionally generous, and one slow round is not enough to remove a host; transient queue pressure happens.
2. Filtering and Silent Refusals
We send a fixed roleplay continuation representative of traffic Reverie supports. The probe checks the finish reason, high-confidence refusal patterns, and empty output.
We also run the probe against known-filtering hosts as positive controls. If a control unexpectedly passes, we do not congratulate every other provider—we mark the probe itself as weak. A test that can no longer detect its known failure mode is not evidence of health.
3. Structured Output
We request a small object with a fixed schema and validate the result. A transport error and malformed output are treated differently: a broken connection says the provider was unreachable, while a successful generation that cannot satisfy the schema is evidence of a capability regression.
This distinction matters. “The endpoint claims to support the parameter” and “the endpoint reliably honors it” are not the same promise.
4. Language Integrity
Reverie supports 17 interface languages, so an English-only smoke test would miss some of the failures our users actually experience.
The canary rotates through our supported locales using synthetic roleplay prompts. A local, deterministic language detector looks for a confident whole-response language switch, sustained code-mixing, and obvious encoding corruption. It does not send real conversations to the probe, and it does not ask another model to make a subjective judgment.
Ambiguous or very short output is inconclusive, not a failure. When a language failure is detected, the canary immediately runs two confirmation attempts and requires a majority. The same locale is repeated in the next round so a provider cannot escape the consecutive-round rule merely because the rotation moved on.
We Designed It to Distrust Its Own Conclusions
Automatically removing inference capacity is useful, but a false positive can create a larger outage than the defect it was meant to solve. The canary therefore has several brakes:
- Transport errors do not count as quality failures. Timeouts, rate limits, and 5xx responses freeze the score instead of advancing a suspension streak.
- One bad round is not enough. A provider must fail two consecutive half-hourly rounds.
- Observation and enforcement are separate. We can run the full system in observe mode, record which hosts would be suspended, notify administrators, and measure false positives before enabling automatic action.
- Capacity has a floor. The canary will not suspend a provider if doing so would leave fewer than two production hosts. It alerts an administrator instead.
- Recovery is tested. Suspended providers remain probeable. Two consecutive clean rounds restore a host early.
- Repeat failures escalate gradually. Suspensions last one day for a first incident, three days for the next, and seven days thereafter. After fourteen clean days, the offense history fades and the ladder resets.
The objective is not to punish providers. It is to move user traffic away from a reproducibly unhealthy route while leaving a clear path back when the problem is fixed.
Three Layers of Protection
The final routing decision combines three kinds of exclusions:
- Reviewed built-in exclusions for defects or policy mismatches we have measured and do not want accidentally reintroduced.
- Runtime incident exclusions that an administrator can add immediately, without waiting for a deployment.
- Timed canary suspensions for providers that cross the automated failure threshold.
These lists are merged into OpenRouter's provider.ignore preference when Reverie builds a request. The user does not have to retry until chance selects a better provider; the unhealthy route is taken out of consideration.
Every automated decision is recorded, administrators are notified on meaningful transitions, and a suspension can be lifted manually when necessary.
What This System Does—and Does Not—Guarantee
The canary is deliberately narrow. It can judge latency, schema validity, high-confidence filtering, language integrity, and encoding health with code. It does not pretend to score whether a character is funny enough, emotionally perceptive enough, or faithful to a complex personality.
Those broader questions still need evaluations built from representative prompts, human review, production telemetry, and—most importantly—user reports.
Nor does this system mean every odd reply proves model degradation. Generative models are probabilistic. A long conversation can accumulate conflicting context. A character definition can contain competing instructions. Sometimes a strange response is simply a strange response.
What has changed is that “same model” is no longer the end of our investigation. We can now isolate the serving route, reproduce objective failures, and keep affected traffic away from it.
Reliability Means Preserving the Experience
Multi-provider routing remains valuable. It gives Reverie the capacity and resilience to keep conversations available when individual endpoints go down.
But resilience is not merely receiving an HTTP response. A reply that arrives too late, loses its required structure, refuses supported content, or suddenly switches language is not a healthy result just because the invoice says the request succeeded.
For an AI character product, reliability means something more human: the character should still feel like the character, whichever machine happens to speak for them.
That is the standard our new routing safeguards are built to protect.
If you notice a sudden change in response quality or language, please send us a report with the model name and approximate time. Those reports help us connect the experience you saw to the exact route that served it.