Verify the Model You Are Actually Being Served

What you need to know

Here is the failure this guide exists to prevent. Your evals passed in March. You shipped, the numbers looked good, and you stopped watching. In June, support tickets start mentioning that the assistant has become oddly terse, or that it refuses a category of request it used to handle. Every call returned 200. The model field says exactly what it said in March. Nothing in your deployment history explains it, and by the time someone suggests the model changed underneath you, there is no way to prove it either way, because nobody measured the baseline.

The root cause is a category error: treating the identifier a provider returns as though it were a measurement. It is a label attached by the party whose behaviour you are trying to verify — accurate about which alias your request resolved to, and silent about the things that actually move behaviour. Weights can be re-tuned under an alias. A safety post-training pass can land without a new public version. A quantised serving variant can appear to relieve capacity pressure. A request from Mumbai and a request from London can be served by different infrastructure.

None of this is misconduct, and it is important to say so plainly. Silent model changes are the normal consequence of running a large model as a continuously deployed service. The problem is not bad faith; it is that a probabilistic component with no changelog obligation sits in the middle of your product, and the only party with an incentive to detect its changes on your traffic, against your quality bar, is you. So build the detector, in three layers of increasing cost and decreasing frequency: a provenance snapshot, a behavioural canary suite that measures distributions rather than examples, and a knowledge-cutoff probe for when something big has shifted.

What actually changes underneath you

"The model changed" covers several distinct events, and only some are visible in metadata. Alias repointing is the loudest: when a floating alias moves, capability, style, refusal boundaries and token economy move together, and it is the easiest change to catch provided you record the resolved identifier rather than the alias you sent. Gradual checkpoint rollout is subtler — during a ramp your requests may be served by either version, producing bimodal results where the same prompt gives the old answer nine times and the new answer once. Single-example checks are useless there; distributional checks catch it immediately.

Safety and preference post-training can move the refusal boundary noticeably while leaving factual capability and every visible identifier untouched, which makes it the change most likely to break a product sitting near a policy line. Serving-side changes round out the list: capacity-tier and region routing shift latency and token-count distributions; quantised variants usually cost negligible accuracy, and when they do not the signature is unchanged easy items and a worse hardest tail; and provider-side changes to default temperature or an injected preamble are invisible in your code while materially changing verbosity or format compliance.

Change type	Observable symptom	Does a version string move?
Alias repointed to a new checkpoint	Broad behaviour shift; several eval metrics move together on the same day	Yes, if you record the resolved identifier rather than the alias you sent
Gradual checkpoint rollout	Bimodal results; identical prompts give two distinct answer styles	Sometimes — the resolved identifier may differ between runs during the ramp
Safety or preference post-training	Refusal boundary moves; tone and hedging change; capability roughly stable	Usually not
Capacity-tier or region routing	Latency and token-count distributions shift; occasional style variation	No, though region may appear in response headers
Quantised or optimised serving variant	Accuracy holds on easy items, dips on the hardest tail of the eval set	No
Provider-side default parameter change	Verbosity, truncation or format-compliance rate changes	No
Your own prompt, SDK or gateway upgrade	Anything at all	No — and this is the most common cause, so rule it out first

That last row deserves emphasis. Most "the model changed" incidents turn out to be a prompt edit, a gateway silently stripping a parameter, or an SDK upgrade that altered a default, which is why the version-control discipline in prompt management and versioning is a prerequisite. Pin your side of the interface before you go looking for movement on theirs.

Cheap provenance checks you can run in CI

The first layer costs almost nothing and catches the loudest failures. Every model call hands back more than the completion: a resolved identifier, often a serving fingerprint, headers describing region or routing, and usage counters. Capture all of it, store one snapshot, diff the next against it. Write it against a thin abstraction rather than a vendor SDK — a single call_model(prompt, **kwargs) returning (text, meta) keeps the harness portable when an SDK reshapes its response objects. The same discipline underpins porting a prompt suite across vendors: the harness is the durable asset, the client library is not.

import datetime
import hashlib
import json
import os

def call_model(prompt, **kwargs):
    """Thin, vendor-agnostic wrapper. Implement once per provider.

    Must return (text, meta) where meta is a plain dict containing at
    least: requested_model, model, system_fingerprint, headers, usage.
    Everything below this function stays unchanged across vendors.
    """
    raise NotImplementedError

PROBE = "Reply with the single word: ping."
HEADER_PREFIXES = ("x-model", "x-request-region", "x-served-by",
                   "anthropic-", "openai-", "google-")

def provenance_snapshot():
    text, meta = call_model(PROBE, max_tokens=8, temperature=0)
    headers = {k.lower(): v for k, v in meta.get("headers", {}).items()}
    return {
        "captured_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
        "requested_model": meta.get("requested_model"),
        "reported_model": meta.get("model"),
        "system_fingerprint": meta.get("system_fingerprint"),
        "headers": {k: v for k, v in headers.items()
                    if k.startswith(HEADER_PREFIXES)},
        "reply_hash": hashlib.sha256(text.strip().encode()).hexdigest()[:16],
        "output_tokens": meta.get("usage", {}).get("output_tokens"),
    }

VOLATILE = {"captured_at"}

def diff_snapshots(old, new):
    changes = []
    for key in sorted((set(old) | set(new)) - VOLATILE):
        if old.get(key) != new.get(key):
            changes.append((key, old.get(key), new.get(key)))
    return changes

if __name__ == "__main__":
    path = "provenance/latest.json"
    new = provenance_snapshot()
    changes = []
    if os.path.exists(path):
        with open(path) as f:
            changes = diff_snapshots(json.load(f), new)

    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w") as f:              # always commit the new baseline
        json.dump(new, f, indent=2, sort_keys=True)

    for key, before, after in changes:
        print(f"PROVENANCE CHANGE {key}: {before!r} -> {after!r}")
    raise SystemExit(1 if changes else 0)

Two details matter more than they look. The new snapshot is written before the non-zero exit, so the job alerts once rather than on every run until a human intervenes. And the header filter is a prefix allowlist, not a full capture, because headers carry request-scoped values such as rate-limit counters that would make every diff fire.

Tokenisation probes are worth adding too: record the reported input-token count for a small fixed corpus and a tokeniser or model-family change shows up immediately. Neither that nor a temperature-zero output hash is proof, though — temperature zero is a low-variance regime, not a determinism guarantee.

Signal	What it catches	Cost per run	False-positive risk
Resolved identifier and response headers	Alias repoints, region and routing changes	Effectively free	Very low
Serving fingerprint field, where offered	Serving-stack and configuration changes	Effectively free	Medium — moves for reasons unrelated to weights
Deterministic canary prompts at temperature zero	Any weight, decoding or preamble change	Pennies	Medium — temperature zero is not determinism
Tokenisation probes on a fixed corpus	Tokeniser or model-family change	Pennies	Low, but only catches large changes
Knowledge-cutoff probe	A genuinely different pre-training run	Under a dollar	Low, but slow and statistically noisy
Behavioural distribution suite	Post-training and serving changes nothing else sees	A few dollars a day	Tunable through the alert rule

Estimating a knowledge cutoff yourself

The heaviest instrument answers a question metadata cannot: is this model built on a different pre-training run from the one I tested? The published methodology worth learning is Shrivu Shankar's, set out on blog.sshh.io in August 2026 under the title "Exploring Claude/GPT Knowledge Cutoffs". He describes three complementary techniques; the first is the one most teams can run themselves.

Incompressible knowledge probes

The core idea is to quiz a model on facts it cannot derive. Shankar calls them incompressible because they cannot be inferred or reasoned towards from anything else — the model either encountered them in training or it did not. Specific dated events work well: who won a particular fixture, which day a named storm made landfall, which company announced which acquisition. Wikipedia's year-in-review pages, such as "2025 in the United States", are a convenient source, already sorted by date.

Administer them as eight-way multiple choice rather than open questions. Multiple choice removes the grading problem entirely — no judge, no fuzzy matching, no partial credit — and eight options puts the chance floor at 12.5 per cent, giving a clear ceiling to read the curve against: a model with no training signal scores near 12.5 per cent correct, or 87.5 per cent error. Draw distractors from other items in the same category, because a quiz whose wrong options are absurd measures common sense rather than memory.

Bucket by month and plot the error rate. What you want is not a cliff but a ramp: error sits low and flat through the periods the model knows, then climbs over one to three months towards the ceiling, and Shankar's interpretation is that this point approximates the completion date of pre-training. Volume matters — below about twenty items per month, sampling noise hides the ramp — so budget twenty to thirty per month across twelve to eighteen months, with repeats at re-shuffled option ordering to damp position bias.

import random
import re
from collections import defaultdict

LETTERS = "ABCDEFGH"          # 8 options -> 12.5% chance floor

# Each fact, built once and committed to the repo so the quiz is stable:
#   {"month": "2025-11", "category": "sport",
#    "question": "Which club won ...?", "answer": "..."}
# Needs at least 8 facts per category so distractors can be drawn.

def attach_distractors(items, rng, k=7):
    for item in items:
        pool = [x["answer"] for x in items
                if x["category"] == item["category"]
                and x["answer"] != item["answer"]]
        item["distractors"] = rng.sample(pool, k)
    return items

def render(item, rng):
    options = [item["answer"]] + item["distractors"]
    rng.shuffle(options)
    correct = LETTERS[options.index(item["answer"])]
    body = "\n".join(f"{LETTERS[i]}. {o}" for i, o in enumerate(options))
    prompt = ("Answer with a single letter and nothing else.\n\n"
              f"{item['question']}\n\n{body}")
    return prompt, correct

def error_rate_by_month(items, seed=7, repeats=3):
    rng = random.Random(seed)
    attach_distractors(items, rng)
    tally = defaultdict(lambda: [0, 0])          # month -> [asked, wrong]
    for item in items:
        for _ in range(repeats):
            prompt, correct = render(item, rng)
            text, _meta = call_model(prompt, max_tokens=4, temperature=0)
            match = re.search(r"\b([A-H])\b", text.strip().upper())
            tally[item["month"]][0] += 1
            if match is None or match.group(1) != correct:
                tally[item["month"]][1] += 1
    return {m: wrong / asked for m, (asked, wrong) in sorted(tally.items())}

def read_inflection(curve, share=0.75, ceiling=0.875, warmup=6, run=2):
    """First month where error closes `share` of the gap between the
    early-period baseline and the chance ceiling, and stays there."""
    months = sorted(curve)
    early = sorted(curve[m] for m in months[:warmup])
    baseline = early[len(early) // 2]
    threshold = baseline + share * (ceiling - baseline)
    streak = 0
    for m in months:
        streak = streak + 1 if curve[m] >= threshold else 0
        if streak >= run:
            return months[months.index(m) - run + 1], round(threshold, 3)
    return None, round(threshold, 3)

The table below shows what a readable curve looks like. These numbers are illustrative — constructed to demonstrate how to read the shape, not measured results from any model. Run the code against your own endpoint for real ones.

Month bucket	Items asked	Error rate (illustrative)	Reading
2025-06	75	0.19	Well inside the known period
2025-07	75	0.21	Flat baseline
2025-08	75	0.18	Flat baseline
2025-09	75	0.24	Within sampling noise of baseline
2025-10	75	0.22	Flat baseline
2025-11	75	0.31	First hint of thinning coverage
2025-12	75	0.46	Ramp begins in earnest
2026-01	75	0.72	Crosses the 0.71 threshold — inflection
2026-02	75	0.83	Confirms the run; near chance
2026-03	75	0.86	At chance (0.875)
2026-04	75	0.87	At chance

Reading that with the helper above: the median of the first six months is 0.22, the ceiling is 0.875, and closing 75 per cent of that gap puts the threshold at 0.71. January 2026 is the first month above it and February confirms the run, so the inflection lands at January 2026 and training signal thins through December 2025. The shape tells you what a single number cannot: a gentle ramp is a normal data-collection tail, whereas a sudden cliff suggests a hard date filter in the pipeline.

The other two techniques, and what Shankar found

Shankar's second technique is data mixture inference — examining tokenisation patterns to draw conclusions about the composition of the training dataset. The third is self-identification and direct date queries: asking a model what today's date is and which model it believes itself to be. He reports that these responses correlate with the factual cutoffs measured by the probes, and that self-identification patterns can indicate which earlier models contaminated the training data.

The findings are worth stating carefully, because they are one researcher's estimates and not lab disclosures. Shankar reports that Anthropic's Opus 4.7 and later share knowledge cutoffs around late December 2025, consistent with a single pre-training run behind that family, and that Opus 5 is an anomaly — a published cutoff of May 2026 despite measured knowledge matching the earlier models. On the OpenAI side, the GPT-5.6 family appears to come from a distinct checkpoint finishing around late February 2026, separate from GPT-5.5. No provider has confirmed any of this, and none of it should be read as a lab admitting anything. What it shows is that the technique produces coherent, comparable signals — which is the reason to run it on your own endpoint.

Behavioural fingerprinting for drift

This is the point where the guide brushes up against a neighbouring problem, so it is worth naming the split: production drift detection covers quality drift arising from your side — changing inputs, changing users, a corpus that has aged. This section covers identity change arriving from the provider side, under a version string that did not move. The instrumentation overlaps; the causes, and therefore the remedies, do not.

Provenance tells you when the label changed. Behavioural fingerprinting tells you when the behaviour changed, which is what you actually care about. The research area is real — task-agnostic model fingerprinting, identifying a model from its output characteristics without access to weights, has been a steady arXiv topic through 2026 — but you do not need a published method. You need a fixed set of prompts, metrics over the responses, and the discipline to compare distributions rather than examples.

A useful canary suite has five families of probe. Refusal-boundary probes sit deliberately close to a policy line in both directions, and the refusal rate across the set moves when safety post-training lands. Formatting-tic probes ask for structured output and measure compliance. Tool-call argument style probes hold the schema and request fixed and watch how the arguments get filled — which optional fields, what date format, batched or serialised. Verbosity distribution is output-token count per prompt, the cheapest sensitive signal there is. And latency distributions catch serving-side changes that content metrics miss entirely.

The critical design choice is distributional. A single example is worthless as a drift signal, because any response can differ from yesterday's for reasons unrelated to the model; what is stable is the distribution over a fixed corpus. If your refusal rate over 40 boundary probes has sat between 0.31 and 0.36 for a month and today reads 0.52, that is a signal; if one probe flipped from answer to refusal, that is Tuesday. If you already run the instrumentation described in instrumenting agents for production, most of the collection plumbing exists already.

Metric	How it is computed	Primarily detects	Suggested rule
Refusal rate	Share of boundary probes declined, over 40 fixed prompts	Safety and preference post-training	3σ from a 30-day rolling baseline, 2 consecutive runs
Format-compliance rate	Share of structured requests that parse and validate	Preamble and decoding changes	2.5σ, 2 consecutive runs — cheap to verify, page-worthy
Mean output tokens	Mean over the full canary corpus	Verbosity tuning, default parameter changes	3σ, 3 consecutive runs — noisy, so require a longer run
Tool-argument shape	Mean populated optional fields per call; date format histogram	Instruction-following and post-training changes	3σ, 2 consecutive runs
p50 and p95 latency	Per-call wall time, excluding client retries	Capacity tier, region and serving-variant changes	3σ on p50; treat p95 as informational only
Judged answer quality	Calibrated judge over a 40-item sample against reference answers	Genuine capability regression	2σ, 3 consecutive runs — expensive, so run it last

The alerting rule is where you set your own trade-off. "Flag when the metric moves beyond three standard deviations of a 30-day rolling baseline on two consecutive runs" is a reasonable default. Loosen it to two deviations on a single run and you detect real changes within a day but page someone most weeks, until the team starts ignoring the alert — worse than no alert at all. Tighten it to four deviations across three runs and false alarms vanish, but genuine drift takes most of a week to surface. No setting gives you both.

import statistics
from collections import deque

class DriftTripwire:
    """Rolling z-score over a fixed-size baseline window.

    Only non-firing observations update the baseline, so a genuine step
    change does not quietly become the new normal. After a change is
    accepted, call reset() to re-baseline deliberately.
    """

    def __init__(self, window=30, z=3.0, consecutive=2, min_history=10):
        self.history = deque(maxlen=window)
        self.z = z
        self.consecutive = consecutive
        self.min_history = min_history
        self.streak = 0

    def observe(self, value):
        if len(self.history) < self.min_history:
            self.history.append(value)
            return False
        mu = statistics.fmean(self.history)
        sigma = statistics.pstdev(self.history) or 1e-9
        fired = abs(value - mu) / sigma >= self.z
        self.streak = self.streak + 1 if fired else 0
        if not fired:
            self.history.append(value)
        return self.streak >= self.consecutive

    def reset(self):
        self.history.clear()
        self.streak = 0

METRICS = ("refusal_rate", "format_compliance", "mean_output_tokens",
           "tool_arg_fields", "p50_latency_ms")

TRIPWIRES = {
    "refusal_rate":      DriftTripwire(z=3.0, consecutive=2),
    "format_compliance": DriftTripwire(z=2.5, consecutive=2),
    "mean_output_tokens": DriftTripwire(z=3.0, consecutive=3),
    "tool_arg_fields":   DriftTripwire(z=3.0, consecutive=2),
    "p50_latency_ms":    DriftTripwire(z=3.0, consecutive=2),
}

def nightly(canary_metrics):
    """canary_metrics: {metric_name: float} from tonight's suite run."""
    return [m for m in METRICS if TRIPWIRES[m].observe(canary_metrics[m])]

Excluding firing observations from the baseline is deliberate: if outliers updated the window, a slow drift would drag the baseline with it and the tripwire would never fire — the boiling-frog failure of naive rolling alerts. And reset() exists because once a change is accepted, the old baseline is meaningless.

Wiring it into CI and production

Cadence follows cost. Provenance snapshots are near-free: run them hourly against every region you serve from, and as a pre-deploy step so a build never ships against an endpoint that changed identity since the last green run. The behavioural canary belongs in a nightly job rather than on every commit, because it measures the provider rather than your code. The knowledge probe is weekly, or on demand once something else has fired. Keep your own regression evals on the pre-merge path as described in putting your evals in CI — the two systems answer different questions.

Store every run as an append-only record: timestamp, endpoint, region, resolved identifier, every metric, and the raw responses. The responses matter more than teams expect, because the question you will want to answer in three months is "what did it used to say?"

The cost, assuming 120 canary prompts at three repeats nightly, averaging 700 input and 300 output tokens per call, at illustrative mid-tier list rates of 3 dollars per million input tokens and 15 per million output, with judge grading on a cheaper model at 1 and 5:

Component	Cadence	Calls per month	Tokens per month (in / out)	Monthly cost
Provenance snapshot (one region)	Hourly	720	14.4k / 5.8k	Under $0.15
Behavioural canary, 120 prompts × 3 repeats	Nightly	10,800	7.56M / 3.24M	$71.28
Judge grading, 40 sampled answers	Nightly	1,200	1.44M / 0.12M	$2.04
Knowledge-cutoff probe, 240 items	Weekly	960	0.17M / 0.004M	$0.58
Total	—	13,680	9.18M / 3.37M	≈ $74

Seventy-four dollars a month, dominated by the nightly suite. If that is too much, halve the repeats before you cut prompts — corpus breadth buys more detection power than repetition. Only the provenance layer needs to run per-region. Set against one engineer-week spent bisecting an unexplained regression, the whole programme is a rounding error.

On paging: the tripwire should open a ticket, not wake anyone. Model drift is almost never a same-hour emergency. The exception is format compliance, which breaks downstream parsers and is a production incident in its own right — route that one through your LLM incident response runbooks.

India and UK considerations

Two regional points change the design rather than decorating it. In India, the sensible default is a canary corpus with no personal data in it at all: synthetic prompts, written by you, committed to your repository. That keeps the tripwire outside any question about processing personal data under the DPDP framework. If you additionally sample production prompt and response pairs for judged grading, treat that as a distinct processing activity with its own basis, retention limit and deletion path. Data-residency commitments cut the same way: if you have told customers their inference stays in an Indian region, your probes must hit that region's endpoint, or you are measuring a model that never serves them.

For UK-based teams the practical pressure comes from serving EU customers. The EU AI Act's transparency and documentation expectations broadly push downstream deployers towards being able to say which system produced a given output, and to keep that documentation current as the system changes — precisely the record you will not have if a model moved underneath you unnoticed. Regulated firms face a parallel expectation from their supervisors around model change control and audit trails. Confirm specifics with counsel; build the dated log either way.

When the tripwire fires

The first move is always to rule yourself out. Diff your own deployment history over the alert window: prompt templates, SDK and gateway versions, feature flags, retrieval index rebuilds. If anything on your side moved, that is the leading hypothesis until disproved — self-inflicted change is far more common than provider change.

If your side is clean, pin. Where the provider offers dated snapshots, switch the affected route from the alias to the dated identifier you were previously resolving to. What pinning buys is not permanence but a fixed comparison target: run the same suite against pinned and current on the same day and read cleanly what differs.

Then diff by task family rather than in aggregate — aggregates hide the shape of a change, and the shape is the diagnosis. A uniform small drop suggests a serving change; a sharp drop confined to the hardest items suggests capability or quantisation; a drop confined to boundary-adjacent requests is post-training; verbosity moving with capability flat is parameter or preference tuning. If your suite leans on an LLM judge, pin the judge too, because a drifting judge measuring a drifting model produces uninterpretable numbers — the discipline in calibrating your LLM judge against humans matters double here.

Then decide: accept and re-baseline, adapt your prompts and re-baseline, or hold on the pin while you plan. Holding is temporary, not a strategy, because dated snapshots are eventually retired and you will be moved whether you are ready or not — the scenario your LLM vendor exit plan exists to cover. If the change alters what users experience, tell them plainly: what changed, when you detected it, what you measured. A behaviour change that is explained is forgiven far more readily than one users discover themselves.

Limits and honest caveats

Everything in this guide is estimation, not proof. That is Shankar's own framing and it should be yours: as he states plainly, "Everything here is an estimate." He names two sources of error that apply to any replication. First, potential offset error if the assumption that measured knowledge correlates with the completion of pre-training turns out to be wrong — a model's knowledge of a period could lag or lead its pre-training end date for reasons of data collection rather than training schedule. Second, vertical strips in self-report data may reflect post-training on recency-biased datasets rather than genuine pre-training dates, so a model's stated sense of "now" can be an artefact of fine-tuning rather than evidence about the base model.

Three more limits are worth internalising. Providers do not owe you a changelog; continuous deployment is normal practice, and detecting the changes that matter to your product is your job. Self-report is unreliable — a model asked which model it is answers from training data and system prompt, not introspection, and can be confidently wrong. And contamination confuses attribution: because each generation trains partly on text produced by the previous one, self-identification can point at an ancestor rather than the model in front of you, exactly the effect Shankar notes.

There is a subtler trap on your own side: a canary corpus that leaks into public training data stops measuring what you think it measures — the mechanism described in benchmark contamination. Keep it private, and refresh a slice periodically while freezing the rest so the time series stays comparable. Accept, finally, that no amount of probing gets you certainty about what runs behind an endpoint. What it gets you is a dated, defensible record of when the behaviour you depend on changed — enough to act on, and far better than finding out from a customer.

Where to start

Order matters more than sophistication. Start with the provenance snapshot — thirty lines and free — and commit its output to your repository, so you get a version-controlled history of what the provider has told you from day one. Next, pin every production route to a dated snapshot where one exists; it costs nothing and converts a moving target into a fixed one. Third, write twenty canary prompts across the five probe families and record metrics nightly even before you have alerting, because the baseline takes a month to accumulate. Only then switch on the tripwire, loosely at first, tightening as you learn what normal variance looks like.

The knowledge-cutoff probe is the last thing to build, not the first: it is the most interesting and the least operationally useful. Either way the durable asset is the one that shows up in every part of this work — a golden set of your own prompts with your own expected behaviours, of the kind described in building your first LLM evaluation suite. Build that and the provenance question becomes answerable. Skip it and the version string is all you will ever have, and it was never evidence in the first place.