Your Coding Agent's Default Model Will Change Under You

What a silent default swap actually breaks

The instinctive worry, when a vendor announces that the model inside your coding agent is changing, is about quality. Will the new one be worse? That is the wrong question, and it is wrong in a way that leads teams to do nothing until the change has already landed.

What actually breaks is calibration. Over months of daily use, a team builds an unwritten model of how its coding agent behaves. You learn roughly how much it over-edits, so you learn how large a diff to expect from a two-line ask and when a large diff means the model misread the request. You learn whether it respects the conventions file in your repository root and how firmly you have to phrase a rule before it sticks. You learn its tool-call reliability — whether it retries a failed shell command sensibly or spirals. You learn its diff discipline: whether it reformats untouched files, whether it invents helper functions rather than importing yours, whether it leaves commented-out code behind. None of that is written down. All of it is load-bearing.

That calibration is spread across at least four surfaces, and a model change perturbs all of them at once. It sits in your prompts and your conventions file, which were tuned against one model's instruction-following. It sits in your continuous integration, where an agent-authored change has to pass gates that were tightened or loosened based on how much noise the previous model produced. It sits in your review culture, where reviewers have learnt which parts of an agent diff need close reading and which can be skimmed. And it sits in your unit economics, where cost per completed task was measured against one model's token appetite and retry behaviour.

Change the model and you have changed one input to four calibrated systems, without a pull request, without a version bump in any file you control, and frequently without anyone on the team noticing on the day it happens. The first signal is usually indirect: review comments get longer, a CI job that used to pass on the first attempt starts needing a second run, or the monthly bill moves in a direction nobody explains.

This is the shape of a dependency upgrade. It is exactly the shape of a major-version bump in a library your build depends on. The difference is that nobody has a runbook for this one. The rest of this article is that runbook: what to have in place before the announcement, how to pin, how to test on your own code, how to roll, how to roll back, and how much time you actually get.

Three kinds of change you are defending against

Before building the runbook it is worth being precise about the category, because "the vendor changed something" covers three distinct events with different detection signals and different blast radii. A process tuned for one will miss the others.

The first is the model swap: the engine underneath the same product surface is replaced. The second is the behavioural or policy change: the model may be identical, but what the agent is permitted to do changes — which tools it may call, whether shell commands run in a sandbox, whether a classifier reviews an action before it executes. The third is the pricing or tier change: the capability is unchanged but the economics or the quota are not, which reshapes how your team uses the tool even when nothing technical moved.

Change type	What moves	Detection signal	Blast radius
Model swap	The engine behind the same product surface. Diff style, tool-call reliability, instruction adherence and token appetite all shift together	Release notes, changelog, an in-product banner. Indirectly: diff size distribution moves, retry rate moves, cost per task moves	Widest. Every agent-authored change, every CI job that runs an agent, every prompt tuned against the old model
Behavioural / policy change	What the agent may do — tool gating, allowlists, sandboxing, approval prompts, a classifier in the loop	Version release notes and, very often, a support ticket from a developer whose workflow stopped mid-task	Narrow but sharp. Automation and headless runs break hardest, because an interactive approval prompt has nobody to answer it
Pricing / tier change	Rate per token, included quota, seat tier boundaries, introductory pricing that expires on a date	Billing pages, pricing announcements, and the finance team noticing before engineering does	Indirect but persistent. Changes which model your team reaches for, which changes output quality without any model changing

Two dated examples make the second and third categories concrete. On 29 May 2026, Cursor 3.6 shipped a new Auto-review run mode that gates Shell, MCP and Fetch tool calls through an allowlist, a sandbox and a classifier subagent. No model changed. What changed was the set of actions the agent would take without asking, and for anyone running the tool non-interactively that is a larger operational event than a model swap. Separately, Anthropic's Claude Sonnet 5 shipped on 30 June 2026 as the new mid-tier default, arriving with introductory pricing of 2 US dollars per million input tokens and 10 US dollars per million output tokens through 31 August 2026 — a default change and a pricing change landing together, with the pricing element carrying its own expiry date. Both are worth noting as pattern, not as prediction: the specific versions will age, the categories will not.

What has to be in place before the announcement

Everything that follows is cheap if you prepared and expensive if you did not, and the preparation is the same regardless of which vendor moves first. Three artefacts, none of them large.

A pinned baseline you can name. Write down, in a file in the repository rather than in someone's memory, exactly which model configuration your team is currently working against: the model identifier, where it is set, who can change it, and the date you last verified it. This sounds trivial. It is routinely the missing piece, because the model was never chosen — it was the default when the tool was adopted and nobody has looked since. You cannot measure a regression against a baseline you cannot name.

A private regression harness on your own repository. This is the load-bearing artefact and it is the one most worth building before you need it. The method — mining real tasks from your own git history, validating that each fails at the parent commit and passes at the merge commit, and grading with fail-to-pass and pass-to-pass checks — is set out in full in the guide to benchmarking coding agents on your own repository with a private harness. It is not restated here. What matters for change management is simply that the harness exists, is runnable by one command, and produces comparable numbers across runs. If it also runs in continuous integration, as described in the guide to putting evals in CI for prompt and agent regression testing, you get drift detection for free and you will often notice a behavioural change before the release notes reach you.

A recorded set of representative tasks. Distinct from the harness, and easier. Fifteen to thirty real requests your team actually makes of the agent, written down verbatim: "add a nullable column and the migration", "port this handler to the new client", "write the tests for this module", "find why this integration test is flaky". Include the ugly ones. Include at least one task that touches your least-documented internal library, because that is where a new model's lack of familiarity with your conventions will show first. These are the tasks you will run manually and read the output of, alongside the automated harness, because some regressions are only visible to a human reading a diff.

Pin: how to pin, and where you cannot

Pinning is the first move, and the honest position is that you often cannot fully do it. Control over model selection is distributed unevenly across tools and across seat tiers, and it is worth mapping your own surfaces before you assume you have an option.

Pinning surface	What it gives you	The limitation	What to do
Per-request model parameter	Explicit model selection on every call, usually via API or a config file the agent reads	Model identifiers are themselves retired on a vendor timetable. A pin is a lease, not a freehold	Pin explicitly, never rely on an implicit default, and record the identifier in the repository. Diary a review every quarter
Org or enterprise policy	A central switch that applies to every seat under the organisation	Frequently held by a group IT or procurement function, sometimes in a different country from the engineers affected	Find out today who holds the switch and how long a change request takes. That lead time is an input to every deadline below
Fallback / revert setting	A per-user or per-team toggle to stay on the previous model after a default change	It expires. This is the most misunderstood control in the whole category	Put the closing date in a shared calendar with a named owner the day the window is announced
No control at all	Nothing — the product exposes one engine and changes it when it changes	Common in consumer tiers, IDE extensions and bundled review bots	Compensate with detection rather than control: keep the harness on a schedule so you learn the day behaviour moves, and keep a second tool warm

The fourth row deserves emphasis because it is where most teams actually sit for at least one of their tools. If you cannot pin, your defence is not control but early warning plus an alternative. Running the harness on a weekly schedule against an unpinnable tool turns an invisible change into a dated observation, which is enough to start the conversation with a number rather than an anecdote. And keeping familiarity with a second agent is cheap insurance; the trade-offs between the main options are laid out in the comparison of Claude Code, Cursor and Codex.

Note also that pinning inside a tool is a narrower problem than provider-level lock-in. If your concern is losing access to a vendor entirely rather than a default moving within one, that is a different discipline with a different artefact, covered in the guide to writing an LLM vendor exit plan before you need it. The two are complementary: the exit plan covers leaving, this runbook covers staying while the ground moves.

Test: designing the A/B on your own repository

The test is a controlled comparison, and the control is the part people skip. Same task set, same prompts, same conventions file, same repository state, two model configurations. Change one thing. If you also take the opportunity to rewrite your prompts, you have learnt nothing about the model.

Before the method, the reason vendor numbers cannot stand in for it. Public coding benchmarks measure self-contained function synthesis from a clear specification: a docstring in, a function out, unit tests grade it. Your work is a different shape entirely — large existing repositories, implicit conventions, internal libraries with no public documentation, half-specified tickets, and a review culture that penalises churn. A model can genuinely improve at the first and be worse for you at the second, because the failure modes that cost you time are over-editing, ignoring your conventions file, and calling tools in an order your environment rejects, none of which a function-synthesis benchmark measures at all.

Measure six things, per task, per configuration.

Metric	How to record it	Why it matters
completed	Boolean: did your existing test suite pass after the agent's change?	The headline number, and the only one most teams collect
diff_lines	Lines added plus deleted, from the resulting diff	Over-editing is the most common silent regression. A model that completes the task by rewriting the file is not equivalent to one that changes four lines
tool_errors	Count of failed tool or shell invocations during the run	Directly predicts how often a headless or CI run will stall
retries	Count of self-corrections after a failure	Distinguishes a model that recovers from one that spirals. Drives both latency and cost
human_edit_lines	Lines a reviewer changed before merging	The truest quality proxy you have. Requires a human in the loop, so sample it rather than running it on everything
cost_usd	Total spend for the run, including retries	Divided by completed tasks, this is cost per completed task — the only comparable unit

That last row is where teams under-invest. Cost per token is not comparable across models with different retry behaviour and different verbosity; cost per completed task is. A configuration that completes a few more tasks while doubling cost per completed task may still be the right choice, but it should be a decision rather than a discovery in next month's invoice. The framing for that decision, including how to attribute spend to a feature rather than a model, is in the guide to LLM unit economics and cost per task.

Here is a framework-agnostic sketch of the runner. It assumes only that your agent can be invoked as a command with a prompt and a model identifier, and that you can measure a diff and run your tests. Adapt the two shell templates to whichever tool you use; everything else is bookkeeping.

#!/usr/bin/env python3
"""Run a fixed task set against two model configurations and record six metrics.

Assumes only:
  - the agent is invocable as a command that takes a prompt and a model id
  - the repo is a git checkout you can reset between tasks
  - `test_cmd` returns 0 on success

Everything vendor-specific lives in AGENT_CMD. Nothing else changes.
"""
import csv, json, pathlib, subprocess, time

REPO      = pathlib.Path("/path/to/your/repo")
BASELINE  = "<current-default-model-id>"
CANDIDATE = "<incoming-default-model-id>"

# The ONE vendor-specific line. Adapt to your agent's CLI.
AGENT_CMD = ["your-agent", "--model", "{model}", "--prompt-file", "{prompt}",
             "--json-log", "{log}", "--non-interactive"]

TEST_CMD  = ["make", "test"]


def reset_repo(ref="main"):
    subprocess.run(["git", "reset", "--hard", ref], cwd=REPO, check=True)
    subprocess.run(["git", "clean", "-fd"], cwd=REPO, check=True)


def diff_lines() -> int:
    out = subprocess.run(["git", "diff", "--numstat"], cwd=REPO,
                         capture_output=True, text=True).stdout
    total = 0
    for line in out.splitlines():
        added, deleted, *_ = line.split("\t")
        # binary files report "-"; skip them rather than crash
        if added.isdigit() and deleted.isdigit():
            total += int(added) + int(deleted)
    return total


def parse_run_log(path: pathlib.Path) -> dict:
    """Pull tool errors, retries and cost from the agent's own run log.

    Field names differ per tool - map them here, once. If your agent does not
    emit cost, substitute tokens x your rate card; do NOT leave the column
    empty, because a comparison without cost is not a decision.
    """
    data = json.loads(path.read_text())
    return {
        "tool_errors": sum(1 for e in data["events"] if e.get("tool_error")),
        "retries":     sum(1 for e in data["events"] if e.get("kind") == "retry"),
        "cost_usd":    float(data.get("usage", {}).get("cost_usd", 0.0)),
    }


def run_task(model: str, task: dict, logdir: pathlib.Path) -> dict:
    reset_repo(task["base_ref"])
    log = logdir / f"{model}--{task['id']}.json"
    prompt = logdir / f"{task['id']}.prompt"
    prompt.write_text(task["prompt"])

    cmd = [a.format(model=model, prompt=prompt, log=log) for a in AGENT_CMD]
    t0 = time.perf_counter()
    subprocess.run(cmd, cwd=REPO)                      # non-zero is data, not fatal
    wall_s = time.perf_counter() - t0

    completed = subprocess.run(TEST_CMD, cwd=REPO,
                               capture_output=True).returncode == 0
    row = {"model": model, "task_id": task["id"], "completed": completed,
           "diff_lines": diff_lines(), "wall_s": round(wall_s, 1),
           "human_edit_lines": ""}      # filled in by a reviewer, sampled
    row.update(parse_run_log(log))
    return row


def main(tasks_file="tasks.json", out="results.csv", samples=3):
    tasks  = json.loads(pathlib.Path(tasks_file).read_text())
    logdir = pathlib.Path("runs"); logdir.mkdir(exist_ok=True)
    rows = []
    # Interleave models and repeat each task: agents are stochastic, so a
    # single run per task tells you about luck, not about the model.
    for i in range(samples):
        for task in tasks:
            for model in (BASELINE, CANDIDATE):
                r = run_task(model, task, logdir)
                r["sample"] = i
                rows.append(r)
    with open(out, "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0]))
        w.writeheader(); w.writerows(rows)
    print(f"wrote {len(rows)} rows to {out}")


if __name__ == "__main__":
    main()

Three notes on running it. Interleave the models rather than running all of the baseline then all of the candidate, so that a vendor-side capacity fluctuation does not land entirely on one arm. Repeat each task at least three times, because coding agents are stochastic and a single run per task measures luck. And leave the human_edit_lines column empty in the automated pass, filling it in for a sample of ten to fifteen tasks by having a reviewer treat the output as a normal pull request. That column is slow to collect and it is consistently the most informative.

Reading the results is a matter of looking for divergence, not for a winner. Suppose your harness shows the candidate completing a similar share of tasks but producing noticeably larger diffs and a higher human-edit rate on the tasks that touch your least-documented internal module. That is not "the new model is worse". That is a specific, addressable finding: your conventions file was tuned against the old model's defaults and needs a rule made explicit that was previously implicit. The method for writing that rule so it actually changes behaviour is in the guide to AGENTS.md and CLAUDE.md files that actually steer agents, and a fix there is usually cheaper than a fight over the default.

Roll out and roll back

You have numbers. Now you need a deployment that produces a clean signal and a retreat that works under time pressure.

Staged rollout: one team, one repository, one week

The unit of rollout is a team on a repository for a working week, not an individual on whatever they happen to be doing. A week matters because it covers a full work rhythm: a planning session, a release, at least one incident-shaped interruption, and the Friday afternoon where everyone is trying to land something before the weekend. A single team matters because it keeps the comparison legible — if six teams adopt at once you will have six confounded opinions and no signal.

Pick a team whose work is representative rather than easiest. The temptation is to start with the greenfield service where everything is clean and documented, and the result is a rollout that tells you nothing about the ten-year-old billing module where the agent will actually struggle.

Watch four things during the week, all of which you can pull from tooling you already have. Diff size distribution on agent-authored changes, compared with the same team's previous month. Review turnaround time, because a reviewer spending longer per pull request is the earliest human signal of a regression. CI first-attempt pass rate on agent-authored branches. And cost per completed task, which should be tracking your harness prediction — if it is not, your task set is not representative and that is a finding in itself.

The rollback runbook

Write this before you start the rollout, not during it. Four lines is enough, and it must name a role rather than a person: who decides, what the exact configuration change is, how long it takes to propagate to every seat, and what the team is told. If the switch lives with a central IT function, "how long it takes" includes their ticket queue, and you should have measured that number rather than assumed it.

Three signals should trigger a rollback rather than a discussion. First, a category of task that previously worked reliably now fails reproducibly — reproducibility is the bar, because a single bad run is noise. Second, tool-call error rate rises enough to break automation, which is worse than a quality regression because it is silent in interactive use and fatal in headless runs. Third, cost per completed task moves outside the tolerance you agreed before the rollout began. Agreeing that tolerance in advance is what turns a rollback from an argument into a procedure.

Deadline arithmetic: a fallback window is a countdown

Here is the part that most often goes wrong, and it goes wrong because a fallback window feels like a safety net when it is actually a timer. On the day the window closes, your options are exactly the ones you have validated. If you have not run the comparison by then, you do not get to choose.

The publicly announced dates for the GitHub Copilot default change in 2026 make useful arithmetic, and they are given here as a dated case study rather than as a permanent fact. At Build 2026, on 2 June 2026, Microsoft positioned Project Polaris as the new default engine for GitHub Copilot, replacing GPT-4 Turbo, with the default swap taking effect from August 2026 across Individual, Business and Enterprise seats. Microsoft described Polaris as its first in-house foundation model in the Copilot stack: a mixture-of-experts coding model that activates only the relevant expert sub-networks per query, with specialised experts trained for lower-resource languages such as Rust and Haskell, running on Microsoft's own Maia silicon. Microsoft also provided an optional fallback window through November 2026 for reverting to GPT-4 Turbo. Reporting describes that fallback principally as tenant-level pinning for Enterprise customers; whether the same toggle reaches every Individual seat is less clearly established, which is itself worth confirming for your own tier rather than assuming.

On performance, precision matters. Microsoft's Build announcement, and reporting of it, stated that Polaris outperformed GPT-4 Turbo on HumanEval and MBPP, particularly in lower-resource languages such as Rust and Haskell. That is a vendor-reported benchmark claim, not an independent evaluation, and this article takes no position on whether Polaris is better or worse for any given codebase. It is a reason to run your own comparison. It is not a substitute for one, for exactly the reasons set out in the testing section above.

Date	Event	What a prepared team is doing
2 June 2026	Announced at Build 2026: Polaris to become the Copilot default, replacing GPT-4 Turbo	Confirm the pinned baseline, check who holds the org policy switch, diary the November close
June – July 2026	Pre-default window	Run the harness and the representative task set against both configurations. Produce the six-metric table
From August 2026	Default swap takes effect across Individual, Business and Enterprise seats	Staged rollout on one team, one repository, one week. Fallback available if the week goes badly
August – October 2026	Validation and prompt or conventions adjustment	Fix what the comparison surfaced. Re-run the harness after each change
November 2026	Optional fallback window closes	Nothing to do, because the decision was made in September on evidence

Read the slack honestly. Roughly five months from announcement to the fallback closing looks generous, and it is — but subtract two to three weeks for a proper comparison, a week per rollout wave, a fortnight of prompt and conventions adjustment, and a contingency margin for the fact that most of this happens alongside your actual roadmap. Then subtract the lead time on whoever holds the org policy switch. What remains is a decision deadline several weeks earlier than the published one. Work backwards from the close, never forwards from the announcement.

This arithmetic has a geography, and it is worth naming. In plenty of organisations the seat policy is set by a parent company or a group IT function in one country while the engineering team absorbing the change sits in Bengaluru, Hyderabad, London or Manchester. The people who feel the regression are not the people who hold the switch, and they may be eight or twelve hours out of sync with them. That gap is not a reason for fatalism; it is a reason to front-load the evidence. A one-page results table sent in July lands very differently from an escalation raised in the second week of August, and it is reusable when the same conversation recurs next year with a different vendor.

Common pitfalls

These recur across teams, vendors and instances, and every one of them is cheaper to avoid than to discover.

Trusting the vendor benchmark. Vendor-reported results are a signal about the model's general direction, produced by a party with an interest in the outcome, measured on tasks that do not resemble yours. They justify running your own comparison. They never replace it. This is not scepticism about any particular vendor; it is the same standard you would apply to a library maintainer's own performance claims before a major upgrade.

Testing on toy tasks. A comparison run on FizzBuzz-shaped problems will find no difference between any two competent models, which teams then report as "no regression". The tasks that discriminate are the ones with implicit context: your internal client library, your migration conventions, the module whose only documentation is a stale comment. If your task set does not include the parts of your codebase you would warn a new joiner about, it will not detect anything.

Measuring quality without measuring cost. A comparison that reports only completion rate is half a decision. Two models can complete the same share of tasks with a twofold difference in cost per completed task, driven entirely by retry behaviour and verbosity, and that difference will show up in the invoice whether or not you measured it.

Forgetting that CI and code-review bots use the same default. This is the most consistently overlooked one. Your automated review bot, your test-generation job and any agent step in your pipeline are all consuming the same default model, and they have nobody watching their output the way a developer watches an IDE. A change in verbosity or comment style turns into review noise across every pull request in the organisation, and the usual symptom is that people quietly start ignoring the bot. If you run agents in your quality gates, include them in the rollout plan explicitly; the tuning that keeps them useful is covered in the guide to AI code review in CI and quality gates that cut noise.

Not versioning prompts and conventions alongside the model. If your conventions file and your prompt templates are not under version control with dated changes, you will never untangle which improvement came from the model settling and which came from an edit somebody made mid-rollout. Bind the records together and the post-mortem writes itself.

The checklist for the next default change

The specific instance in this article will be history within a year. The sequence will not, because it recurs every few months across every vendor. Keep this somewhere you will find it when the next announcement lands.

Standing, before any announcement. Record the pinned baseline — model identifier, where it is set, who can change it, date last verified. Keep a private regression harness on your own repository that runs with one command. Keep fifteen to thirty representative tasks written down verbatim. Version prompts and the conventions file with dated changes. Know who holds the org policy switch and how long their change requests take.

Within a week of the announcement. Classify the change: model swap, behavioural or policy, pricing or tier — it may be more than one. Find the fallback mechanism and its closing date, and put that date in a shared calendar with a named owner. Compute the decision deadline by working backwards from the close, subtracting evaluation, rollout, adjustment and policy lead time. Identify which of your surfaces you can pin and which you cannot.

Before the default lands. Run the six-metric comparison on your own repository, interleaved, at least three samples per task. Collect human-edit-after on a sample of ten to fifteen tasks. Produce a one-page results table. Agree the rollback tolerance with whoever owns the budget before you need it.

At rollout. One team, one repository, one week, on representative rather than convenient code. Watch diff size distribution, review turnaround, CI first-attempt pass rate and cost per completed task. Collect friction reports in one named channel. Include CI and review bots in the plan. No individual opt-outs.

Before the window closes. Decide on evidence, well ahead of the published date. Record the new baseline. Write down what changed in your prompts and conventions, and why. File the results table where the next person will find it, because there will be a next time and the arithmetic will be the same.

None of this is a complaint about vendors. Defaults change because products improve, and a vendor that never changed its default would be a worse tool to depend on. What is unreasonable is depending on something that influences every line of code your team ships while treating it as unmanaged. You would not run an unpinned major version of a build dependency and hope. This is the same problem wearing a different hat, and it deserves the same discipline.

It also happens to be excellent proof-of-work. A repository containing a task set, a comparison runner, a dated results table and a short record of what you changed in response is a far more credible signal of engineering judgement than any claim on a CV — and it is the sort of thing the people hiring for platform and developer-experience roles across Bengaluru, Chennai, London and Manchester actively look for.