Reliable Structured Output in Production: Prompting Patterns for Claude, GPT-5 and Gemini

Why structured output breaks in production: the six failure modes

Before reaching for a fix, it helps to know precisely what goes wrong. Analysis of production LLM pipelines across 2 million API calls (TokenMix, 2026) found that without any form of schema enforcement, JSON parsing fails 8 to 15 per cent of the time — a rate that sounds small until your pipeline handles a thousand requests a day and every failure is a thrown exception or a silently wrong downstream write.

The failures cluster into six distinct modes, and they require different interventions:

Schema violations. The model produces syntactically valid JSON that does not match the schema — a missing required field, an integer where a string was expected, a discriminator field with an unexpected value. This is the most common failure and the one prompting patterns fix most directly.

Truncation. Long nested schemas push the output towards the context window limit. The model begins generating a valid response, runs out of space, and stops mid-JSON. The result is unparseable. This is a schema design problem as much as a prompting problem — schemas that require hundreds of tokens to fill are fragile.

Partial JSON. A subtler variant of truncation: the model completes the JSON structure but omits optional nested objects entirely rather than filling them with null. Validators that expect the keys to be present even when empty will reject the output.

Model refusals. Some schemas inadvertently trigger safety classifiers. A schema that asks for a threat_level field or a personal_data object may cause the model to refuse the tool call entirely, or to fill the field with a disclaimer string rather than the expected value. Renaming the field usually resolves this — risk_category instead of threat_level, for example.

Schema complexity thresholds. All three major providers degrade measurably as schema nesting depth increases. Research published in early 2026 (DeepJSONEval) found strict evaluation scores declining by 17 to 37 percentage points for five-to-seven-level nested tasks compared with flat schemas. OpenAI enforces a hard limit of five levels of depth; Anthropic and Gemini reject schemas above internal complexity thresholds. Hitting the threshold produces either a hard API error or silently poor output, depending on the provider.

Hallucinated field values. The model fills every required field, the JSON validates, but the values are invented. An invoice_date that was not in the source text, a confidence_score that reflects nothing, a category that is plausible but wrong. Structural compliance is not the same as semantic correctness — the chain-of-thought patterns in section four target this specifically.

Native API modes: tool_use, JSON schema and response_schema — what each guarantees

Each major provider offers a native structured output mechanism that goes beyond asking the model to "please return JSON". Understanding what each mechanism actually guarantees — rather than what it implies — is the foundation for choosing the right approach.

Anthropic: tool_use. The Anthropic approach is to define your schema as a tool with an input_schema property and instruct the model to call that tool. This activates a constrained decoding path that achieves around 99.8% schema compliance (per Anthropic documentation and independent benchmarks including TokenMix 2026), compared with 85 to 92% for unguided JSON prompting. The guarantee is that the response will conform to your JSON Schema; it does not guarantee the field values are factually correct.

OpenAI: JSON schema mode with strict: true. OpenAI distinguishes between two modes. JSON mode (response_format: {type: "json_object"}) guarantees syntactically valid JSON but makes no promise about schema conformance — it is considered legacy for production use. JSON schema mode (response_format: {type: "json_schema", json_schema: {...}, strict: true}) uses a Context-Free Grammar engine that masks invalid tokens before generation: the model physically cannot produce output that violates your schema. As of June 2026 this is supported across GPT-5.5, GPT-5.4 and GPT-5.4-mini (per OpenAI documentation). With strict: true, required fields are always present and type constraints are always respected.

Gemini: response_schema. Gemini exposes structured output via a response_mime_type of application/json and a response_schema parameter. Like the other providers, this activates constrained generation rather than post-hoc parsing. Gemini achieves around 99.7% compliance on well-formed schemas (per Google AI documentation). The key limitation is that Gemini supports a subset of the JSON Schema specification — recursion is not supported, enums with many values can trigger rejections, and very deeply nested schemas produce InvalidArgument: 400 errors. The API documentation explicitly advises simplifying schemas that fail: shorter property names, reduced nesting, fewer constraints.

Provider	Mechanism	Compliance rate	Max nesting depth	Latency overhead
Claude (Anthropic)	tool_use / native Structured Outputs (output_format)	~99.8%	No hard limit; internal threshold applies	150–300 token overhead; ~10–15% latency
GPT-5 (OpenAI)	JSON schema mode with strict: true	~99.9%	5 levels (hard limit)	Schema compilation ~50–200 ms first call; cached thereafter
Gemini 2.5 (Google)	response_schema + application/json	~99.7%	No hard limit; complex schemas rejected at API level	~10–20% latency; schema tokens count toward input limit
Any model (prompt-only)	Prose instruction + few-shot examples	85–92%	Degrades sharply above 3 levels	Minimal overhead; no compilation step

The practical consequence is clear: for anything in production, use native structured output modes rather than prose-only prompting. The compliance gap between 85 to 92% (prompt-only) and 99.7 to 99.9% (constrained decoding) translates directly to error rates in your pipeline — at 1,000 requests per day, moving from 90% to 99.8% compliance reduces daily parse failures from 100 to 2. The sections below assume you are using native modes and focus on the prompting patterns that make native modes work at their best.

Schema-first prompting: put the schema in the system prompt too

A common mistake is to define the schema only in the API parameter — the input_schema on the tool, or the json_schema in the response format — and leave the system prompt silent about structure. This works for simple flat schemas but degrades on complex ones. The reason is that constrained decoding only enforces structure; it does not help the model understand the meaning of each field, the expected range of values, or the relationship between fields. That understanding lives in the system prompt.

Schema-first prompting means defining the schema in both places: the API parameter for structural enforcement, and the system prompt for semantic orientation. The system prompt version does not need to be a literal JSON Schema object — a clear prose description with examples of expected values is often more effective for the model's comprehension than a machine-readable schema it has to parse. The combination raises not just structural compliance but value-level quality.

SYSTEM = """
You are an invoice extraction assistant. You will be given raw invoice text and
must extract the following structured fields. Use the extract_invoice tool.

Fields and expected values:
- invoice_number: string, e.g. "INV-2026-00123". Required. If absent, return null.
- invoice_date: ISO 8601 date string, e.g. "2026-06-11". Required. If absent, return null.
- vendor_name: string — the name of the company issuing the invoice. Required.
- line_items: array of objects, each with:
    - description: string — the item or service description
    - quantity: number — must be positive
    - unit_price: number — must be positive, in the currency of the invoice
    - total: number — quantity × unit_price; verify the arithmetic
- currency: ISO 4217 code, e.g. "GBP", "INR", "USD". Required.
- total_amount: number — the grand total as stated on the invoice. Required.
- tax_amount: number or null — any stated VAT/GST amount. Null if not present.

Important: only extract values explicitly stated in the invoice text.
Do not infer or calculate values that are not written down, except for
line_item.total which you may verify arithmetically.
"""

Several details in that system prompt are doing real work. The explicit instruction "only extract values explicitly stated" reduces hallucination of missing fields. The arithmetic note on line_item.total invites the model to apply a check rather than blindly copy a number that may be wrong in the source. The examples of expected values — "INV-2026-00123", "GBP" — orient the model on format without over-constraining it. And the null instructions for optional fields prevent the model from inventing plausible-sounding values to avoid leaving a field empty.

Chain-of-thought scaffolding for complex schemas

For flat schemas with a handful of fields, schema-first prompting and native structured output modes are sufficient. When schemas grow complex — three or more levels of nesting, conditional discriminator fields, arrays of heterogeneous objects — a different failure mode becomes dominant: the model fills every field according to the schema but the values are internally inconsistent or drawn from the wrong part of the source text. Structural compliance is high; semantic correctness is low.

The pattern that addresses this is "think before you fill": ask the model to produce a brief reasoning step before populating the structured fields. This is not chain-of-thought prompting in the traditional sense of asking for step-by-step reasoning across a long problem — it is a lighter, targeted technique where the model summarises its understanding of the input in a scratch field before committing to structured values. The effect is that the model orients itself on the source material before beginning to fill schema fields sequentially, which reduces both hallucination and internal inconsistency.

The practical way to implement this with native structured output is to add a reasoning or scratch_pad field at the top of your schema — a string field where the model is instructed to write its brief analysis before filling the remaining fields. Because JSON objects in most serialisers produce fields in schema order, placing scratch_pad first means the model generates its reasoning before generating any structured value, which is exactly when reasoning helps.

EXTRACT_CONTRACT_TOOL = {
    "name": "extract_contract",
    "description": "Extract structured fields from a legal contract document.",
    "input_schema": {
        "type": "object",
        "properties": {
            "scratch_pad": {
                "type": "string",
                "description": (
                    "Before filling the fields below, briefly identify: "
                    "(1) the parties and their roles, "
                    "(2) the contract type, "
                    "(3) any ambiguous or missing fields you will need to handle."
                )
            },
            "contract_type": {
                "type": "string",
                "enum": ["service_agreement", "nda", "employment", "licensing", "other"]
            },
            "parties": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name":  {"type": "string"},
                        "role":  {"type": "string", "enum": ["client", "vendor", "employee", "employer", "licensor", "licensee"]},
                        "jurisdiction": {"type": "string", "description": "Country or state/region of incorporation, e.g. 'England and Wales', 'Maharashtra, India'"}
                    },
                    "required": ["name", "role"]
                }
            },
            "effective_date":    {"type": ["string", "null"], "description": "ISO 8601 date or null if not stated"},
            "termination_date":  {"type": ["string", "null"], "description": "ISO 8601 date or null if not stated"},
            "governing_law":     {"type": ["string", "null"], "description": "Jurisdiction clause, e.g. 'English law'"}
        },
        "required": ["scratch_pad", "contract_type", "parties", "effective_date", "termination_date", "governing_law"]
    }
}

The scratch_pad field is worth keeping in your stored output during development — it gives you a readable audit trail of what the model understood about each document and is invaluable for diagnosing systematic extraction errors. In production you can strip it from the output before writing downstream if it is not needed by consumers.

A note on the research: some studies have found that unstructured chain-of-thought prompting can increase hallucinations in certain complex open-ended reasoning tasks. The think-before-you-fill pattern is not the same thing — it is a tightly scoped, structured reasoning step over a specific document, not an open-ended reasoning chain. The key difference is that the reasoning is constrained to "what is in this document" rather than "reason about this problem generally". When used this way, the pattern consistently reduces value-level errors in extraction tasks.

Three working few-shot templates

Few-shot examples in the system prompt are the single most reliable way to communicate expected output format to a model. A schema tells the model what fields to produce; a few-shot example tells it how a good answer looks in practice. Here are three working templates covering the common production patterns.

Template 1: Simple flat object. Suitable for classification, tagging or single-entity extraction with no nesting.

SYSTEM = """
Classify the support ticket into the following structure.
Use the classify_ticket tool.

Example:
User message: "My payment failed but I was still charged twice"
Tool call output:
{
  "category": "billing",
  "subcategory": "duplicate_charge",
  "urgency": "high",
  "sentiment": "frustrated",
  "requires_human_review": true
}
"""

Template 2: Nested schema with arrays. For documents with multiple line items, events or entities. Note the explicit example of an empty array — this prevents the model from omitting the key when no items are found.

SYSTEM = """
Extract all action items from the meeting notes using the extract_actions tool.

Example:
Meeting notes: "Alice to send the revised spec by Friday.
Bob will schedule the client call. No blockers raised."
Tool call output:
{
  "meeting_summary": "Spec revision and client call scheduling discussed.",
  "action_items": [
    {
      "assignee": "Alice",
      "task": "Send revised spec",
      "due_date": "2026-06-13",
      "priority": "normal"
    },
    {
      "assignee": "Bob",
      "task": "Schedule client call",
      "due_date": null,
      "priority": "normal"
    }
  ],
  "blockers": []
}

Note: if no action items are found, return an empty array: "action_items": []
"""

Template 3: Conditional fields. For schemas where certain fields are required only when a discriminator field has a specific value. This pattern is one of the most common sources of schema violations — models fill conditional fields regardless of the discriminator, or omit them when they should be present.

SYSTEM = """
Classify each document using the classify_document tool.

The schema has conditional fields:
- If document_type is "invoice", you MUST populate invoice_details and set contract_details to null.
- If document_type is "contract", you MUST populate contract_details and set invoice_details to null.
- If document_type is "other", set both invoice_details and contract_details to null.

Example A (invoice):
{
  "document_type": "invoice",
  "confidence": 0.97,
  "invoice_details": {"vendor": "Acme Ltd", "total": 1250.00, "currency": "GBP"},
  "contract_details": null
}

Example B (contract):
{
  "document_type": "contract",
  "confidence": 0.91,
  "invoice_details": null,
  "contract_details": {"parties": ["Acme Ltd", "Beta Corp"], "governing_law": "English law"}
}
"""

The explicit conditional logic in the system prompt — "If X, you MUST populate Y and set Z to null" — combined with worked examples for each branch dramatically reduces the conditional-field violation rate. Without it, models frequently fill all optional objects regardless of the discriminator value, because they are trying to be helpful rather than schema-compliant. For more on prompting strategies that handle complex instructions, see our guide on plan-first workflows for Claude Code which applies similar decomposition thinking to agentic tasks.

Constrained decoding vs prompt-only: when to use each

Constrained decoding — the mechanism behind native structured output modes — works by computing a finite state machine (FSM) from your JSON Schema and using it to mask invalid tokens at each generation step. The model cannot produce output that violates the schema because invalid tokens are simply unavailable at each position. The compliance rate is effectively 100% for structural constraints (field presence, types, enum values). The trade-offs are real but manageable.

The primary overhead is schema compilation: the first time you submit a schema to a provider, the FSM must be built. This takes 50 to 200 milliseconds. For subsequent requests with the same schema, the compiled FSM is cached and the per-token overhead is under 40 microseconds on modern inference engines (XGrammar, the default backend for vLLM and SGLang as of March 2026). Early-generation constrained decoding added 50 to 200% latency in the worst case; the 2026 generation adds approximately 10 to 30%, mostly from the first-request compilation.

Prompt-only approaches — no API-level schema enforcement, just system prompt instructions and few-shot examples — have lower baseline overhead and work with any model regardless of whether it supports structured output modes. They are appropriate when: the output is simple (flat, 5 or fewer fields); the downstream consumer can tolerate occasional repair retries; you are prototyping and the schema is changing frequently enough that compilation overhead is annoying; or you are using a model or provider that does not support native structured output.

One scenario that catches teams off guard: schema complexity can push you out of constrained decoding and back to prompt-only against your will. A schema that exceeds a provider's complexity threshold is rejected at the API level — you get a validation error before the model even runs. The correct response is to redesign the schema, not to retry. Schemas that fail provider validation are also schemas that degrade model quality even with constrained decoding, because the FSM becomes so complex that valid token paths at each position remain very broad. Simplifying the schema is always the right move: break it into two sequential calls, lift deeply nested objects to separate top-level schemas, and use references where the same sub-schema appears multiple times.

Handling violations gracefully: retry, repair prompt and partial extraction

Even with native structured output modes active, violations do occur — though rarely. More importantly, prompt-only paths and legacy integrations will continue to produce failures. A production pipeline needs a layered response strategy rather than a single try-except that either swallows errors silently or propagates them as unhandled exceptions.

The three-tier pattern is: first, attempt native structured output; second, on failure, send a repair prompt; third, if the repair fails, attempt partial extraction. Only raise an unhandled exception or trigger a human-review queue if all three tiers fail.

import json
import anthropic
from pydantic import ValidationError

client = anthropic.Anthropic()

def extract_with_fallback(source_text: str, schema_tool: dict, validator, max_repairs: int = 2):
    """
    Three-tier structured extraction with repair and partial fallback.

    validator: a Pydantic model class (or any callable that raises on invalid input).
    """
    messages = [{"role": "user", "content": source_text}]

    # Tier 1: native structured output via tool_use
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        tools=[schema_tool],
        tool_choice={"type": "tool", "name": schema_tool["name"]},
        messages=messages,
    )

    raw = None
    for block in response.content:
        if block.type == "tool_use" and block.name == schema_tool["name"]:
            raw = block.input
            break

    if raw is None:
        return _partial_extract(source_text, schema_tool)

    # Validate — structural compliance should be ~100%, but value-level validation may fail
    try:
        return validator(**raw)
    except (ValidationError, ValueError) as exc:
        error_msg = str(exc)

    # Tier 2: repair prompt — up to max_repairs attempts
    for attempt in range(max_repairs):
        repair_messages = [
            {"role": "user", "content": source_text},
            {
                "role": "assistant",
                "content": [{"type": "tool_use", "id": "repair_0", "name": schema_tool["name"], "input": raw}]
            },
            {
                "role": "user",
                "content": (
                    f"The output above failed validation with this error:\n{error_msg}\n\n"
                    "Please call the tool again with corrected values. "
                    "Only change what is needed to fix the validation error. "
                    "Do not alter values that were already correct."
                )
            }
        ]
        repair_response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=2048,
            tools=[schema_tool],
            tool_choice={"type": "tool", "name": schema_tool["name"]},
            messages=repair_messages,
        )
        for block in repair_response.content:
            if block.type == "tool_use" and block.name == schema_tool["name"]:
                raw = block.input
                break
        try:
            return validator(**raw)
        except (ValidationError, ValueError) as exc:
            error_msg = str(exc)

    # Tier 3: partial extraction — return whatever fields are valid, flag the rest
    return _partial_extract(source_text, schema_tool)


def _partial_extract(source_text: str, schema_tool: dict) -> dict:
    """
    Last-resort: ask for only the top-level required fields in a flat structure.
    Returns a dict with an 'extraction_partial': True flag for downstream handling.
    """
    required_fields = schema_tool["input_schema"].get("required", [])
    simple_prompt = (
        f"Extract only these fields from the text as a JSON object with no nesting: "
        f"{', '.join(required_fields)}. "
        f"If a field is not present, use null. Return only the JSON object."
    )
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": f"{simple_prompt}\n\nText:\n{source_text}"}],
    )
    text = response.content[0].text.strip()
    # Attempt to parse; wrap in try/except — this is the final fallback
    try:
        partial = json.loads(text[text.index("{"):text.rindex("}") + 1])
        partial["extraction_partial"] = True
        return partial
    except (ValueError, json.JSONDecodeError):
        return {"extraction_partial": True, "raw_response": text}

Several design choices in this code are worth noting. The repair prompt includes the original error message verbatim — this is the single most important ingredient; without it the model has no idea what to fix. The instruction "only change what is needed to fix the validation error" prevents the model from "helpfully" rewriting the entire output. The max_repairs=2 default is conservative: a third repair attempt rarely succeeds and adds latency; if two repairs fail, the schema or source text is the problem. The partial extraction tier deliberately strips all nesting — if the full nested schema is failing, a flat extraction with only top-level required fields is far more likely to succeed and gives downstream systems something to work with rather than nothing. For more on building robust evaluation loops around output quality, see our guide to building an LLM evaluation suite with golden sets and judge models.

The nested-schema trap and multi-model comparison

The most common self-inflicted wound in structured output engineering is a schema that works perfectly in isolation and silently degrades when real production data arrives. The mechanism is schema complexity: as nesting depth increases, all models degrade — constrained decoding cannot compensate for a schema so complex that virtually any token is valid at most positions in the output.

The threshold at which degradation becomes measurable is around three to four levels of nesting. Below three levels, compliance rates on well-formed schemas are consistently above 99% across providers. Above four levels, strict evaluation scores drop by 17 to 37 percentage points on hard tasks (DeepJSONEval, 2026). Above five levels, OpenAI returns a hard error; Anthropic and Gemini may return an API rejection or silent quality loss depending on schema properties such as property name length, enum cardinality and the number of optional properties.

The diagnostic signal to watch is not error rate alone but the pattern of errors. If you see the model consistently omitting the same nested object, consistently filling a discriminator field correctly but its dependent children incorrectly, or consistently producing valid JSON that fails value-level validation, the schema complexity is the likely cause. The correct response is schema decomposition: rather than one deep schema, two sequential calls with flat schemas. The first call extracts top-level entities; the second call takes each entity and extracts its sub-fields. Total latency increases modestly; value-level accuracy improves substantially.

The table below summarises the multi-model comparison as of June 2026, drawing on published benchmarks and provider documentation:

Dimension	Claude 3.7 / Sonnet 4	GPT-5.4 / GPT-5.4-mini	Gemini 2.5 Pro / Flash
Structural compliance (native mode)	~99.8%	~99.9% (strict: true)	~99.7%
Structural compliance (prompt-only)	~88–92%	~85–90%	~86–91%
Max schema depth (hard limit)	No hard limit; internal threshold	5 levels	No hard limit; API rejects complex schemas
Latency overhead (native mode)	+10–15%; 150–300 token overhead	+50–200 ms first request (FSM compile); near-zero cached	+10–20%; schema tokens count toward input limit
Conditional / discriminator fields	Strong with explicit system prompt guidance	Strong; CFG enforces enum constraints strictly	Moderate; optional properties with many values can fail
Value-level accuracy (nested schemas)	High; benefits from scratch_pad reasoning pattern	High; extended-thinking variants further reduce hallucination in complex schemas	Moderate on deep nesting; recommend decomposition above 3 levels
Schema complexity sensitivity	Prohibits recursive schemas; rejects above internal threshold	5-level hard limit enforced at API level	Sensitive to property name length, enum cardinality, optional property count

A practical observation across all three providers: value-level accuracy on complex schemas differs more between prompting approaches than between models. A well-crafted schema-first system prompt with think-before-you-fill and appropriate few-shot examples narrows the gap between providers substantially. Choosing the "best" model for structured output without investing in prompt engineering is leaving the majority of the available quality on the table.

For a deeper look at how context engineering relates to prompt engineering in 2026 agentic pipelines, see our news article on whether context engineering has replaced prompt engineering. The structured output domain is one where both still matter: context engineering (what you put in the context window) and prompt engineering (how you instruct the model) each have distinct levers that compound when applied together. Related reading on the tool-calling side — the layer above structured output in an agentic pipeline — is in our piece on LLM tool calling and function schemas for agents.

Production checklist: ten items before you ship

Bringing together all of the patterns above, here is the pre-ship checklist for any feature that relies on structured output from an LLM. It is also a useful frame for reviewing an existing integration that is producing errors in production.

1. Use native structured output mode. Confirm that the API call uses tool_use, strict JSON schema mode, or response_schema — not plain JSON mode or prompt-only. Log the mode in your observability layer.
2. Schema in the system prompt and the API parameter. The schema is defined in both places: API parameter for structural enforcement, system prompt for semantic orientation with field descriptions and worked examples.
3. Null handling is explicit per optional field. Every optional field has an explicit instruction in the system prompt: "if not present, return null". Do not rely on the model's default behaviour.
4. Schema depth is three levels or fewer. If the schema needs more than three levels, document why, and plan a decomposition strategy for when compliance begins to degrade. Four or more levels is a yellow flag; five or more is a red flag.
5. Conditional fields have worked examples for every branch. If the schema has conditional or discriminator fields, there is a few-shot example in the system prompt for each meaningful branch value.
6. A retry-and-repair loop is implemented. There is a validation step after every structured output call, with at least one repair attempt that includes the error message in the repair prompt.
7. A partial extraction fallback exists. If all repair attempts fail, the pipeline falls back to extracting only required top-level fields in a flat request, and flags the result with an extraction_partial marker for downstream handling.
8. Value-level validation is separate from structural validation. There is a layer of business-logic validation (field ranges, cross-field consistency, referential integrity) in addition to the structural schema validation. Structural compliance does not imply value correctness.
9. All failures are logged with schema version and model. Every parse failure, validation failure and repair attempt is logged with the schema version identifier, model name, and error message so that schema complexity problems are visible as patterns over time.
10. The schema is versioned and changes go through a compliance check. Schema changes are treated as interface changes: new required fields require a migration plan, field renames are tracked, and any schema change is tested against a golden set of source documents before deployment.

Teams that work through this checklist before shipping typically find one or two items they have not addressed. The ones most often missed are items 3 (null handling), 7 (partial extraction fallback) and 10 (schema versioning). Items 3 and 7 prevent the quiet data-quality failures that only surface weeks after launch. Item 10 prevents the infrastructure debt that accumulates when a schema changes in production and downstream consumers start receiving fields they did not expect — or stop receiving fields they relied upon.

If you are working on multi-agent systems where structured output feeds into tool-calling pipelines, the agent memory management guide at agent memory management patterns for 2026 covers the architectural patterns for passing structured state between agents. The structured output layer you build here is the contract between agents: get it right at the prompting level before you build the orchestration on top of it.