Reliable JSON From Any LLM: Constrained Decoding in Production

What you need to know

• There are four levels of reliability. Plain text plus a parser, JSON mode, function/tool calling, and native structured output with constrained decoding. Each is more reliable and slightly more constrained than the last.
• Constrained decoding makes the structure a guarantee, not a hope. Your schema is compiled to a grammar or finite-state machine, and at every step any token that would break the grammar is masked before sampling. The output cannot be structurally invalid.
• Every major provider now supports it. OpenAI Structured Outputs (strict json_schema), Google Gemini (responseSchema), Anthropic (Structured Outputs beta plus strict tool use), and the open-source stack — vLLM guided decoding, Hugging Face TGI, llama.cpp GBNF, plus the Outlines and Instructor libraries.
• Structurally valid is not the same as correct. A response can match your schema perfectly and still be wrong. Validation, bounded retries and evals are the part of the system that constrained decoding cannot do for you.

If you have ever shipped a feature that calls an LLM, extracts a few fields and writes them to a database, you have felt the failure mode. It works in the demo. It works for the first thousand requests. Then a model returns a trailing comma, or wraps the JSON in a Markdown code fence, or invents a key you never asked for, and your json.loads throws at 2am. The instinct is to bolt on a regex, then a retry, then a second regex. The honest fix is to stop asking the model to behave and start making misbehaviour impossible.

That is what constrained decoding does. This is the deep companion to our structured-output prompting patterns guide — that one is about how you phrase the request; this one is about the machinery underneath the request that makes the structure a hard guarantee. If you only read one section, read the one on why valid is not the same as correct.

The four-level maturity model

Teams tend to climb this ladder one rung at a time, usually after each rung breaks in production. It is worth knowing all four up front so you can skip straight to the rung your use case actually needs.

Level 1 — Plain text plus a parser

You ask the model for JSON in the prompt, then parse whatever comes back. This is the "prompt and pray" baseline. It is fine for a prototype and nothing else. The model will, eventually, return prose before the JSON, a code fence around it, a comment inside it, or a subtly different key. You end up writing increasingly defensive extraction code, which is itself a source of bugs.

Level 2 — JSON mode

Most providers expose a "JSON mode" (response_format: { type: "json_object" } on OpenAI-compatible APIs). This guarantees the output is syntactically valid JSON — it will parse. It does not guarantee the shape. You can still get the wrong keys, missing required fields, or an array where you wanted an object. JSON mode is now widely treated as a legacy option for production extraction: it solves the parser-crash problem but not the schema problem.

Level 3 — Function / tool calling

You define a tool with a JSON-schema parameter spec and force the model to call it. The arguments come back as a structured object. This was the workhorse pattern for two years and is still excellent, especially when the structured output is genuinely an action ("call create_ticket with these fields"). The catch historically was that tool arguments were schema-guided, not schema-guaranteed — the model usually matched the schema but could drift on optional fields, enums or deep nesting. Strict tool use closes that gap (see the provider matrix).

Level 4 — Native structured output with constrained decoding

You hand the API a JSON schema and ask for output that conforms to it, with a strict flag set. Under the hood the provider compiles the schema into a grammar and constrains generation so the output cannot violate the schema structurally. This is the top rung. Every required field is present, every type is right, every enum value is legal. The trade-off is that you must be able to express your output as a schema the engine can compile — which rules out a few exotic JSON Schema features, and rewards schemas that are tight rather than free-form.

Level	Mechanism	Guarantees	Best for
1 · Text + parse	Prompt asks for JSON; you parse the string	Nothing	Prototypes only
2 · JSON mode	json_object response format	Valid JSON syntax — any shape	Loose extraction where shape varies
3 · Tool calling	Forced function call with a parameter schema	Structured args; strict mode makes shape exact	When the output is an action to take
4 · Constrained decoding	Schema compiled to a grammar; illegal tokens masked	Exact schema conformance, structurally	Production data extraction and pipelines

How constrained decoding actually works

An LLM generates one token at a time. At each step it produces a probability distribution — a logit — over the entire vocabulary, often 100,000-plus tokens. Normally you sample from that distribution (with temperature, top-p and so on) and append the chosen token. Constrained decoding inserts one step between the logits and the sampler.

First, before any generation, your schema is compiled. A JSON Schema is converted into a context-free grammar or, equivalently, a finite-state machine that describes every string the schema permits. Think of it as a map of "from this state, here are the only characters that could legally come next". After an opening brace, the only legal next thing is a key from your schema (or whitespace); after a key and a colon for an integer field, the only legal characters are digits and a sign.

Then, at each decoding step, the engine asks the grammar which tokens are legal given the partial output so far, and builds a mask. Every token that would move the output into an illegal state has its logit set to negative infinity. After masking, the distribution is renormalised and sampling proceeds as usual — but now it is mathematically impossible to sample a token that breaks the schema. Repeat to the end state and you have output that is guaranteed to parse and to match the shape.

# Conceptual pseudocode — what every grammar engine does at each step
logits = model.forward(tokens)              # raw scores over the whole vocab
mask    = grammar.allowed_token_mask(state) # which tokens keep the output legal
logits[~mask] = float("-inf")               # forbid everything else
next_id = sample(softmax(logits))           # sample only from legal tokens
state   = grammar.advance(state, next_id)   # move the FSM forward

The grammar engines doing this work are worth knowing by name. xgrammar is now the default in vLLM, SGLang, TensorRT-LLM and MLC-LLM, and is the fastest for most schemas. Outlines pioneered the FSM-from-regex/JSON-Schema approach and remains a strong choice when you reuse very complex schemas across many requests, because its one-off compile cost amortises. llama.cpp ships GBNF (GGML BNF), a grammar format you write or generate from a schema, which is how you get constrained output from a local quantised model with no Python in the loop.

The provider matrix

The mechanisms differ in name and ergonomics but the underlying idea is the same everywhere. Here is the landscape as of June 2026 — verify model coverage against each provider's live docs, because supported-model lists move monthly.

Provider / tool	Mechanism	Strict guarantee?	Notes (verify dates in docs)
OpenAI	response_format with json_schema, strict: true	Yes — grammar-constrained	Uses a context-free-grammar engine to mask invalid tokens. JSON mode (json_object) is the older, syntax-only option. Refusals returned in a separate field.
Google Gemini	responseMimeType: application/json + responseSchema	Yes — schema-constrained	Pass a schema (subset of OpenAPI) on the generation config; supports enums and property ordering. Also offers a JSON-only mode without a schema.
Anthropic Claude	Structured Outputs beta (JSON outputs + strict tool use)	Yes — beta; tool-use otherwise	Public-beta feature, header anthropic-beta: structured-outputs-2025-11-13, initially Sonnet 4.5 / Opus 4.1. Compiles the schema into a grammar and restricts generation. Forced single-tool-use remains the long-standing fallback.
vLLM (self-host)	Guided decoding: guided_json, guided_choice, guided_regex, guided_grammar	Yes — grammar-constrained	Backends: xgrammar (default), Outlines, lm-format-enforcer. Works with any open-weight model you serve.
Hugging Face TGI	grammar parameter (JSON schema or regex)	Yes — grammar-constrained	Uses Outlines under the hood. Same guarantee for any model TGI serves.
llama.cpp	GBNF grammar file (--grammar / grammar field)	Yes — grammar-constrained	Write GBNF directly or convert a JSON Schema to GBNF. Ideal for fully local, no-Python deployments.
Outlines (library)	FSM compiled from regex / JSON Schema / Pydantic	Yes — grammar-constrained	Wraps Transformers, vLLM, llama.cpp. Compile-once, reuse-many makes it efficient for hot schemas.
Instructor (library)	Pydantic model → provider strict mode + validate + retry	Inherits provider's; adds re-ask	Thin layer over OpenAI, Anthropic, Gemini and local backends. Returns a typed Pydantic object and retries on validation failure.

OpenAI strict json_schema

The hosted reference implementation. Set strict: true and the model literally cannot produce a non-conforming response.

from openai import OpenAI

client = OpenAI()

schema = {
    "type": "object",
    "additionalProperties": False,
    "required": ["sentiment", "score", "topics"],
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
        "score": {"type": "number", "minimum": 0, "maximum": 1},
        "topics": {
            "type": "array",
            "items": {"type": "string"},
            "maxItems": 5
        }
    }
}

resp = client.chat.completions.create(
    model="gpt-5.2",
    messages=[
        {"role": "system", "content": "Classify the customer message."},
        {"role": "user", "content": "The London delivery was late but support was great."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "classification", "schema": schema, "strict": True},
    },
)

msg = resp.choices[0].message
if msg.refusal:
    handle_refusal(msg.refusal)        # refusal is a first-class branch, not an error
else:
    data = json.loads(msg.content)     # guaranteed to match `schema` structurally

Pydantic + Instructor

If you already model your domain with Pydantic, Instructor lets you skip hand-writing JSON Schema. You define a model, get a typed object back, and Instructor wires the provider's strict mode plus an automatic validate-and-reask loop. Pydantic validators run after decoding, so you can enforce semantic rules the grammar cannot.

import instructor
from openai import OpenAI
from pydantic import BaseModel, Field, field_validator

class Invoice(BaseModel):
    invoice_number: str
    currency: str = Field(pattern="^(INR|GBP|USD|EUR)$")
    total: float = Field(gt=0)
    line_items: list[str] = Field(max_length=50)

    @field_validator("total")
    @classmethod
    def total_is_plausible(cls, v: float) -> float:
        # Semantic guard the grammar can't express
        if v > 10_000_000:
            raise ValueError("total exceeds plausible ceiling; re-ask")
        return v

client = instructor.from_openai(OpenAI())

invoice = client.chat.completions.create(
    model="gpt-5.2",
    response_model=Invoice,        # Pydantic model drives the schema
    max_retries=2,                 # re-ask on validation failure, bounded
    messages=[{"role": "user", "content": raw_invoice_text}],
)
# `invoice` is a validated Invoice instance, fully typed

vLLM guided decoding (self-hosted)

When you serve an open-weight model yourself, the same guarantee is one extra field on the request. vLLM exposes an OpenAI-compatible endpoint, so you add guided_json via extra_body.

from openai import OpenAI

# Point the OpenAI client at your own vLLM server
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B",
    messages=[{"role": "user", "content": "Extract the order details."}],
    extra_body={
        "guided_json": {
            "type": "object",
            "required": ["order_id", "status"],
            "properties": {
                "order_id": {"type": "string"},
                "status": {"type": "string",
                           "enum": ["placed", "shipped", "delivered", "cancelled"]}
            },
            "additionalProperties": False
        },
        "guided_decoding_backend": "xgrammar"
    },
)

The crucial caveat: valid is not the same as correct

This is the section every team learns the hard way. Constrained decoding guarantees structural validity. It says nothing about whether the values are right. The model can return a perfectly schema-conformant object in which the extracted total is off by a decimal place, the date is the invoice date instead of the due date, or the sentiment is confidently wrong. The JSON parses, your types check, and the data is garbage.

Three layers catch this, in order of cost:

1. Schema-level constraints. Push as much semantics as you can into the schema itself. Enums instead of free strings. Numeric minimum/maximum. pattern for currency codes and IDs. maxItems on arrays. Anything the grammar can enforce is something you never have to validate downstream.
2. Application-level validation. Run the decoded object through Pydantic (Python) or zod (TypeScript) with custom validators that encode business rules — a due date after the invoice date, a total that equals the sum of line items, a postcode that matches a UK or India format. These are checks the grammar cannot express.
3. Re-ask and fallback. When validation fails on a rule, re-ask the model with the validation error fed back in, capped at a small number of attempts. After the cap, fall back: route to a larger model, flag for human review, or return a typed error. Never retry unbounded.

// TypeScript: zod validation + bounded re-ask
import { z } from "zod";

const Invoice = z.object({
  invoiceNumber: z.string(),
  currency: z.enum(["INR", "GBP", "USD", "EUR"]),
  total: z.number().positive(),
  lineItems: z.array(z.object({ desc: z.string(), amount: z.number() })),
}).refine(
  (inv) => Math.abs(inv.total - inv.lineItems.reduce((s, li) => s + li.amount, 0)) < 0.01,
  { message: "total must equal the sum of line items" }, // semantic rule
);

async function extractInvoice(text: string, maxRetries = 2) {
  let lastError = "";
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const raw = await callLLMWithSchema(text, lastError); // strict json_schema call
    const parsed = Invoice.safeParse(raw);                // structural pass already guaranteed
    if (parsed.success) return parsed.data;               // semantic checks passed too
    lastError = parsed.error.issues.map((i) => i.message).join("; ");
  }
  throw new Error(`extraction failed after retries: ${lastError}`); // fall back / escalate
}

Beyond the request path, structured tasks need evals like any other LLM feature. Hold out a golden set of inputs with hand-checked expected outputs, and score new prompts and models against it — field-by-field exact match for deterministic fields, and an LLM-as-judge for the fuzzy ones. Our guides on building an LLM evaluation suite with golden sets and LLM-as-a-judge evals that hold up in production cover the mechanics; the point here is that the eval is what tells you whether your schema-valid output is also a correct output before you trust it in a pipeline.

Practical patterns that pay off

Design schemas that are easy to constrain

The single biggest lever is schema design. Prefer an enum to a free string whenever the value comes from a known set — a status, a category, a country code. Enums collapse a whole branch of the grammar into a tiny set of legal tokens, which is both more reliable and faster to decode. Avoid deeply recursive schemas and exotic JSON Schema keywords; not every engine supports the full spec, and a schema that fails to compile is worse than one that is slightly less expressive.

Handle refusals as a branch, not an exception

A safety refusal inside a structured-output call is awkward: the model is being asked to produce a fixed shape, but it also needs a way to decline. OpenAI solves this with a dedicated refusal field on the message. With self-hosted grammars, give the model an escape hatch in the schema itself — an oneOf between your result object and a small refusal object — so a decline is still legal output your router can act on rather than a parse failure.

Streaming structured output

You can stream constrained output token by token, but a half-streamed JSON object is not yet valid JSON. Two patterns work: stream and parse incrementally with a tolerant streaming-JSON parser to drive a progressive UI, or stream into a buffer and only validate-and-commit on completion. For anything that writes to a database, buffer and validate at the end. For a UI that fills in fields as they arrive, incremental parsing with a partial-object library is the better experience.

Cost and latency overhead

Token cost is unchanged — you pay for the tokens generated whether or not the decode is constrained. The overhead is two parts: the one-off schema-to-grammar compile, which you amortise by reusing the schema, and a small per-token masking cost, which modern engines such as xgrammar have driven down to near-negligible for typical schemas. In practice constrained decoding is often faster end-to-end than the unconstrained equivalent, because you stop wasting tokens on Markdown fences, apologies and re-asks — and because a guaranteed-parse output deletes the retry loop entirely.

Conclusion: make the structure impossible to get wrong

The progression is simple. Stop parsing strings and hoping. Move to a schema you can hand to the decoder, set the strict flag, and let the grammar make structural failure impossible. Then spend the effort you used to spend on defensive parsing on the part that actually matters — validating that the structurally-perfect output is also semantically correct, with typed validators, bounded retries and a golden-set eval to catch regressions before they ship.

Constrained decoding is now table stakes across OpenAI, Google, Anthropic and the entire open-source serving stack, so there is no provider lock-in argument against adopting it. The teams that get this right ship LLM features that behave like ordinary software: typed in, typed out, with the failure modes in places you can see and test. The teams that do not are still debugging trailing commas at 2am.

For the prompt-side companion to this piece — how to phrase the request so the model produces good values inside the guaranteed shape — read our structured-output prompting patterns guide. Constrained decoding fixes the shape; good prompting and good evals fix the content.