Few-Shot Prompting in Production: Count and Example Selection

What you need to know

Few-shot prompting — pasting a handful of worked examples into a prompt before asking the model to handle a new case — is one of the oldest tricks in the prompt-engineering toolbox, and it remains one of the most reliably useful. It is also one of the most sloppily implemented. Most production prompts ship with either too many examples dragged in from an early experiment and never revisited, or too few, chosen at random from whatever data was closest to hand, in whatever order they happened to be typed. Both mistakes cost real accuracy and real money, and both are fixable with a few hours of deliberate work.

This guide is a practical walk-through of what the research and the production evidence actually say: how many examples typically earn their keep before returns diminish, how to select examples that generalise rather than examples that merely look tidy, why the order you place them in is not a cosmetic choice, how prompt caching changes the cost calculus of running more examples, and how to measure the impact properly instead of shipping on a hunch.

• Shot count — most classification and extraction tasks plateau within three to eight well-chosen examples; complex reasoning tasks can benefit from more.
• Selection — diverse, representative examples, chosen deliberately or retrieved by similarity, consistently beat randomly sampled ones.
• Order — the sequence you place examples in can matter as much as which examples you pick.
• Caching — prompt caching changes the cost of running more examples, but only if your examples are stable and your call volume is high enough to reuse the cache.
• Retrieval — dynamic, retrieval-based few-shot usually wins on messy or heterogeneous inputs, at the cost of extra infrastructure.
• Measurement — none of the above is safe to guess. Confirm it with an eval harness or an A/B test before you ship it.

How many examples actually help — the shot-count curve

The foundational result here goes back to the original GPT-3 paper, which showed that moving from zero examples to a handful produced a large jump in task performance, with the model settling into a much shallower improvement curve as more examples were added within its available context. That basic shape — steep early gains, then diminishing returns — has held up across the years and across model families since, even as context windows have grown by orders of magnitude.

In practice, that means the first two or three examples usually do most of the work. They establish the input-output format, disambiguate an otherwise vague instruction, and anchor the model's sense of what "correct" looks like for your task. Examples four through eight typically add smaller, still-real gains — covering an edge case, a second label class, a formatting quirk. Beyond that point, more examples often stop helping and can start actively hurting: a recent study specifically examining this "over-prompting" failure mode found that stuffing a prompt with too many examples can degrade performance rather than improve it, likely because irrelevant or redundant examples dilute the signal and consume attention that would otherwise go toward the actual query.

Task complexity shifts where the plateau sits. A binary sentiment classifier or a simple field-extraction task for, say, triaging Hindi-English support tickets into billing, technical or account-access categories will often saturate at three to five examples — there just are not that many distinct patterns to demonstrate. A multi-step reasoning task, such as walking through whether a UK financial-services communication trips a specific compliance flag, has more moving parts to cover and can keep earning marginal gains out to a dozen or more examples before it flattens.

Selecting examples: diverse and representative beats random

Which examples you show matters more than most teams assume, and the research is fairly consistent on the direction of the effect: examples chosen for diversity and representativeness — covering the input distribution rather than clustering around whatever was easiest to label — outperform randomly sampled ones. A widely cited study on selective annotation found that random retrieval over an unrepresentative pool of examples fails to realise the benefit that a deliberately diverse, well-annotated pool provides, and that combining thoughtful selection of the underlying example pool with similarity-based retrieval at inference time is what actually drives the gain, rather than either technique alone.

The dominant practical technique for "similarity-based retrieval" is to embed a bank of labelled examples, embed the incoming query, and pull the nearest neighbours by cosine similarity — an approach with roots in early work on what makes good in-context examples for large language models, and now a standard building block in production retrieval-augmented prompting. For a builder in Bengaluru handling a support-ticket stream that mixes English, Hindi and Hinglish, this means the examples the model sees for a given ticket are ones that actually resemble it in language and topic, not a fixed English-only set that happened to be in the original prompt draft. For a compliance team in London reviewing customer communications, it means the model sees examples closest to the specific pattern it needs to flag, rather than a generic set of "compliance-adjacent" text.

One nuance worth knowing: research on selection strategy has also found that enforcing a strict per-label balance across your example set can backfire, by forcing in redundant examples just to hit a quota. Selecting across labels, prioritising genuinely different inputs over evenly-split categories, tends to preserve more of the diversity that makes the technique work in the first place.

Order effects: the same examples, a different sequence, a different answer

This is the finding that surprises builders most. A now well-known study on prompt order sensitivity found that simply reshuffling an identical set of few-shot examples — same content, same count, different sequence — could move a classification task's accuracy from near state-of-the-art down to close to random chance on some benchmarks, with no change to the examples themselves. The proposed explanation is a form of recency bias: models tend to weight the examples nearer the end of the prompt more heavily than the ones earlier on, so whichever example lands last has an outsized influence on how the model interprets the task.

The practical takeaway is not that you need to solve order optimisation formally — most teams don't have the budget to search permutations the way the original research did. It is that order deserves the same deliberateness as selection: put your clearest, most representative example last, keep weaker or edge-case examples earlier where their influence is smaller, and — critically — keep the order stable once you ship it. A prompt that silently reorders its few-shot block between calls (for instance, because examples are pulled from an unordered set or dictionary) is running a different, untested prompt on every request.

When many-shot changes the picture

Everything above assumes the small-to-moderate shot counts most production prompts actually use. Long-context models have opened up a genuinely different regime: many-shot in-context learning, where a prompt carries not three or eight examples but hundreds or, in research settings, thousands. Google DeepMind researchers demonstrated in 2024 that expanding well beyond the traditional few-shot range produced further, sometimes substantial, gains across a range of generative and discriminative tasks — and introduced variants that generate synthetic chain-of-thought examples with the model itself, to work around the practical bottleneck of not having thousands of human-labelled examples on hand.

Two things temper how relevant this is to most production systems. First, cost scales roughly linearly with example count in the uncached case — doubling your examples roughly doubles the tokens you pay for on every call, so many-shot only makes economic sense with caching in place, or for very high-value tasks where the accuracy gain clearly outweighs the spend. Second, most production tasks that reach for prompting rather than fine-tuning are exactly the kind of narrow, well-scoped classification or extraction problems that plateau in the three-to-eight-shot range described earlier — many-shot's biggest wins have shown up on harder, more open-ended tasks with a genuinely large space of patterns to demonstrate. Know that the many-shot regime exists and works before reaching for it, but do not assume your ticket-triage prompt needs 200 examples when eight would do.

The caching economics of "how many"

Shot count is not just an accuracy question — it is a cost question, and prompt caching has changed that cost question meaningfully. As of mid-2026, the major model providers all discount cached prefix tokens by roughly 90% relative to a fresh, uncached input token on a cache hit, though the mechanics of getting there differ. Anthropic charges a premium on the token that first populates the cache — about 1.25× the base input price for a five-minute cache lifetime, or roughly double for an extended one-hour lifetime — after which repeat reads of that same prefix cost a fraction of the base rate. OpenAI applies its cache discount automatically with no separate write charge, and as of 2026 defaults to a much longer cache retention window than the short-lived caches common a couple of years earlier. Google's Gemini models offer a comparable cache-hit discount but structure explicit caching around a separate storage charge rather than a write premium — a genuinely different cost shape worth checking against your own request volume before assuming it behaves like Anthropic's model.

What this means for shot count: if your few-shot examples are static and sit in a stable prefix — a system prompt that does not change between requests — and your call volume is high enough that the same cached prefix gets reused many times before it expires, the marginal cost of each additional example drops sharply after the first cache write. That changes the calculus of "can we afford eight examples instead of three" quite a bit compared with a cold, uncached call, where every example is paid for in full on every single request. Our deep dive on prompt caching to cut LLM bills and the cross-provider comparison in prompt caching across Anthropic, OpenAI and Gemini both go deeper on the provider-specific mechanics.

The corollary matters just as much: if your traffic is bursty enough that cache entries routinely expire before being reused, or if your few-shot examples change on every request because you are doing per-query retrieval, you are largely paying the uncached rate regardless of provider. In that world, a smaller, well-chosen static set can beat a larger dynamically retrieved one purely on cost, even if the dynamic set would have been marginally more accurate.

Dynamic retrieval-based few-shot: choosing examples per query

Static few-shot works well when your inputs are relatively homogeneous. It works less well when they are not — when a support queue spans a dozen product lines, several languages, and both routine questions and edge-case complaints, no fixed set of three to eight examples can be representative of every incoming ticket. This is where dynamic, retrieval-based few-shot earns its complexity: instead of hard-coding a fixed example block, you maintain a larger bank of labelled examples, embed each incoming query, retrieve the nearest neighbours from that bank, and assemble the prompt's few-shot section fresh for every request.

The evidence for this approach is task-specific but consistently positive where it has been tried. In one published study on a biomedical entity-extraction task, swapping a fixed few-shot prompt for one built from retrieval — first with simple keyword-based (TF-IDF) retrieval and then with semantic (SBERT embedding) retrieval — produced meaningful accuracy improvements over the static baseline, with the semantic retrieval method generally the stronger of the two. The pattern generalises reasonably well to any task with a large or evolving label space: the wider and messier your input distribution, the more a fixed example set leaves on the table.

The cost is real, though, and worth naming plainly: you now need an embedding pipeline, a vector store or index, a retrieval call on every request, and a diversity constraint so retrieval doesn't just return five near-duplicate examples for a common query pattern. That adds latency — typically tens of milliseconds for a well-indexed lookup, more if your example bank is large or poorly indexed — and it partially works against the caching economics described above, since a dynamically assembled few-shot block cannot sit in a long-lived static cache the way a fixed one can. For teams who want the effect of "good examples per query" with less bespoke retrieval infrastructure, tools like DSPy's programmatic prompt optimisation can compile example selection into the prompt-building step rather than requiring you to hand-roll a retrieval service from scratch.

A worked example: a retrieval-backed prompt template

The sketch below is a simplified support-ticket triage prompt for a fintech operating across both India and the UK — a genuinely heterogeneous input stream, since tickets arrive in English, Hindi and Hinglish, and cover everything from routine KYC document questions to UK-specific compliance flags. The system instruction and output schema are stable and sit first, so they are cacheable; the few-shot examples are retrieved per ticket and capped at five, so the block stays bounded regardless of how large the underlying example bank grows. For more on the structured-output side of this pattern, see our guide on structured output prompting patterns in production.

SYSTEM = """You are a support-ticket triage assistant for a fintech
operating in India and the UK. Read the ticket and return exactly one
JSON object, nothing else.

Categories: billing, kyc-verification, technical, account-access,
            compliance-flag, other
Urgency: low, medium, high

Output schema:
{"category": "...", "urgency": "...", "reason": "<one short clause>"}
"""

FEWSHOT_TEMPLATE = """Example:
Ticket: "{ticket}"
Output: {output_json}
"""

def build_prompt(ticket_text, example_bank, k=5):
    # Retrieve the k nearest labelled examples by embedding similarity,
    # with a diversity constraint so near-duplicates are not all returned.
    retrieved = example_bank.search(ticket_text, k=k, diverse=True)

    shots = "".join(
        FEWSHOT_TEMPLATE.format(ticket=ex.text, output_json=ex.label_json)
        for ex in retrieved
    )

    # Stable content first (cacheable), volatile content last.
    return f"{SYSTEM}\n{shots}\nTicket: \"{ticket_text}\"\nOutput:"

Two design choices worth calling out. First, k=5 is a cap, not a target — measured against a held-out set, not chosen because it felt reasonable. Second, the retrieval call is isolated behind example_bank.search() precisely so the retrieval strategy — TF-IDF, embedding similarity, a hybrid, or eventually a compiled DSPy program — can be swapped and A/B tested without touching the rest of the prompt.

Zero-shot vs 3-shot vs 8-shot vs dynamic retrieval: the rough tradeoffs

The table below is a directional guide, not a single benchmark result — actual numbers vary by task, model and domain, and you should treat the accuracy column as "typical shape of the effect" rather than a number to cite. Use it to reason about which configuration to try first, then confirm with your own evaluation harness.

Setup	Typical accuracy pattern	Token & cost overhead	Latency impact	Best for
Zero-shot	Baseline; weakest on ambiguous formats or unfamiliar labels	Lowest — no example tokens	Lowest	Simple, well-known tasks; no labelled examples available yet
3-shot (static)	Captures most of the available few-shot gain on simple-to-medium tasks	Small, fixed overhead; fully cacheable if stable	Minimal add over zero-shot	The sensible production default while you measure further
8-shot (static)	Modest gain over 3-shot on many tasks; larger on tasks with more label classes or edge cases	Roughly 2-3× the example-token overhead of 3-shot; still cacheable	Slightly higher than 3-shot	Tasks with several label classes or known recurring edge cases
Dynamic retrieval-based few-shot	Strongest on heterogeneous or evolving inputs; little advantage on narrow, uniform tasks	Similar per-call token cost to fixed k-shot, plus embedding/retrieval calls	Adds a retrieval hop; typically tens of milliseconds with a well-indexed bank	Large or messy label spaces, multilingual input, growing example pools

How to actually measure this

Everything in this guide is a starting hypothesis, not a rule to apply blindly. The only reliable way to know whether three, eight, or a dynamically retrieved set of examples is right for your task is to measure it against your own data, with your own model, on your own metric. Two complementary approaches do this well.

The first is an offline evaluation harness: a held-out, labelled dataset that was not used to write or select your examples, run through each candidate configuration — zero-shot, 3-shot, 8-shot, dynamic retrieval — and scored on the metric that actually matters for your task, whether that is exact-match accuracy, an F1 score against labelled spans, or an LLM-as-judge rubric for open-ended outputs. Our guide to building a first LLM evaluation suite with golden sets and judges covers how to build this from scratch, and wiring it into continuous integration for prompt and agent regression means a shot-count or selection-strategy change that quietly regresses accuracy gets caught before it ships, not after a customer complains.

The second is a live A/B test: route a percentage of real production traffic to the candidate configuration, and compare business-relevant outcomes — resolution rate, escalation rate, human-correction rate — against the incumbent. This matters because offline evaluation sets can drift out of sync with what real traffic actually looks like, particularly for a task with a genuinely long tail of input variety. Track cost and latency alongside accuracy in both approaches; a configuration that wins on accuracy but doubles your per-request spend or adds two hundred milliseconds of retrieval latency is not automatically the right choice — that is a product decision, not just a modelling one.

Common mistakes worth avoiding

• Stale examples. Examples chosen during an early prototype, never revisited as the task, model or user base evolved.
• Over-stuffing. Adding examples "just in case" past the point where they are measurably helping, paying token cost for nothing.
• Random selection as a permanent default. Fine for a first draft; not a substitute for deliberate or retrieval-based selection once the prompt matters.
• Unstable ordering. Examples pulled from a data structure or process that does not guarantee a fixed sequence between calls.
• Ignoring cache placement. Putting volatile, per-request content ahead of the stable few-shot block, breaking the cache for everything downstream.
• Shipping on instinct. Changing shot count or selection strategy without an evaluation harness or A/B test to confirm the change actually helped.

Where this leaves you

None of this requires exotic infrastructure to get right. Start with three clear, representative examples in a fixed, stable order, placed ahead of any volatile content so the block is cacheable. Measure against a held-out set before you add a fourth. If your inputs are genuinely heterogeneous — multiple languages, a wide label space, an evolving taxonomy — invest in retrieval-based selection once the static approach has demonstrably plateaued, not before. And whichever configuration you land on, write it down, version it, and re-measure it the next time your traffic, your model, or your task changes underneath you. Few-shot prompting rewards exactly the kind of unglamorous discipline — pick deliberately, order consistently, cache what's stable, measure everything — that most other parts of a production LLM system already demand.

Research citations: Su et al., "Selective Annotation Makes Language Models Better Few-Shot Learners" (arXiv:2209.01975); Lu et al., "Fantastically Ordered Prompts and Where to Find Them" (ACL 2022); Agarwal et al., "Many-Shot In-Context Learning", Google DeepMind (arXiv:2404.11018); provider pricing and caching documentation from Anthropic, OpenAI and Google, as published in mid-2026.