Meta-Prompting: Using an LLM to Critique and Rewrite Your Prompts

What you need to know

Meta-prompting means using an LLM to operate on prompts themselves — writing a first draft, critiquing an existing one, or proposing a rewrite — instead of a human doing all of that editing by hand. It is not a single technique so much as a family of them, ranging from a quick "what's wrong with this prompt?" question you ask in a chat window, up to fully automated optimisers that search over thousands of candidate instructions without a human reading any of them.

This guide focuses on the version most builders can actually put into production this week: the critique-rewrite loop. You draft a prompt, run it against a small but real evaluation set, ask a second LLM call to diagnose the failures, use that diagnosis to rewrite the prompt, and re-run the eval to confirm the rewrite genuinely helped rather than just reading more confidently. It is a lightweight, largely manual process that sits between traditional hand-tuned prompt engineering and heavier automated optimisation frameworks like DSPy — and it is worth understanding on its own terms before reaching for either extreme.

• The loop is only as good as the eval set — a critique without a scored, held-out set to check against is just a plausible-sounding opinion.
• Naive loops drift — left unchecked, they tend to reward verbosity, wander from the original product intent, and overfit to whatever examples the critic has seen most often.
• It complements, not replaces, manual prompt engineering — a human still owns the product requirements and the final sign-off.
• DSPy-style optimisation is a different, more automated tool — useful once a task is stable enough to define a metric function and let a search process run unsupervised.
• The technique has real published evidence behind it — this is not just vibes; self-critique loops have measurable gains on checkable tasks.

The basic critique-rewrite loop

Strip away the tooling and the pattern is four steps repeated until you stop seeing gains:

1. Draft. Write (or take an existing) prompt — instructions, format constraints, any few-shot examples.
2. Run. Execute the prompt against every item in a labelled evaluation set and score each output against the label or rubric.
3. Critique. Feed the prompt, a sample of its failing outputs, and the expected outputs to a critic LLM call, and ask it to diagnose specific, falsifiable reasons the prompt is producing wrong or weak outputs — not a generic "make it clearer".
4. Rewrite and re-run. Use the critique to produce a new prompt version, run it against the same eval set, and compare the score to the previous version. Keep the rewrite only if the score improved; otherwise, discard it and try a different critique angle.

The loop closes when a rewrite fails to beat the previous best score on your held-out set for two or three consecutive attempts, or when you have hit a pre-agreed budget of iterations. Both stopping conditions matter — without them, it is easy to keep "improving" a prompt against noise in a small eval set indefinitely.

def critique_rewrite_loop(prompt, eval_set, llm, max_rounds=5, patience=2):
    best_prompt = prompt
    best_score, failures = run_eval(best_prompt, eval_set, llm)
    rounds_without_gain = 0

    for round_num in range(max_rounds):
        if rounds_without_gain >= patience:
            break

        critique = llm.complete(
            CRITIQUE_TEMPLATE.format(
                prompt=best_prompt,
                failures=sample_failures(failures, n=8),
            )
        )
        candidate_prompt = llm.complete(
            REWRITE_TEMPLATE.format(
                prompt=best_prompt,
                critique=critique,
                original_intent=ORIGINAL_SPEC,   # anchor against drift
            )
        )

        score, new_failures = run_eval(candidate_prompt, eval_set, llm)

        if score > best_score:
            best_prompt, best_score, failures = candidate_prompt, score, new_failures
            rounds_without_gain = 0
        else:
            rounds_without_gain += 1   # discard candidate, try again

    return best_prompt, best_score

Three details in that sketch matter more than they look. First, the critique call only ever sees a sample of failures, not the whole eval set — this keeps the critique focused and keeps token cost manageable as your eval set grows. Second, ORIGINAL_SPEC is passed into every rewrite, not just the first one — an explicit anchor against the intent drift discussed below. Third, a candidate that does not beat the best score is discarded, not accepted anyway "because the critique sounded reasonable" — the score is the only thing allowed to accept a rewrite.

Building a lightweight eval harness first

None of the above works without something to measure against, and this is the step most teams skip or under-invest in. A usable eval harness for a critique-rewrite loop does not need to be elaborate — it needs three things: a representative set of labelled examples, a scoring function that reduces each output to a comparable number, and a way to sample and inspect failures, not just see an aggregate score.

Start smaller than you think you need. Thirty to fifty examples that genuinely cover your known edge cases — different languages if your product spans India and the UK, different input lengths, the handful of ambiguous cases that caused a support escalation last month — will surface more real signal than five hundred near-duplicate happy-path examples. Split that set in two: a working set the loop iterates against every round, and a smaller held-out set you check only every few rounds, specifically to catch overfitting to the working set before it reaches production.

Scoring depends on the task. Exact-match or field-level accuracy works for structured extraction. An LLM-as-judge rubric works for open-ended generation, provided the judge prompt is itself stable, versioned and periodically checked against human ratings — our guides on building a first LLM evaluation suite with golden sets and judges and LLM-as-a-judge bias and calibration go deeper on getting that right. Whatever the metric, the discipline is the same one used across production ML: never let a prompt change ship on the strength of a critique alone, only on the strength of a score that moved.

Common failure modes of naive meta-prompting

The critique-rewrite pattern is simple to describe and easy to get subtly wrong. Three failure modes show up repeatedly once teams put it into production.

The critic rewards verbosity

Critic and judge models carry a well-documented tendency to favour longer, more heavily qualified text — sometimes called verbosity or length bias — independent of whether the extra text actually improves the output. Left unchecked across many rounds of critique-rewrite, this shows up as prompt bloat: each round adds another caveat, another "be sure to also consider", another paragraph of edge-case handling, until the prompt is three times its original length and marginally worse, because the added instructions dilute the signal the model needs to focus on. The fix is not to avoid critique models — it is to score length as a cost, not a free variable. Cap prompt length explicitly in your rewrite instructions, or track token count alongside your accuracy metric and reject rewrites that gain accuracy only by growing the prompt disproportionately.

Drift from the original intent

Each round of a critique-rewrite loop optimises against the last critique in isolation. Over several rounds, a prompt can accumulate a series of individually reasonable edits that collectively wander away from what the prompt was actually meant to do — a support-triage prompt that started narrowly scoped to billing and technical categories slowly grows extra categories, softened constraints, and hedged language, none of which anyone explicitly asked for. This is the automation-era version of scope creep, and the fix is the same one used in the code sketch above: keep the original product specification in the loop as an explicit, unchanging reference every rewrite is checked against, not just the most recent critique.

Overfitting to a small eval set

A prompt tuned across many rounds against the same twenty or thirty examples will, almost by construction, start fitting quirks specific to those examples rather than the general pattern they were meant to represent — the same overfitting risk that shows up in any iterative optimisation against a fixed, small dataset. A prompt that scores perfectly on its working set can quietly regress on the traffic it never saw. The held-out set discussed above is the direct countermeasure: check it periodically, not every round, and treat a growing gap between working-set and held-out performance as the signal to stop iterating and either grow the eval set or ship the current best version.

What the research actually says

The critique-rewrite pattern is not a novel trick invented for production prompt engineering — it draws directly on a line of published research on self-improving language model outputs. Self-Refine (Madaan et al., 2023) formalised close to this exact loop: a single LLM generates an initial output, the same model critiques its own output, and then refines it based on that critique, with no additional training or supervised data required. Across seven diverse tasks — from dialogue generation to mathematical reasoning — the technique produced an average performance gain of roughly 20% when evaluated with GPT-3.5-era and GPT-4-era models. The core insight that generalises well to prompt-level meta-prompting: the same model that produces an imperfect output is often capable of correctly diagnosing what is wrong with it, even though it could not get it right zero-shot.

Reflexion (Shinn et al., 2023) pushed a related idea further for multi-step agents, introducing three distinct roles — an Actor that produces actions, an Evaluator that scores the outcome, and a Self-Reflection component that turns that score into verbal feedback stored in an episodic memory buffer for future attempts. On the HumanEval coding benchmark, this verbal self-critique loop lifted GPT-4's pass rate from roughly 80% to 91%, entirely through better use of feedback across attempts rather than any change to the underlying model. The separation of roles is directly relevant to prompt meta-critique: a critic call that is explicitly scoped to diagnosis, separate from the call that proposes the rewrite, tends to produce more specific, actionable feedback than a single call asked to both critique and fix in one pass.

On the fully automated end of the spectrum, OPRO — "Large Language Models as Optimizers" (Yang et al., 2023, Google DeepMind) — used an LLM as an iterative optimiser: at each step, the model is shown previously generated prompt candidates alongside their scores and asked to propose a new candidate, gradually converging on stronger prompts purely from that feedback loop, without gradient access to the underlying model. OPRO-optimised prompts outperformed human-written ones by up to 8% on GSM8K and by up to 50% on harder Big-Bench Hard tasks — a useful data point on how much headroom purely prompt-level search can find even without touching model weights. Automatic Prompt Engineer (Zhou et al., 2022) showed a related result earlier: an LLM asked to generate and search over candidate instructions discovered a zero-shot chain-of-thought prompt that outperformed the well-known, human-authored "let's think step by step" phrasing on the majority of tasks tested.

Two caveats worth carrying forward from this research into production use. First, all of these results were measured on tasks with a clear, checkable success signal — code that passes tests, maths answers that are right or wrong, benchmark accuracy. The technique is meaningfully weaker on genuinely open-ended, low-signal tasks, which is exactly why the eval harness matters as much as the critique prompt. Second, none of these papers ran their loops unsupervised forever — each used a bounded number of iterations and a held-out check, the same discipline described in the failure-modes section above.

How this differs from manual prompt engineering

Manual prompt engineering is what most builders still do most of the time: read outputs, form a hypothesis about what is wrong, edit the prompt by hand, and re-test — all judgement calls made by a person who understands the product requirements. It scales poorly with iteration count (a human can only read so many failure cases in a day) but it carries context a critic model does not automatically have — unstated business constraints, tone requirements, regulatory sensitivities specific to, say, a UK financial-services product or an India KYC flow.

Meta-prompting's critique-rewrite loop automates the mechanical parts of that process — reading failures at volume, drafting a diagnosis, proposing a rewrite — while still requiring a human to define the eval set, own the original intent, and sign off on what ships. It is a force-multiplier on manual prompt engineering, not a replacement for the judgement involved in it. Where it earns its keep is in the tedious, high-volume parts of iteration: reading fifty failure cases and spotting the pattern is exactly the kind of task an LLM does quickly and a human does slowly and with fatigue-driven inconsistency.

How this differs from DSPy-style automated optimisation

DSPy takes a structurally different approach, and it is worth being precise about the mechanism rather than treating it as "more automated meta-prompting". DSPy compiles a program by running it against training examples and collecting execution traces, then uses those traces in one of two ways depending on the optimiser: BootstrapFewShot selects the best-performing traces to use as few-shot demonstrations, while an optimiser like MIPROv2 jointly proposes new natural-language instructions and few-shot demonstration sets, then uses Bayesian optimisation to search over combinations of both against your metric function. Nobody reads or approves an intermediate critique in that loop — the search process explores candidates purely by score, and the final compiled program embeds whichever configuration performed best on the validation set, with no further optimisation happening at inference time. Our dedicated guide on DSPy in production walks through the compile step end to end.

The practical difference is supervision and interpretability. A critique-rewrite loop produces a human-readable diagnosis at every step — you can read why the critic thinks the prompt is failing, and reject a rewrite that sounds wrong even if it scored well, because you understand the failure it is responding to. DSPy's search is opaque by comparison: you get a compiled program that scores well on your metric, but not a narrated explanation of why any particular instruction or example was chosen. That trade-off makes critique-rewrite loops a better fit early in a task's life, while requirements are still shifting and you want visibility into every change, and DSPy-style optimisation a better fit once the task and metric are stable enough to trust an unsupervised search to find gains a human iteration cycle would take too long to reach.

Many production teams use both in sequence rather than picking one: a manual or LLM-assisted critique-rewrite pass to get a prompt from broken to workable and to nail down the metric definition itself, followed by a DSPy compile once the task is stable, to search a wider space of instruction and example combinations than a human-supervised loop could practically try. The critique-rewrite loop's real output, in that sequence, is often less the prompt itself and more a well-defined, well-labelled eval set — which is exactly what DSPy's optimisers need to run well.

Approach	Effort to set up	When to use	Main failure mode
Manual prompt engineering	Low — no tooling beyond a place to test prompts	Early exploration, small change volume, high-context or regulated domains	Doesn't scale with iteration count; slow, inconsistent human judgement
Meta-prompting (critique-rewrite loop)	Medium — needs a labelled eval set and two extra LLM calls per round	Stable-ish task, moderate iteration volume, want a human still reading every diagnosis	Verbosity creep, intent drift, overfitting to a small working set
DSPy-style automated optimisation	Higher — needs a defined metric function, training data, and optimiser configuration	Task and metric are stable; want to search a wide instruction/example space unsupervised	Opaque results; metric that doesn't fully capture product intent gets over-optimised

A worked example: bad prompt, critique, improved prompt

Here is a genuinely weak prompt for a fintech support-ticket triage task, spanning both Indian and UK customers, followed by a realistic critique and the rewrite it produced.

Bad prompt (v1):

Look at this support ticket and tell me what category it is and how
urgent it is. Be helpful and thorough.

Ticket: {ticket_text}

Run against a 40-example eval set covering billing, KYC, technical, account-access and compliance-flag tickets in English, Hindi and Hinglish, this prompt scored 61% exact-match accuracy on category and produced wildly inconsistent urgency labels — sometimes a word like "urgent", sometimes a 1–5 scale, sometimes a paragraph of reasoning instead of a label at all.

Critique (generated against the failure sample):

The prompt does not define the category set, so the model invents
categories that don't match your taxonomy on 9 of 15 sampled failures
(e.g. "general enquiry", "other issue" — not valid labels).

It does not specify an output format, so urgency is returned in at
least three different formats across the sample, which is unparseable
downstream.

"Be helpful and thorough" actively encourages long, hedged responses
where a structured one is needed — this instruction works against
your actual requirement and should be removed, not softened.

No examples are given for the two categories with the lowest
per-class accuracy (kyc-verification, compliance-flag), which are
also the two categories most often confused with each other.

Improved prompt (v2):

You are a support-ticket triage assistant for a fintech operating in
India and the UK. Read the ticket and return exactly one JSON object,
nothing else.

Categories: billing, kyc-verification, technical, account-access,
            compliance-flag, other
Urgency: low, medium, high

Output schema:
{"category": "...", "urgency": "...", "reason": ""}

Example:
Ticket: "Aadhaar upload keeps failing, verification stuck 3 days"
Output: {"category": "kyc-verification", "urgency": "medium",
         "reason": "identity document upload failure blocking onboarding"}

Example:
Ticket: "Your app flagged my transfer to HMRC as suspicious, why?"
Output: {"category": "compliance-flag", "urgency": "high",
         "reason": "customer disputing an automated compliance hold"}

Ticket: {ticket_text}
Output:

Re-run against the same 40-example set, v2 scored 89% exact-match category accuracy with fully consistent, parseable urgency output — a jump attributable to three specific, falsifiable fixes the critique identified: a closed category set, a fixed output schema, and two targeted examples for the previously weakest categories. Note what did not change: the critique did not ask for a longer prompt, a friendlier tone, or extra caveats — it asked for constraints the original prompt was missing. That is the shape a useful critique should take, and a useful mental check on any critique that instead asks you to add words rather than remove ambiguity.

Putting it into practice

If you are starting from nothing, the pragmatic sequence is: pick one prompt that is underperforming and matters enough to be worth the investment, label thirty to fifty representative examples covering its known edge cases, write a scoring function that reduces each output to a single comparable number, then run two or three rounds of the critique-rewrite loop by hand before you consider automating any part of it. Reading the critiques yourself for the first few rounds tells you whether the critic model is finding real issues or hallucinating plausible-sounding ones — a signal you need before trusting the loop to run with less supervision.

Once the pattern is working and you trust the eval set, wrap the loop in code roughly like the sketch earlier in this guide, add the discard-on-no-improvement and patience-based stopping condition, and log every accepted rewrite with its score so you have an audit trail of what changed and why — the same discipline covered in our guide on prompt management and versioning in production. Wire the eval into CI so a prompt change that regresses the score gets caught before it ships, as described in putting evals into CI for prompt and agent regression testing. And revisit the eval set itself periodically — traffic patterns shift, and an eval set that was representative at launch drifts out of date exactly the way a hand-tuned prompt does.

Research citations: Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback" (arXiv:2303.17651); Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" (arXiv:2303.11366); Yang et al., "Large Language Models as Optimizers" (OPRO), Google DeepMind (arXiv:2309.03409); Zhou et al., "Large Language Models Are Human-Level Prompt Engineers" (APE); DSPy documentation, Stanford NLP, as published in 2026.