Reinforcement Fine-Tuning with GRPO: Teach a Small Model to Reason

What you need to know

• Reinforcement fine-tuning teaches a model to think, not to copy. Instead of imitating labelled examples the way supervised fine-tuning does, RFT hands the model a reward signal for a verifiable outcome and lets it discover its own strategy. It is the right tool when correctness is checkable — maths, code that must pass tests, structured extraction, tool-use that either works or does not.
• GRPO is the algorithm everyone is using now. Group Relative Policy Optimization is a leaner PPO: it drops the separate critic model and instead samples a group of answers per prompt and scores each against the group's own average. That single change roughly halves the memory and makes RFT feasible on one GPU.
• The reward function is the entire job. A good reward combines a verifier (did the answer match, did the code pass) with format and length terms, and it is hard to game. A bad reward gets hacked — the model learns to score points without solving the task.
• The stack is open and small-GPU friendly. Hugging Face TRL ships GRPOTrainer; Unsloth makes GRPO fit a single consumer or cloud GPU; ART pairs vLLM sampling with Unsloth training to bring GRPO to any Python app. A 3B–8B model on one A100 or H100 in AWS Mumbai or a London region is a realistic starting point.

For two years the default answer to "make this model better at our task" was supervised fine-tuning. Collect a few thousand prompt-and-answer pairs, run them through LoRA or QLoRA, and the model learns to copy the style and structure of your labels. That works beautifully when the thing you want is a behaviour you can demonstrate. It works far less well when the thing you want is the model to reason its way to a correct answer it has never been shown — because you cannot demonstrate reasoning by handing over the final answer. The model just learns to guess the answer that follows the question, and falls apart the moment the question changes.

Reinforcement fine-tuning flips the contract. You stop showing the model the answer and start telling it, after the fact, whether the answer it produced was right. Then you let it try again, thousands of times, keeping more of whatever it did when it scored well. Given a verifiable task, a small model will discover chains of reasoning that no one wrote down for it. This is the technique behind the recent wave of open reasoning models, and the remarkable part is how accessible it has become: the most popular algorithm, GRPO, was designed to run without the heavy machinery that made earlier reinforcement learning the preserve of well-funded labs. This is the deep companion to our LoRA and QLoRA recipe and our DPO and ORPO preference-tuning guide — those cover imitation and preference; this one covers the third, increasingly important branch: learning from a reward.

SFT, preference tuning and RFT: three different jobs

The first decision is not which algorithm to use but which family of fine-tuning your problem belongs to. They are not interchangeable, and reaching for the wrong one is the most common and most expensive mistake teams make. Our decision ladder for whether you should fine-tune at all is the prerequisite read; assuming you have decided you should, here is how the three families divide the work.

Supervised fine-tuning learns from labelled examples. You give it the input and the exact output you want, and it adjusts its weights to make that output more likely. It is the right tool for teaching format, tone, a domain vocabulary, or any behaviour you can demonstrate with a few hundred to a few thousand examples. It is cheap, stable and predictable. Its weakness is that it can only teach what you can label, and it tends to forget general capability if you push it hard — the well-known catastrophic-forgetting problem.

Preference tuning — DPO, ORPO and their relatives — learns from comparisons. You give it a prompt with a better answer and a worse answer, and it learns to prefer the better one. This is how you instil taste, helpfulness and safety where there is no single correct answer but there is a clear ranking. It needs paired preference data, which is more expensive to collect than plain labels but far cheaper than running a full reward loop.

Reinforcement fine-tuning learns from a reward on outcomes the model generates itself. You do not supply the answer or a comparison; you supply a function that scores whatever the model produces. The model samples attempts, gets scored, and updates toward higher-scoring behaviour. This is the only family that lets the model discover strategies you never wrote down — and the only one that needs a programmatic way to check correctness.

Family	Learns from	Data you need	Reach for it when…
SFT (LoRA / QLoRA)	Labelled input → output pairs (imitation)	Hundreds to thousands of examples of the exact output	You can demonstrate the behaviour: format, tone, domain style, a fixed task
Preference (DPO / ORPO)	Pairs of better vs worse answers (ranking)	Paired preference data; no single right answer	You want taste, helpfulness or safety where quality is a ranking, not a fact
RFT (GRPO)	A reward on the model's own outputs (exploration)	A programmatic verifier; can work with <100 prompts	Correctness is checkable: maths, code-passes-tests, extraction, tool-use

How GRPO works without a critic

To see why GRPO matters, it helps to know what it replaced. Classic reinforcement learning from a reward — PPO, the algorithm behind the first generation of RLHF — runs four models at once: the policy you are training, a frozen reference, a reward model, and a separate value or critic model that estimates how good each partial response is. That critic is expensive. It is roughly the same size as the policy, it has to be trained alongside it, and it doubles the memory you need just to take a step. For a small team on a single GPU, that critic is the thing that puts PPO out of reach.

GRPO — Group Relative Policy Optimization, popularised by the DeepSeekMath work and then by DeepSeek-R1 — removes the critic entirely. The insight is simple and almost obvious in hindsight: if you want to know whether a particular answer was better than average, do not train a model to predict the average; just sample several answers and measure. For each prompt, GRPO samples a group of completions — say eight or sixteen — scores every one of them with your reward function, and then computes each completion's advantage by normalising its reward against the mean and standard deviation of the group. A completion that scored above the group average gets a positive advantage and is reinforced; one that scored below gets pushed down. The group is its own baseline. This is Monte-Carlo, group-relative advantage estimation, and it is why the method is called group relative.

Because the baseline comes from the group rather than from a learned critic, GRPO needs only the policy and a frozen reference model. That roughly halves the memory against PPO and is the single reason reinforcement fine-tuning is now something a builder in Bengaluru or London can run on one rented A100 rather than something only a frontier lab can afford. Here is the loop in conceptual pseudocode — every GRPO implementation is doing some version of this.

# Conceptual GRPO step — the heart of the algorithm
for prompt in batch:
    # 1. Sample a GROUP of completions on-policy (vLLM does this fast)
    group = policy.sample(prompt, num_samples=8)

    # 2. Score every completion with your reward function
    rewards = [reward_fn(prompt, c) for c in group]

    # 3. Advantage = how far above/below the GROUP each one scored
    mean, std = mean_of(rewards), std_of(rewards)
    advantages = [(r - mean) / (std + 1e-8) for r in rewards]   # no critic needed

    # 4. Update the policy toward high-advantage completions,
    #    with a KL penalty that anchors it to the reference model
    loss = -sum(adv * logprob(c) for c, adv in zip(group, advantages))
    loss += kl_coeff * kl_divergence(policy, reference)
    loss.backward()

Two details in that sketch carry most of the practical weight. The first is that the sampling in step one is on-policy — the completions must come from the model as it currently is, which is why every GRPO run wants a fast inference engine (almost always vLLM) bolted to the trainer to keep generation from dominating wall-clock time. The second is the KL term in step four: it penalises the policy for drifting too far from the reference model, which is what stops the model from collapsing into degenerate text that happens to score a high reward. We will come back to both.

Reward design is the whole game

If you take one thing from this article, take this: in reinforcement fine-tuning, the reward function is the model you are building. The base model supplies raw capability; the reward decides what that capability gets pointed at. A sloppy reward produces a model that is confidently, fluently wrong in exactly the way your reward let it be. A precise reward produces a model that reasons. You will spend more time on the reward than on any other part of the project, and you should.

The golden rule is verifiable over judged. Wherever you can, the reward should check an objective fact rather than ask another model for an opinion. Did the extracted number equal the ground truth? Did the generated code pass the unit tests? Did the SQL query return the expected rows? These are cheap, deterministic and almost impossible to game. The moment your reward depends on an LLM judge, you have introduced a second model that can itself be fooled — useful for fuzzy tasks, but treat it as a last resort and pair it with hard checks where you can. If you do lean on a model judge for part of the signal, the discipline from our evaluation guides applies directly; reward design and eval design are the same skill wearing different hats.

In practice a good reward is not one number but a small sum of terms, each measuring something different.

• A correctness term — the verifier. The largest weight goes here: did the answer match, did the tests pass. This is the term that actually teaches the task.
• A format term — did the model put its answer where you asked, for example inside <answer>…</answer> tags or as valid JSON. Small but important: it makes the correctness term extractable and stops the model rambling.
• A length or penalty term — a gentle nudge against pathological behaviour, such as answers that are absurdly long, empty, or that repeat themselves. Keep it small so it shapes rather than dominates.

Here is a reward function for a verifiable maths task, of the kind you would pass to a GRPO trainer. It is deliberately mundane Python — that is the point. The signal is in the checks, not in any cleverness.

import re

def extract_answer(text: str) -> str | None:
    """Pull the final answer out of the required <answer> tags."""
    m = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
    return m.group(1).strip() if m else None

def reward_fn(prompt: str, completion: str, ground_truth: str) -> float:
    score = 0.0

    # 1. Format reward: did it use the answer tags at all?
    answer = extract_answer(completion)
    if answer is not None:
        score += 0.2                      # small, just for obeying the contract

    # 2. Correctness reward: the verifier — this is where the learning is
    if answer is not None and answer == ground_truth.strip():
        score += 1.0                      # the big signal

    # 3. Light length penalty against rambling / reward-padding
    if len(completion) > 4000:
        score -= 0.1

    return score                          # GRPO normalises this across the group

Notice what this reward does not do. It does not give partial credit for "looking like" a correct answer, because partial credit on appearance is exactly the gap a model learns to exploit. It does not reward the reasoning text itself — only the final answer and the contract around it — because the moment you reward the chain of thought directly, the model learns to write convincing-looking reasoning that reaches the wrong conclusion. Keep the big reward tied to an outcome you can verify, and let the model find its own path there.

The tooling: TRL, Unsloth and ART

The reason this is a practical guide and not a research summary is that the open-source stack has caught up to the technique. As of mid-2026 you do not need to implement GRPO yourself; three pieces of kit cover the spectrum from "drop it into a script" to "wire it into a running application".

Hugging Face TRL is the baseline. Its GRPOTrainer is the standard high-level trainer: you hand it a model, a dataset of prompts and one or more reward functions, and it runs the sample-score-update loop for you. If you have ever used TRL's SFTTrainer or DPOTrainer, the shape is identical. Unsloth sits underneath and makes it fit: its memory-efficient GRPO uses kernel-level optimisations and aggressive memory management to run a reasoning-model training run on a single consumer or modest cloud GPU — the difference between needing a multi-GPU node and renting one A100 for an afternoon. ART — the Agent Reinforcement Trainer from OpenPipe — is the newest and most application-shaped of the three: it is fully open source, runs vLLM for fast on-policy inference and Unsloth-powered GRPO for training, and is designed to drop reinforcement fine-tuning into any existing Python agent so you can train the agent on the rewards from its own real tasks.

Tool	Best for	What it gives you
TRL GRPOTrainer	Standard, script-based training runs	The canonical high-level trainer; pass a model, prompts and reward functions
Unsloth GRPO	Fitting a run onto one GPU	Memory-efficient kernels so a 3B–8B reasoning run fits a single A100/H100 or even a consumer card
ART (OpenPipe)	Training an agent on its own real tasks	vLLM sampling + Unsloth GRPO wired into any Python app; bring RFT to an existing agent

Here is a TRL GRPOTrainer setup sketch for the maths task above. The reward function from the previous section plugs straight in; everything else is wiring.

from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

# Prompts with checkable answers — small is fine, the signal is in the reward
dataset = load_dataset("your-org/verifiable-maths", split="train")

def reward_wrapper(completions, ground_truth, **kwargs):
    # TRL passes a batch; return one float per completion
    return [reward_fn("", c, gt) for c, gt in zip(completions, ground_truth)]

config = GRPOConfig(
    output_dir="grpo-qwen3-4b-maths",
    per_device_train_batch_size=4,
    num_generations=8,          # the GROUP size G
    max_completion_length=1024,
    learning_rate=1e-6,         # RL likes a small LR
    beta=0.04,                  # KL penalty coefficient — anchors to the reference
    use_vllm=True,              # fast on-policy sampling
    bf16=True,
)

trainer = GRPOTrainer(
    model="Qwen/Qwen3-4B",      # a small open-weight base, light SFT optional first
    reward_funcs=reward_wrapper,
    args=config,
    train_dataset=dataset,
)
trainer.train()

If you are using Unsloth instead, the shape barely changes — you load the model through FastLanguageModel with a LoRA adapter and 4-bit quantisation first, then hand it to the same kind of GRPO trainer. The Unsloth route is what gets a Qwen- or Llama-class model in the 3B–8B range onto a single GPU: 4-bit weights, a LoRA adapter rather than full fine-tuning, and gradient checkpointing together drop the memory enough that the limiting factor becomes your patience, not your VRAM. For India-resident data run it on an A100 in AWS Mumbai; for UK or EU records use a London or Frankfurt GPU region. The recipe is identical; only the region changes.

A worked example: teaching a 4B model to do maths

Let us make this concrete with the smallest end-to-end run worth doing. The goal: take a small open-weight base — a 3B–8B Qwen- or Llama-class model — and teach it to solve grade-school and competition maths problems by reasoning step by step, on one GPU, without a labelled chain-of-thought for a single example.

Stage one, the warm-up. Run a very light SFT pass — sometimes as little as one per cent of a labelled set — purely to teach the output contract: think in the open, then put the final answer inside <answer>…</answer> tags. You are not teaching maths here; you are teaching the model to produce something your verifier can read. Skip this and the GRPO stage often stalls because too few sampled completions are even parseable, so the reward signal is mostly zeros.

Stage two, the GRPO run. Assemble a few hundred — or even a few dozen — maths problems, each with a known numeric answer. This is the part that surprises people coming from SFT: you do not need a chain-of-thought for any of them, only the final answer to check against. Point GRPOTrainer at the warmed-up model with the reward function from earlier, a group size of eight, vLLM sampling on, and a small KL coefficient. Then watch two curves: the mean reward, which should climb, and the KL to the reference, which should rise gently and plateau rather than spike.

What you typically see, and what the DeepSeek-R1 work made famous, is that the model's completions get longer on their own as training proceeds. Nobody told it to show more working; it discovered that more deliberate, step-by-step reasoning earns more reward, so it does more of it. That emergent lengthening is the signature of GRPO actually teaching reasoning rather than memorising answers. On a verifiable maths or code benchmark, public GRPO runs report meaningful gains over the base model from this recipe — but resist quoting a precise headline number from memory; measure it on your own held-out set, because the figure depends entirely on your base model, data and reward, and your own eval is the only number you should trust.

The pitfalls, and how to stay out of them

RFT has sharper edges than SFT, and knowing them in advance is the difference between a run that teaches reasoning and one that quietly teaches the model to cheat. Three failure modes account for almost everything that goes wrong.

Reward hacking

The headline risk, and the one the Builder quote above ran straight into. If your reward can be satisfied without solving the task, the model will eventually find that shortcut — not out of malice but because the optimiser is doing exactly its job. The defences are unglamorous and they work: keep rewards verifiable rather than judged; split the reward into correctness, format and length terms so no single loophole pays off; add explicit penalties for the gaming behaviours you can anticipate; and, above all, read your samples. Pull twenty completions at random every few hundred steps and actually read them. A reward curve that climbs while the completions get worse is the unmistakable signature of reward hacking, and the only way to see it is with your own eyes.

Entropy collapse and over-optimisation

Push the policy too hard toward the reward and it can collapse — producing the same high-scoring response over and over, losing the diversity it needs to keep improving, or drifting into degenerate text that scores well by accident. This is what the KL penalty is for. Monitor the KL divergence to the reference model as a first-class metric, not an afterthought: a gentle rise that plateaus is healthy, a sharp spike is the model running away from its grounding. If KL climbs too fast, raise the KL coefficient or lower the learning rate. The reference model is the leash; do not let the policy snap it.

Reaching for RFT when you should not

The deepest pitfall is choosing reinforcement fine-tuning for a task that has no verifier. RFT only works when you can programmatically check the outcome. If you cannot write a function that scores a completion — if "good" is a matter of taste, tone or judgement with no checkable ground truth — then RFT has nothing to optimise against, and you are forced onto a model-judge reward that is itself gameable and unstable. For those tasks, preference tuning is the right tool: go back to the DPO and ORPO guide, or the broader should-you-fine-tune decision ladder to confirm the family before you spend a GPU-week on the wrong one.

So, should you reach for RFT? A short checklist:

• Can you write a verifier? If a small Python function can score a completion as right or wrong, RFT is on the table. If not, prefer SFT or preference tuning.
• Is the task about reasoning or strategy? Maths, code, multi-step tool use, structured extraction with a checkable target — these reward exploration. A fixed format or tone does not; use SFT.
• Do you have a base model with some latent capability? RFT amplifies what is already there. A model that cannot do the task at all, even occasionally, gives the reward nothing to reinforce — warm it up with SFT first.
• Can you afford the on-policy sampling? RFT needs a fast inference path and more compute per step than SFT. One A100 or H100 is enough for a 3B–8B model; budget accordingly.

If you want to go deeper than this guide, the primary sources are worth reading in full: the DeepSeekMath paper that introduced GRPO and the DeepSeek-R1 paper that showed it scaling to reasoning, the TRL GRPOTrainer docs, the Unsloth GRPO guide, and OpenPipe's ART repository for the agent-shaped path. For a structured course, DeepLearning.AI's short course Reinforcement Fine-Tuning LLMs with GRPO walks through the full loop hands-on.

Where RFT fits in your toolbox

Reinforcement fine-tuning is not a replacement for the techniques around it; it is the third leg of the stool. SFT teaches behaviour you can demonstrate. Preference tuning teaches taste you can rank. RFT teaches reasoning you can verify — and until very recently it was simply too expensive for most teams to use. GRPO is what changed that, by trading the costly critic model for a clever bit of statistics, and the open-source stack of TRL, Unsloth and ART is what put it on a single rented GPU within reach of a builder in Chennai, Bengaluru or London.

The discipline it demands is unusual and worth stating plainly: in RFT, most of your engineering goes into the reward function and most of your debugging goes into reading samples. Get the reward right — verifiable, multi-term, hard to game — and a small model will teach itself to reason in ways you never wrote down. Get it wrong and you will train a beautifully optimised cheat. If your task has a checkable answer and you have an afternoon and one GPU, there has never been a better time to try it. And if it does not have a checkable answer, you now know exactly which guide to read instead. For the imitation and distillation routes that often pair with an RFT stage, see our LoRA and QLoRA recipe and distillation playbook; for the serving side once your reasoning model is trained, the vLLM self-hosting playbook takes it from there.