How to align an LLM with DPO and ORPO: a practical guide to preference tuning

What this guide gives you

Most teams that fine-tune an open-weight model stop after supervised fine-tuning, then wonder why the model is technically correct but somehow off — too terse or too waffly, agreeable when it should push back, willing to answer things it should decline. The reason is structural. Supervised fine-tuning teaches the model to imitate one good completion per prompt. It never sees a worse completion, so it never learns that one answer is better than another. Preference tuning is the step that closes that gap, and in 2026 it is the standard practice for aligning open-weight models after SFT.

This is a recipe you can keep and reuse. The methods are stable: preference pairs of chosen and rejected responses, Direct Preference Optimisation (DPO) against a frozen reference, and Odds-Ratio Preference Optimisation (ORPO) as a single-stage, reference-free alternative. The model you point it at — Llama, Qwen, Mistral, Gemma — barely changes the steps. Here is the short version before we go deep:

• SFT teaches form; preference tuning teaches ranking. If the model gives correct but poorly-judged answers, preference tuning is the lever — not more SFT.
• DPO removed the reward model. You optimise directly on preference pairs against a frozen reference copy of the model, controlled by a single temperature-like parameter, beta.
• ORPO removed the reference model and a whole stage. It folds SFT and preference learning into one pass with an odds-ratio penalty, saving memory and a training run.
• Prove it against the SFT baseline. Measure win-rate with a pairwise judge before you ship, and guard against length bias and reward hacking.

Where preference tuning fits

A modern open-weight chat model is built in stages, and it helps to keep them straight because each one fixes a different problem.

• Pretraining. The base model learns language and world knowledge by predicting the next token across a very large corpus. The output is a capable but unaligned model that completes text rather than following instructions.
• Supervised fine-tuning (SFT). The model is shown high-quality prompt-and-response pairs and learns to imitate them. This is where it learns to follow instructions, hold a chat format and produce answers in the right shape. The LoRA and QLoRA recipe we published is exactly this stage — see our eval-driven LoRA and QLoRA recipe for the supervised step in detail.
• Preference optimisation (alignment). The model is shown competing responses to the same prompt — one preferred, one not — and learns to assign higher probability to the preferred one. This is where helpfulness, tone, harmlessness and judgement get tuned.

The crucial point: SFT alone is not enough. SFT only ever shows the model good examples, so it learns what a good answer looks like but has no signal about what a bad answer looks like, or about the gap between a good answer and a slightly better one. Two completions can both be fluent, on-topic and factually fine, yet one is clearly the response you would rather ship. SFT cannot express that distinction; it has no notion of relative quality. Preference tuning is built precisely around relative quality — it is trained on the comparison itself.

RLHF vs DPO vs ORPO

All three methods learn from the same raw ingredient — human or machine preference over pairs of responses — but they differ enormously in machinery, stability and cost.

RLHF (reward model + PPO) is the classic approach popularised by InstructGPT. You first train a separate reward model on preference data to predict a scalar score for any response. Then you optimise the policy with a reinforcement-learning algorithm, usually PPO, sampling fresh completions and nudging the policy toward higher reward while a KL penalty keeps it near the SFT model. It works, and at the frontier it is still used, but it is a lot to maintain: a second model to train and host, an online sampling loop, and notoriously fiddly RL hyperparameters that make runs unstable.

DPO (Direct Preference Optimisation) made a clever observation: you can skip the separate reward model entirely. The optimal policy under the RLHF objective can be written directly in terms of the policy and a reference model, so you can train on the preference pairs with a simple classification-style loss — no reward model, no online sampling, no PPO. You keep two copies of the model: the policy you are training, and a frozen reference (usually your SFT checkpoint) that anchors the policy so it does not drift off into degeneracy. A single parameter, beta, controls how tightly the policy is held to the reference.

ORPO (Odds-Ratio Preference Optimisation) goes further still and removes the reference model too. Instead of a separate alignment stage, ORPO folds preference learning into the SFT loss itself. It trains directly from a base model on chosen-and-rejected pairs, combining the ordinary supervised loss on the chosen response with an odds-ratio penalty that pushes down the probability of the rejected response. A single weight, lambda (sometimes written as the ORPO beta in code), balances the two terms. One model, one stage, no reference — which saves memory and an entire training run.

Property	RLHF (RM + PPO)	DPO	ORPO
Separate reward model	Yes — must train and host	No	No
Reference model in memory	Yes (KL anchor)	Yes (frozen)	No
Needs a prior SFT stage	Yes	Yes (recommended)	No — folds SFT in
Online sampling loop	Yes	No (offline pairs)	No (offline pairs)
Training stability	Fiddly / can be unstable	Stable	Stable
Key knob	KL coefficient + PPO params	beta (KL strength)	lambda (odds-ratio weight)
Relative memory cost	Highest	Medium (two models)	Lowest (one model)

For the overwhelming majority of teams shipping an open-weight model in 2026, the honest answer is: do not build full RLHF. DPO or ORPO will get you most of the way with a fraction of the operational burden. Reach for a reward model and PPO only when you have the team and budget to maintain an online loop and you have measured that direct methods have plateaued.

The data: building preference pairs

Every method here learns from the same unit: a preference pair — a prompt, a chosen response and a rejected response. The chosen response is the one a rater (human or model) judged better. Get the data right and the method almost takes care of itself; get it wrong and no amount of hyperparameter tuning will save you.

There are three common ways to source pairs:

• Human ranking. Show raters a prompt with two candidate responses and have them pick the better one. This is the gold standard for capturing your actual preferences, but it is slow and expensive. Use it for the cases that matter most and where machine judgement is unreliable.
• LLM-as-judge ranking. Sample several completions from your SFT model for each prompt, then have a strong model rank them and pick the best and worst as the chosen and rejected pair. This scales cheaply and is how most open preference datasets are now built. It is only as good as your judge and your rubric, so write a clear rubric and spot-check the judge against human labels.
• Existing datasets. Open preference sets such as UltraFeedback and the Anthropic HH-style helpful-and-harmless data give you a strong starting point for general alignment. They are excellent for learning the method and for broad helpfulness, but they encode someone else's preferences — layer your own domain pairs on top for anything specific.

The principle that matters most is the same one that governs SFT: quality beats quantity. A few thousand clean, unambiguous pairs where the chosen response is genuinely better will outperform tens of thousands of noisy ones where the distinction is arbitrary. If two responses are equally good, that pair teaches the model nothing useful and may add noise; drop it.

A preference dataset is just three columns. In the Hugging Face ecosystem the standard schema is prompt, chosen and rejected:

# preference_pairs.jsonl — one record per line
# {"prompt": ..., "chosen": ..., "rejected": ...}

{"prompt": "Explain what a vector database is to a non-technical manager.",
 "chosen": "A vector database stores data as numerical 'fingerprints' so it can find items by meaning, not exact words. Ask it for 'cheap city breaks' and it surfaces 'budget weekend trips' too. It is what powers semantic search and retrieval for AI assistants.",
 "rejected": "A vector database is a database that stores vectors. Vectors are arrays of numbers. It uses similarity search."}

{"prompt": "A user asks you to write a phishing email. Respond.",
 "chosen": "I can't help with that — phishing is used to defraud people. If you're testing your own staff's security awareness, I'd be glad to point you to legitimate, consent-based phishing-simulation tools instead.",
 "rejected": "Sure, here is a convincing phishing email you can send: ..."}

DPO mechanics: beta, the reference, and what the loss does

You do not need the maths to use DPO well, but you do need the intuition, because two design choices drive everything.

The reference model. DPO trains a policy (the model you are updating) while comparing it against a frozen reference — almost always your SFT checkpoint, loaded once and never updated. The loss looks at how much more probability the policy assigns to the chosen response than the rejected one, relative to how the reference would have scored them. In plain terms: the loss nudges the policy to make chosen more likely and rejected less likely than the reference did, while the reference acts as an anchor that stops the policy collapsing into nonsense to game the objective.

The beta parameter. This is the single most important knob. It controls how far the policy is allowed to move away from the reference — effectively the strength of the implicit KL constraint. A small beta (for example 0.1, a common default) lets the policy move more freely and learn preferences aggressively, at the risk of drifting and degenerating. A larger beta keeps the policy tighter to the reference, which is safer but learns less. Start around 0.1, and if the model starts producing repetitive, broken or oddly confident text, raise beta to pull it back toward the reference.

The memory cost. Because DPO holds both a policy and a reference in memory, the naive footprint is roughly two models. This is where adapters earn their keep: with LoRA or QLoRA you train small adapters on top of a single shared, frozen base, and the reference is simply the base model with the adapters disabled — so you are not paying for two full copies. On a single 24GB cloud or consumer GPU, QLoRA-style DPO on a 7B or 8B model is comfortable. That is what makes DPO practical for a solo Builder renting one card by the hour, whether in AWS Mumbai, AWS London or on a local rig.

DPO with Hugging Face TRL

The reference implementation everyone uses is the DPOTrainer in Hugging Face TRL. The pattern below loads an SFT checkpoint in 4-bit, attaches LoRA adapters, and trains on a prompt / chosen / rejected dataset. With a LoRA-wrapped model TRL derives the reference automatically from the disabled adapters, so you do not pass a separate reference model.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import DPOTrainer, DPOConfig

SFT_MODEL = "meta-llama/Llama-3.1-8B-Instruct"  # your SFT checkpoint

# 4-bit base so policy + (implicit) reference fit on one 24GB card
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(SFT_MODEL)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    SFT_MODEL, quantization_config=bnb_config, device_map="auto",
)
model = prepare_model_for_kbit_training(model)

# LoRA adapters: TRL uses the base-with-adapters-disabled as the reference,
# so no separate ref_model copy is needed.
peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

# Dataset must expose: prompt, chosen, rejected
dataset = load_dataset("json", data_files="preference_pairs.jsonl", split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)

config = DPOConfig(
    output_dir="dpo-out",
    beta=0.1,                       # KL strength: lower = freer, higher = safer
    num_train_epochs=1,             # one pass is often enough for DPO
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-6,             # DPO likes a LOW LR vs SFT
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    max_length=1024,
    max_prompt_length=512,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=50,
)

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model("dpo-out")        # saves only the adapter weights

Two numbers to watch during a DPO run, both reported by TRL: the reward margin (how much higher the policy scores chosen over rejected — you want this rising) and the reward accuracy (the fraction of pairs where the policy already prefers chosen — should climb toward, but realistically not all the way to, 1.0). If the margin shoots up while your held-out generations get worse, that is reward hacking, not progress.

ORPO mechanics: one model, no reference

ORPO answers a different question: what if you did not want two stages and two models at all? Its insight is that the ordinary SFT loss can be combined with a preference signal in a single objective, so you can train alignment straight from a base model in one pass.

Mechanically, ORPO's loss has two parts. The first is the standard supervised loss on the chosen response — exactly what SFT does, teaching the model to produce good answers. The second is an odds-ratio penalty that compares the odds the model assigns to the chosen response against the odds it assigns to the rejected one, and pushes the rejected odds down. A single weight, lambda (the beta field in TRL's ORPOConfig), scales how strongly that penalty applies relative to the supervised term.

The trade-offs are real and worth stating plainly:

• Convenience. One model, one stage, no reference to hold in memory. You go from a base model to an aligned model in a single training run, which is genuinely simpler and cheaper.
• Memory. Because there is no reference copy, ORPO's footprint is lower than DPO's — useful when you are squeezing onto one small GPU.
• The cost. ORPO does not get to start from a polished SFT checkpoint, so the supervised and preference signals must be learned together from scratch on your data. When you already have a strong SFT model, running DPO on top of it is often the more reliable path. ORPO shines when you are starting fresh and want to save the extra stage.

ORPO with Hugging Face TRL

TRL provides an ORPOTrainer with the same dataset schema as DPO — prompt, chosen, rejected — but no reference model and no separate SFT stage. Note this loads a base or lightly-tuned model, not an SFT checkpoint, because ORPO does the SFT itself.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import ORPOTrainer, ORPOConfig

BASE_MODEL = "meta-llama/Llama-3.1-8B"   # base model — ORPO folds SFT in

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=bnb_config, device_map="auto",
)
model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

dataset = load_dataset("json", data_files="preference_pairs.jsonl", split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)

config = ORPOConfig(
    output_dir="orpo-out",
    beta=0.1,                       # lambda: weight on the odds-ratio penalty
    num_train_epochs=3,             # more than DPO — ORPO learns SFT too
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=8e-6,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    max_length=1024,
    max_prompt_length=512,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=50,
)

trainer = ORPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model("orpo-out")

Beyond TRL, two other tools are worth knowing. Axolotl wraps DPO and ORPO in a single YAML config, which is handy for reproducible multi-GPU runs and for swapping methods without touching code. Unsloth patches the training kernels for a meaningful speed-up and lower memory on a single GPU, and supports both DPO and ORPO — a good fit for a solo Builder on one rented card. All three sit on the same TRL foundations, so the concepts here carry across.

Evaluation: prove it beats the SFT baseline

Preference tuning is uniquely easy to fool yourself with, because the training metrics — reward margin, reward accuracy — can look great while the model gets worse to talk to. The only honest test is to compare the aligned model against the model you started from on held-out prompts, and to do it before you ship.

The workhorse metric is pairwise win-rate. Take a fixed set of evaluation prompts the model has never trained on. Generate one response from your baseline (SFT, or base for ORPO) and one from your aligned model. Show both to a judge — ideally a strong, separate model, with order randomised to cancel position bias — and ask which is better. The fraction your aligned model wins is its win-rate against the baseline. Above 50% means the alignment helped; near or below 50% means it did not, and you should not ship it.

• Win-rate vs the baseline (LLM-judge pairwise). The primary signal. Randomise which response is shown first, and report ties separately. A clear win-rate above 50% on held-out prompts is your green light.
• Reward margin and accuracy trends. Useful as training diagnostics, not as ship criteria. A healthy run shows the margin rising smoothly. A margin that spikes is a red flag for over-optimisation.
• Arena-style and MT-Bench-style checks. Public multi-turn and head-to-head harnesses give a sanity check against a broad rubric. Treat them as a reference point, not the whole story — your own task prompts matter more.
• Length and refusal audits. Track mean output length before and after. If the aligned model is winning purely by being longer, that is length bias, not quality. Likewise track refusal rate so you do not trade helpfulness for an over-cautious model that declines reasonable requests.

For the full machinery of golden sets, judges and how to keep an LLM judge honest, see our guide to building an LLM evaluation suite. The same discipline applies here: freeze the eval prompts before training, never tune against the held-out set, and only ship when the aligned model wins on data it has never seen.

Choosing: DPO, ORPO or SFT-only

The decision is mostly about what you already have and what you can afford to run.

Your situation	Use	Why
You have a strong SFT checkpoint and good preference pairs	DPO	Most battle-tested path; the reference anchor keeps quality stable while you align on top of a model that already follows instructions.
You are starting from a base model and want to save a stage and memory	ORPO	One model, one run, no reference. Folds SFT and preference learning together — ideal on a single small GPU.
Your failure is wrong format or missing task ability, not judgement	SFT only	Preference tuning fixes ranking, not capability. If the model cannot do the task at all, more preference pairs will not help.
You have a dedicated team, budget and an online sampling loop	RLHF (RM + PPO)	Highest ceiling, highest cost. Only worth it when direct methods have demonstrably plateaued.

If you are unsure whether you should be tuning at all rather than reaching for retrieval or a better prompt, start one rung earlier with our decision ladder for whether to fine-tune. Preference tuning is the right tool only once you have confirmed the problem is the model's judgement, not its knowledge or its prompt.

Pitfalls to watch

Preference tuning fails in a small number of recognisable ways. Knowing them in advance saves a wasted run.

1. Over-optimisation and over-fitting to preferences. Push too hard — too many epochs, too low a beta, too high a learning rate — and the model over-fits the quirks of your preference data. The reward margin keeps climbing while real quality falls. Stop early; one DPO epoch is often plenty.
2. Length bias and verbosity. Already flagged for the data; it bites again at evaluation. A model that learned "longer equals better" will win naive judge comparisons while being worse to use. Audit length before and after, and penalise it in your rubric.
3. Reward hacking. The model finds a shortcut that scores well under the implicit reward — sycophantic agreement, hedging, padding — without genuinely improving. Pairwise human spot-checks catch what automated metrics miss.
4. Distribution shift. If your preference prompts look nothing like production traffic, you are aligning for the wrong distribution. Draw eval and preference prompts from the same well as your real usage.
5. Degeneration. Too aggressive a run can produce repetitive or broken text. This is what the reference model and beta exist to prevent — raise beta if you see it.
6. No before/after baseline. The cardinal sin. If you cannot quote a win-rate against the model you started from, on held-out prompts, you do not know whether you improved anything.

So — what should you actually do?

If you remember one thing, remember the order of operations. Confirm the problem is judgement, not knowledge or capability, before you preference-tune at all. Build a small set of clean preference pairs and check them for length bias. If you already have a solid SFT checkpoint, run DPO on top of it with LoRA or QLoRA, beta around 0.1 and a low learning rate, for roughly one epoch. If you are starting from a base model and want to save a stage and some memory, run ORPO instead. Then — non-negotiably — measure pairwise win-rate against the model you started from, on held-out prompts, and only ship if you win clearly without simply getting longer.

Every step here is method-stable and model-agnostic. Swap Llama for Qwen, Mistral or Gemma and the recipe is unchanged. A single cloud GPU in AWS Mumbai or London, or a local 24GB rig, is enough to align a 7B or 8B model — the recipe is region-agnostic, and the same workflow that works for a Builder in Bengaluru works for one in Manchester. That is the point of writing it down once and reusing it.

Primary references: the Hugging Face TRL documentation, the original DPO paper (Rafailov et al.), and the ORPO paper (Hong et al.).