LoRA and QLoRA from scratch: an eval-driven fine-tuning recipe you can reuse

What this recipe gives you

Fine-tuning has a reputation as a dark art reserved for teams with eight-figure GPU budgets. It is not. With LoRA and QLoRA you can adapt a 7B or 8B open-weight model on a single consumer-grade card, in an afternoon, for the price of a couple of cups of coffee. The hard part was never the compute. The hard part is knowing whether you should fine-tune at all, building a dataset that matches how you will actually call the model, and proving the result is better rather than merely different.

This is a recipe you can keep and reuse across models and tasks. The methods here are stable: low-rank adaptation, 4-bit quantisation, an 80/10/10 split, an eval set built first. The specific model you point it at — Llama, Qwen, Mistral, Gemma — barely changes the steps. Here is the short version before we go deep:

• Fine-tune for behaviour, not facts. If the model gets the format, tone, schema or policy wrong, tuning helps. If it is missing knowledge, RAG helps. The 2026 default is hybrid.
• QLoRA is the budget default. Quantise the frozen base to 4-bit, train low-rank adapters in higher precision on top. A 7B model fits comfortably on a 24GB GPU.
• Prompt-format parity is non-negotiable. Train on the exact prompt template you will use at inference, or your gains evaporate in production.
• Build the eval set before you train. No frozen golden set means no honest way to tell improvement from noise.

Prerequisites

You do not need a cluster. You need a clear task, a little data, and a GPU you can rent by the hour. Before you start, have the following in place:

• A defined task with a measurable output. "Classify support tickets into one of nine queues" or "rewrite a product description in our house style" — something you can grade. Open-ended chat is the hardest thing to fine-tune well; start narrower.
• A base model. An instruction-tuned open-weight model in the 3B–8B range is the sweet spot for a first adapter. Smaller iterates faster; larger forgives messier data.
• A GPU. A single 24GB card is enough for QLoRA on any 7B or 13B model. As of mid-2026, a single-A10G instance (24GB, the AWS g5.xlarge class) runs around US$1.00/hr on-demand in us-east-1, with a modest premium in the AWS Mumbai (ap-south-1) and London (eu-west-2) regions. India-based Builders can also apply for IndiaAI Mission subsidised compute, advertised in the sub-₹150/hr band — cheaper than commercial cloud for sustained runs.
• Python with the PEFT stack. transformers, peft, bitsandbytes, datasets, accelerate and trl. Pin your versions; the fine-tuning stack moves fast.

Step 1 — Decide: fine-tune, RAG, or prompt?

This is the most expensive decision you will make, so make it deliberately. The cleanest framing in 2026 is this: fine-tuning teaches form, retrieval supplies facts. Diagnose your failure mode first.

• The model is missing or repeating stale information. That is a knowledge problem. Reach for RAG so the facts stay current and traceable, or simply put the facts in the prompt. Fine-tuning here is the wrong tool — you would be baking today's facts into weights that go stale tomorrow.
• The model knows the answer but presents it wrong. Wrong output format, inconsistent tone, weak classification, poor adherence to your policy or schema — that is a behaviour problem, and that is what fine-tuning fixes well.
• Both. Then do both. The pragmatic production pattern is a thin LoRA or QLoRA adapter for behaviour and output shape, paired with retrieval for knowledge at inference time. The two solve different problems and compose cleanly.

A useful sequence to climb in order is prompt → RAG → fine-tune → distil. Exhaust the cheaper rungs before paying for the dearer ones. If you want the longer version of this decision, we wrote a dedicated decision ladder for whether to fine-tune at all.

Step 2 — LoRA vs QLoRA vs full fine-tuning

Full fine-tuning updates every weight in the model. It works, but it is expensive: fully fine-tuning a 7B model in 16-bit needs roughly 100–120GB of VRAM once you account for optimiser states and gradients — multiple H100-class GPUs for a single run. For most teams that is overkill for what is, in effect, a styling job.

LoRA (Low-Rank Adaptation) freezes the base weights entirely and inserts small trainable matrices — the adapters — alongside the layers you target. You train a fraction of a percent of the parameters. The base stays in 16-bit, so memory is dominated by holding the base model in VRAM.

QLoRA goes one step further: it quantises the frozen base to 4-bit using the NF4 (4-bit NormalFloat) data type before training, while the adapters themselves train in higher precision. Quantising the base cuts its memory footprint by roughly 75% versus 16-bit, which is what lets a 7B model — needing only around 12GB with QLoRA — fit on a 24GB consumer card with headroom to spare. Double quantisation shrinks the quantisation constants further, and paged optimisers absorb the memory spikes during gradient checkpointing.

Approach	VRAM (7B model)	Trainable params	Quality	When to choose
Full fine-tuning	~100–120GB	100%	Ceiling	Large domain shift, ample GPU budget
LoRA (16-bit base)	~16–20GB	<1%	Near-full on most tasks	You have a 24GB+ card and want best adapter quality
QLoRA (4-bit base)	~12GB	<1%	Within a hair of LoRA on most tasks	Budget default; single 24GB consumer GPU

Independent comparisons through 2025–2026 consistently report QLoRA reaching roughly 80–90% of full fine-tuning quality, and in many task-specific cases matching standard LoRA and full fine-tuning closely enough that the gap is negligible. The headline trade-off: QLoRA uses meaningfully less VRAM than LoRA — base-weight memory drops about 75% — for a small and often unnoticeable quality cost. For a first adapter, start with QLoRA. You can always rerun in LoRA if your eval set says the 4-bit base is holding you back.

Step 3 — Choose rank, alpha and target modules

Three hyperparameters define a LoRA configuration, and beginners over-think all three.

• Rank (r) is the capacity of the adapter. Ranks of 8–64 cover most tasks. Too low underfits; too high wastes the parameter savings and can overfit a small dataset. Start at 16.
• Alpha (lora_alpha) scales the adapter's contribution; the effective scale is alpha / r. The widely used heuristic is alpha = 2 × rank, so alpha 32 for rank 16. Keep alpha / r at least 1.
• Target modules decide which layers get adapters. Attention-only (q_proj, k_proj, v_proj, o_proj) is the lean option. Including the MLP projections (gate_proj, up_proj, down_proj) gives broader coverage and usually better quality for a bit more memory.

Here is a sane QLoRA configuration with bitsandbytes and PEFT. This loads the base in 4-bit and attaches the adapters:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

MODEL = "meta-llama/Llama-3.1-8B-Instruct"  # any instruct base works

# 4-bit NF4 quantisation with double quant — the QLoRA recipe
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL)
tokenizer.pad_token = tokenizer.eos_token  # most causal LMs lack a pad token

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,            # alpha = 2 * r
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # expect <1% trainable

Step 4 — Build your eval set BEFORE you tune

This is the step that separates a reusable recipe from a one-off experiment, and it is the step most teams skip. If you do not have a frozen benchmark, you cannot distinguish a real improvement from random noise, and you will be tempted to declare success on vibes.

Build a golden set: a representative collection of inputs paired with either expected outputs or a scoring rubric. Cover your common cases and — crucially — your known failure cases. Then score the base model on it. That number is your baseline. You only ship an adapter that beats the baseline on held-out data.

import json

# golden_eval.jsonl — built by hand or curated from production logs, BEFORE training
# one record per line: {"input": ..., "expected": ...}

def load_golden(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def exact_match(pred, expected):
    return pred.strip().lower() == expected.strip().lower()

def score_model(generate_fn, golden):
    """generate_fn(input_text) -> model output string"""
    hits = 0
    for row in golden:
        pred = generate_fn(row["input"])
        if exact_match(pred, row["expected"]):
            hits += 1
    return hits / len(golden)

golden = load_golden("golden_eval.jsonl")
baseline = score_model(base_generate, golden)   # the number to beat
print(f"Base model score: {baseline:.3f}")

Exact match suits classification and structured extraction. For open-ended generation, swap in a rubric scored by an LLM judge, or a task metric such as ROUGE for summarisation. The mechanics of golden sets and judges deserve their own treatment — see our guide to building an LLM evaluation suite. The non-negotiable principle is the same regardless of metric: the eval set is frozen before training, and the test portion is never looked at until the very end.

Step 5 — Construct the dataset and split 80/10/10

Quality and consistency beat volume every time. A few hundred clean, consistent examples can shift behaviour; 1,000–10,000 is a comfortable range for a task adapter. Ten thousand sloppy examples will underperform five hundred clean ones.

The detail that quietly ruins most first attempts is prompt-format parity: the template you train on must match the template you call at inference, token for token. If you train on a bare instruction but serve through the model's chat template, the adapter has learned a distribution you never use in production, and the gains vanish.

from datasets import load_dataset

# Format every example with the EXACT chat template you will use at inference.
def to_chat(example):
    messages = [
        {"role": "system", "content": "You are a support-ticket classifier."},
        {"role": "user",   "content": example["input"]},
        {"role": "assistant", "content": example["expected"]},
    ]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    return {"text": text}

ds = load_dataset("json", data_files="data.jsonl", split="train")
ds = ds.map(to_chat)

# 80 / 10 / 10 — train / validation / test
ds = ds.shuffle(seed=42)
n = len(ds)
train = ds.select(range(0, int(0.8 * n)))
val   = ds.select(range(int(0.8 * n), int(0.9 * n)))
test  = ds.select(range(int(0.9 * n), n))   # do not touch until the end

Step 6 — Train: watch validation loss, stop early

The training loop itself is the easy part. With trl's SFTTrainer it is a few lines. The discipline is in what you watch.

from trl import SFTTrainer, SFTConfig

args = SFTConfig(
    output_dir="adapter-out",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # simulate a larger batch on a small GPU
    learning_rate=2e-4,              # higher than full FT; LoRA likes 1e-4 to 3e-4
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    load_best_model_at_end=True,     # keep the checkpoint with lowest val loss
    metric_for_best_model="eval_loss",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=train,
    eval_dataset=val,
    dataset_text_field="text",
)
trainer.train()
trainer.save_model("adapter-out")    # saves ONLY the adapter weights (tiny)

Three things matter while this runs:

• Learning rate. LoRA tolerates and prefers a higher rate than full fine-tuning — 1e-4 to 3e-4 is the usual band, with 2e-4 a good first guess. Too high and the adapter thrashes; too low and it barely learns.
• Validation loss. Watch the gap between training loss and validation loss. Training loss falling while validation loss rises is the textbook sign of overfitting — common with small datasets and high rank.
• Early stopping. Set load_best_model_at_end and keep the checkpoint with the lowest validation loss. More epochs is not more better; for many task adapters the best checkpoint lands well before the final epoch.

When training finishes, re-run the Step 4 eval — on the adapter this time — and compare against the baseline. If it does not beat the base model on held-out data, you have learned something useful and cheap: this task did not need fine-tuning, or your data needs work. That is a result, not a failure.

Step 7 — Merge, serve and avoid the pitfalls

An adapter is tiny — often a few tens of megabytes — and you have two ways to serve it. You can load the base model and apply the adapter at runtime, which lets you hot-swap multiple adapters on one served base. Or you can merge the adapter into the base weights for a single standalone model, which removes any per-request adapter overhead.

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Merge for serving: load base in FULL precision, attach adapter, merge, save.
# Note: merge into a 16-bit base, not the 4-bit quantised one you trained against.
base = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "adapter-out")
merged = merged.merge_and_unload()
merged.save_pretrained("merged-model")
tokenizer.save_pretrained("merged-model")
# Serve merged-model with vLLM, TGI or your engine of choice.

The pitfalls that catch teams, in rough order of frequency:

1. Train/inference template mismatch. Already flagged, worth repeating — it is the number one cause of "it worked in the notebook, not in prod".
2. No frozen eval set. If you cannot quote a baseline number and a post-tune number on the same held-out data, you do not know whether you improved anything.
3. Merging into the wrong precision. Train against a 4-bit base for memory, but merge into a 16-bit base for serving quality. Merging adapters into a quantised base degrades the result.
4. Over-training. Three epochs is often plenty. Watch validation loss and stop at the best checkpoint rather than the last one.
5. Catastrophic forgetting. Narrow data can erode general ability. If the model must stay broadly capable, keep your adapter rank modest and mix in a little general-purpose data.

So — what should you actually do?

If you take one thing from this recipe, make it the order of operations. Diagnose the failure mode first; only fine-tune if it is a behaviour problem. Build the eval set and capture a baseline before you touch a GPU. Default to QLoRA, rank 16, alpha 32, attention plus MLP targets. Train on the exact prompt template you will serve. Watch validation loss, stop early, merge into 16-bit, and only ship if you beat the baseline on held-out data.

Every step here is method-stable and model-agnostic. Swap Llama for Qwen or Mistral and the recipe is unchanged. That is the point of writing it down once and reusing it — and if your real bottleneck turns out to be serving cost rather than behaviour, a distillation pass or a sharper retrieval setup may serve you better than another adapter.

Primary references: the Hugging Face PEFT documentation, the original QLoRA paper (Dettmers et al.), and the LoRA paper (Hu et al.).