Pick and Quantise a Small Model for On-Device AI: A GGUF Guide

What you need to know

Running a language model on the hardware you already own — a laptop, a single consumer GPU, an Apple Silicon Mac, or a small edge box on a factory floor — is no longer a research curiosity. It is a deployment pattern that ships. The two decisions that make or break it are which model you choose and how aggressively you quantise it. Get them right and a 7-to-8-billion-parameter model runs happily on an 8GB GPU or a 16GB Mac, with no API bill and no data leaving the device. Get them wrong and you either run out of memory before the first token or watch quality fall off a cliff on the one task you actually care about.

• GGUF Q4_K_M is the default — it runs on CPU, consumer GPUs and Apple Silicon, keeps roughly 92% of full-precision quality and cuts model size around 75% versus FP32.
• 8GB VRAM is the sweet spot for 3B to 8B models at 4-bit — an 8B-class model at Q4_K_M fits an 8GB laptop GPU with room for context.
• GGUF is the portable format; GPTQ and AWQ are GPU-oriented alternatives. llama.cpp is the reference engine that everything else is built on.
• Always validate on your own task — quantisation loss is task-dependent, and code and maths degrade faster than chat.

Why run a model on-device at all

The case for on-device inference is not nostalgia for self-hosting. It is a set of concrete advantages that matter to builders shipping real products across India and the UK.

Data residency and privacy. When the model runs on the device, the prompt and the response never leave it. For a UK firm processing patient notes, legal documents or anything touched by data-protection obligations, that is the difference between a compliance review that passes and one that drags on for months. For an Indian fintech handling customer financial data under the DPDP framework, keeping inference local sidesteps an entire category of cross-border transfer questions.

Latency and offline operation. A model on the device answers in the time it takes to generate tokens, with no network round-trip. An edge deployment in a warehouse in Pune or a retail unit in Manchester keeps working when the connection drops. For interactive features, the absence of a network hop is felt by users even when the network is healthy.

Cost. A model running on hardware you already own has a marginal cost of roughly zero per request. For high-volume, low-margin workloads — classification, extraction, routing, drafting — that economics is hard to beat. If you are weighing this against a hosted endpoint, the companion piece on self-hosted LLM serving costs walks through where the crossover sits once you add batching.

The catch is that on-device means small. You are not running a frontier model on a laptop. You are running a 3-to-8-billion-parameter model, quantised, and the art is in making that small model good enough for your specific job.

Choosing a model by RAM budget

Start from the constraint, not the model. The single number that decides everything is how much memory you have to give the model — VRAM on a GPU, or unified system RAM on Apple Silicon. Once you know that, the parameter count and the quantisation level fall out almost mechanically.

The 8GB VRAM tier is the workhorse of on-device AI, because it covers the bulk of consumer gaming GPUs and mid-range laptops. At that budget, 3B-to-8B models at 4-bit are the sweet spot, and an 8B-class model fits at Q4_K_M with enough headroom for a useful context window. As of June 2026 the strong small bases to reach for are Llama 3.1 8B Instruct, Qwen2.5 7B Instruct and Mistral 7B Instruct v0.3. At the very small end, Gemma 4 E2B quantisation-aware-training checkpoints come in around 1GB, which opens up phones and the smallest edge boxes.

RAM / VRAM budget	Recommended params	Example models (June 2026)	Q4_K_M size (approx.)
2–4GB	1B–3B	Gemma 4 E2B (QAT), small 3B instruct bases	~1–2GB
8GB	3B–8B	Llama 3.1 8B Instruct, Qwen2.5 7B Instruct, Mistral 7B Instruct v0.3	~4–5GB
16GB	7B–13B	8B at Q5_K_M, 13B-class at Q4_K_M	~5–8GB
24GB+	13B–32B	Larger instruct models, longer context budgets	~8–20GB

Two practical notes. First, the Q4_K_M size in the table is the weights alone; leave headroom for the KV cache, which grows with your context length. Second, do not over-index on benchmark leaderboards when picking between similarly sized bases. A 7B model that is half a point higher on a generic benchmark can easily be worse on your extraction schema. Pick two or three candidates in your tier and let your own evaluation decide.

What GGUF actually is

GGUF is a file format. It packages the model weights, the tokeniser, the metadata and the quantisation scheme into a single self-contained file that the llama.cpp family of runtimes can load directly. It replaced the older GGML format and is now the de facto standard for distributing models meant to run locally. When you download a model from Hugging Face for use with Ollama, LM Studio or Jan, you are almost certainly downloading a GGUF.

The reason GGUF matters to builders is portability. The same GGUF file runs on a CPU, on an NVIDIA or AMD GPU, and on Apple Silicon, because llama.cpp abstracts the hardware underneath. You are not locked to a vendor toolchain. You can develop on a Mac, test on a Linux box with a consumer GPU, and ship to an edge device, all with the same artefact.

The reference engine is llama.cpp. It is worth internalising that Ollama, LM Studio, Jan and KoboldCpp are all wrappers and conveniences built on top of llama.cpp. They differ in user experience — Ollama gives you a tidy command-line pull-and-run flow, LM Studio gives you a desktop GUI — but the inference underneath is the same engine. That is good news: a quirk you learn in one carries over to the others.

Quantisation levels explained

Quantisation reduces the precision of the model weights. A full-precision model stores each weight as a 32-bit or 16-bit floating-point number; a 4-bit quantisation stores it in roughly 4 bits. Fewer bits per weight means a smaller file, less memory and faster inference, at the cost of some accuracy. The clever part of modern GGUF quantisation is that it is not uniform — the "K" quant types use a mix of precisions across different parts of the model, spending more bits where they matter and fewer where they do not. That is why a Q4_K_M model holds quality far better than a naive flat 4-bit would.

Level	Bits (approx.)	Relative size	Quality retention	When to use
Q4_0	~4-bit (flat)	~28% of FP32	Lower than K-quants	Older flat 4-bit; used by some QAT checkpoints
Q4_K_M	~4-bit (mixed)	~28% of FP32	~92% of full precision	The default — smallest with low loss
Q5_K_M	~5-bit (mixed)	~34% of FP32	Higher than Q4_K_M	Worth it if RAM is plentiful or task is sensitive
Q8_0	~8-bit	~53% of FP32	Near-lossless	When quality matters more than size

In plain terms: Q4_K_M is where you start. It is the smallest quantisation that keeps quality high — around 92% of full precision, with under 3% typical loss — which is why it is the default download for most popular models. Step up to Q5_K_M when you have memory to spare or your task is sensitive, and the extra accuracy is worth the larger footprint. Q8 is near-lossless and large; reach for it when quality matters more than size and you can afford the memory. Q4_0 is the older flat 4-bit scheme — you will mostly encounter it in quantisation-aware-training checkpoints that were prepared that way, rather than choosing it yourself.

GGUF vs GPTQ vs AWQ

GGUF is not the only quantisation format. The two you will hear most often alongside it are GPTQ and AWQ. All three achieve broadly the same compression — they cut model size by about 75% and roughly halve VRAM at 4-bit — so the choice is not about how small they get. It is about where they run and what ecosystem they plug into.

• GGUF is the portable, local-ecosystem default. It runs on CPU, GPU and Apple Silicon, and it is what llama.cpp, Ollama, LM Studio, Jan and KoboldCpp consume. If your target is a laptop, a Mac or a mixed-hardware edge fleet, GGUF is almost always the right answer.
• GPTQ is GPU-oriented. It was one of the first post-training quantisation methods to see wide adoption and is well supported in GPU-centric serving stacks. It shines when you are serving from a GPU and want a format those servers expect.
• AWQ (activation-aware weight quantisation) is also GPU-oriented and is often favoured in high-throughput GPU serving for its accuracy at 4-bit. Like GPTQ, it assumes a GPU is doing the work.

The decision rule is simple. If the model runs on varied or CPU-inclusive hardware, choose GGUF. If you are serving from GPUs in a data centre and your serving framework expects GPTQ or AWQ, use those. The reference GGUF engine remains llama.cpp; the others live in the GPU-serving world. If your deployment is the latter, the self-hosted serving guide covers the throughput trade-offs in more depth.

The sizing maths, with a worked example

You do not need to guess whether a model will fit. A back-of-the-envelope formula gets you within a useful margin:

rough VRAM ≈ (params × bits-per-weight ÷ 8) + KV-cache + overhead

The first term is the weights. Take the parameter count, multiply by the bits per weight, divide by eight to get bytes. The second term is the KV cache, which holds the attention state for your context and grows with context length. The third is runtime overhead — the framework, the compute buffers, a little slack.

Work it through for an 8B model at 4-bit. The weights are 8 billion × 4 bits ÷ 8 = 4 billion bytes, so about 4GB, and in practice closer to 4 to 5GB once the mixed-precision K-quant scheme and alignment are accounted for. Add a gigabyte or two for the KV cache at a typical context length, plus overhead, and you land comfortably inside an 8GB budget. That is exactly why 8B at Q4_K_M is the canonical 8GB-VRAM choice.

When a model does not fit, you are not stuck. The standard escape hatch is to offload some layers to the CPU. llama.cpp lets you choose how many layers run on the GPU; the rest run on the CPU and main RAM. A partial offload is slower than a full-GPU run because the CPU layers are the bottleneck, but it is meaningfully faster than running everything on the CPU — and it lets you run a larger model than your VRAM alone would allow.

CPU offload and Apple Silicon unified memory

Apple Silicon changes the sizing picture in a way that is genuinely useful for builders. Instead of separate VRAM and system RAM, it uses unified memory: the model weights and the KV cache share the same pool as everything else the machine is doing. A 16GB Mac comfortably runs a 7-to-8B model at Q4_K_M, because there is no rigid VRAM ceiling separate from system memory — the whole 16GB is available to the model, minus what the OS and your other applications are using.

This makes Apple Silicon an excellent development target. You can prototype an on-device feature on a MacBook, measure its memory and latency honestly, and have confidence that the same GGUF will behave on a Linux GPU box. The unified-memory architecture also means there is no host-to-device copy for the weights, which helps latency.

On a discrete-GPU machine, the picture is the split one the sizing maths assumes: weights and KV cache must fit in VRAM, or you offload the overflow to CPU. The mental model to keep is that unified memory trades a hard VRAM wall for contention with the rest of the system, while a discrete GPU gives you dedicated memory with a firm ceiling. Both are workable; they fail in different ways.

Validating quality after quantising

This is the step most teams skip, and it is the one that bites them. Quantisation loss is real but task-dependent, and generic benchmarks tell you almost nothing about how the quantised model behaves on your job. A model that loses three points on a public leaderboard might lose nothing on your classification task and a great deal on your code-generation task.

The fix is cheap. Build a small golden set — fifty to a couple of hundred examples drawn from your real workload, each with a known-good answer — and run both the full-precision and the quantised model against it. Compare. If the quantised model holds up, ship it. If it drops on the cases you care about, step up a quantisation level and measure again. This is the same discipline that distilling a teacher model into a student demands: trust measurement over intuition.

Pay particular attention to the failure modes that quantisation makes worse. Code that almost compiles, arithmetic that is plausibly but subtly wrong, multi-step reasoning that loses a thread — these degrade faster than open conversation and are exactly where a silently lower-quality model does the most damage. If you are routing between models to control cost, the patterns in cutting LLM costs with caching and routing pair well with on-device quantisation: send the easy traffic to the small local model and reserve a larger model for the hard cases your golden set flags.

Tooling: llama.cpp, Ollama and LM Studio

You rarely need to quantise a model yourself, because the community publishes GGUFs at every common level on Hugging Face. But knowing how is worth the few minutes it takes, and it lets you quantise a model you have fine-tuned. With llama.cpp built, the command is a one-liner:

# Quantise an FP16 GGUF down to Q4_K_M with llama.cpp
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

For day-to-day running, Ollama is the lowest-friction path. It pulls a model in a chosen quantisation by tag and runs it with one command:

# Pull an 8B model at Q4_K_M and run it
ollama pull llama3.1:8b-instruct-q4_K_M
ollama run llama3.1:8b-instruct-q4_K_M "Summarise this paragraph in one sentence."

If you want to drive a GGUF from Python — to wire it into an application rather than a chat prompt — llama-cpp-python loads the file directly and lets you set how many layers go to the GPU:

from llama_cpp import Llama

llm = Llama(
    model_path="model-Q4_K_M.gguf",
    n_gpu_layers=-1,   # -1 = offload all layers that fit; lower this to offload some to CPU
    n_ctx=8192,        # context window; larger means more KV-cache memory
)

out = llm("Extract the invoice total as a number:", max_tokens=32)
print(out["choices"][0]["text"])

LM Studio and Jan cover the same ground with a desktop GUI, which is handy for non-engineers on the team who want to evaluate models without touching a terminal. All of them are reading the same GGUF and running the same llama.cpp engine underneath, so a model you validate in one behaves identically in the others. Hugging Face's own documentation, at huggingface.co/docs, is the place to confirm which quantisations a given model ships with.

Dual-market deployment notes

The on-device pattern adapts neatly to a hybrid architecture, and the regional details differ between markets. For a team that wants local inference most of the time but needs to burst to the cloud for heavy jobs, run the quantised GGUF on-device and fail over to a larger model on a GPU instance — AWS Mumbai (ap-south-1) for an Indian deployment, London (eu-west-2) for a UK one — keeping the burst region close to your users and your data.

The residency angle pulls in opposite directions depending on the market. An Indian edge deployment — a quantised model running on a small box in a retail outlet or a manufacturing line — keeps inference at the point of use, which is ideal where connectivity is intermittent and latency to a central region would hurt. A UK on-premises deployment, by contrast, is often driven by data-residency and sector rules: a healthcare or legal firm runs the model inside its own perimeter precisely so that regulated data never leaves it. The same GGUF artefact serves both stories; what changes is why you are keeping it local.

Conclusion

On-device AI is a sizing problem first and a model problem second. Start from your memory budget, pick a model that fits the tier — a 3-to-8B instruct base for the common 8GB case — quantise to Q4_K_M, and ship it as a portable GGUF that runs the same on a Mac, a consumer GPU and an edge box. Step up to Q5_K_M or Q8 only when a golden set drawn from your own task tells you the extra memory buys real quality, and watch the reasoning, code and maths tasks most closely. The tooling — llama.cpp and the conveniences built on it — has matured to the point where the hard part is no longer running the model. It is choosing well and measuring honestly, and that part is entirely in your hands.

For the cloud-serving side of the same trade-off, see the deep dive on self-hosted LLM serving, and for context on the open-weight models that make all of this possible, the report on Meta's Llama 4 open-weight release.