Qwen 3.6 27B: Frontier Coding on a Single 24GB GPU

What this guide gives you

This is a how-to, not a leaderboard write-up. The headline is simple: a dense 27-billion-parameter coding model now reportedly fits on a single 24GB consumer GPU at Q4 quantisation, runs entirely on your own hardware, and is good enough for day-to-day agentic coding. For an AI builder in Bengaluru or Bristol, that changes the maths on cost, privacy and offline working. Here is what you will walk away with:

• The workflow first — how to wire a local model into an agentic coding loop with editors and agents you already know.
• The hardware reality — what a 24GB card actually buys you, and the VRAM arithmetic behind quantisation choices.
• The runtimes — Ollama for the fastest start, llama.cpp for control, vLLM for throughput.
• The economics — a one-off GPU plus electricity versus a month of heavy per-token cloud usage.
• The honest limits — where a hosted frontier model still wins, and why community benchmark numbers are not your numbers.

Why local coding agents now matter

For two years the default answer to "which coding model" was a hosted API. That default is being challenged on three fronts, and all three matter to builders in India and the UK.

Cost. A heavy agentic coding habit — an agent that reads your repo, plans, edits, runs tests and iterates — burns tokens fast. A single engineer running an agent through a working day can rack up a meaningful monthly bill on a hosted flagship. A local model has a fixed up-front cost and then a marginal cost of roughly your electricity. For a small team or a solo builder watching runway, that swing is the whole argument.

Privacy and data sovereignty. Sending proprietary source code to a third-party API is a governance question, not just a technical one. Under India's DPDP Act and the UK's data-protection regime, builders handling personal data inside their codebases and test fixtures have to think carefully about where that data flows. A model that runs on a machine you control keeps the code, the prompts and the logs on-premises. For a fintech in Mumbai or a health-tech startup in Manchester, that can be the difference between a feasible pilot and a blocked one.

Offline and air-gapped work. Patchy connectivity, secure client environments, regulated networks — there are plenty of settings where the cloud simply is not an option. A self-hosted model works on a train, in a locked-down VPN, or inside an air-gapped lab. We have covered the broader self-hosted agent picture before in our look at the OpenClaw open agent stack, and local model weights are the missing piece that makes that stack fully independent of any vendor.

The hardware — what a 24GB card buys you

The whole premise here is that a 27B dense model can be squeezed onto a single prosumer card. The cards in play are the ones builders can actually buy:

• RTX 4090 (24GB) — the workhorse. Widely available, strong price-to-VRAM, and the reference point for "does it fit".
• RTX 5090 (32GB) — the comfortable choice. The extra 8GB is not linear; it is the headroom that turns a tight fit into a usable context window.
• RTX A5000 / workstation cards (24GB) — the quiet, blower-cooled option for an office or a rack.

The arithmetic is straightforward. Model weights at full FP16 precision need roughly two bytes per parameter, so a 27B model is about 54GB — well past a single 24GB card. Quantisation shrinks the weights. At Q4 you are down to roughly half a byte per parameter for the bulk of the tensors, which community trackers put around 17 to 20GB for a 27B model. That fits on 24GB, but only just — and you still need memory for the KV cache that holds your context.

Quantisation, explained for builders

Quantisation trades numerical precision for memory. Lower precision means smaller weights and faster inference, at some cost to quality. For a 27B model the practical options look like this — treat the VRAM figures as approximate, since they depend on the exact GGUF build and your KV-cache settings.

Format	Approx weights (27B)	Fits 24GB?	Quality trade-off
FP16 (full)	~54GB	No	Reference quality; needs multi-GPU or 80GB cards
Q8	~28-29GB	No (fits 32GB)	Near-lossless; the safe choice if you have the VRAM
Q4_K_M (GGUF)	~17-20GB	Yes (tight)	The sweet spot — small, measurable quality loss for coding
Q3 / lower	~13-15GB	Yes (roomy)	Noticeable degradation; only when VRAM is scarce

For most builders, Q4_K_M is the default. It is the format that lets a 27B model live on a 24GB card with acceptable quality for coding tasks, and it is what the community has converged on for exactly this size class. Go to Q8 if you have a 32GB card and want to remove quantisation as a variable; drop below Q4 only if you are genuinely out of memory.

The runtimes — Ollama, llama.cpp, vLLM

You have three sensible ways to actually serve the model, depending on how much control you want.

Ollama is the easiest start. It wraps llama.cpp, manages the model download and quantisation, and exposes an OpenAI-compatible endpoint that your coding tools can talk to. For a single developer machine, this is the fastest path from nothing to a working local model.

# Pull a Q4 quantised build and run it
ollama pull qwen3.6:27b-q4_K_M
ollama run qwen3.6:27b-q4_K_M

# Ollama exposes an OpenAI-compatible API on localhost:11434
# Point your coding agent at:
#   base_url: http://localhost:11434/v1
#   api_key:  ollama   (any non-empty string)

llama.cpp is the layer underneath. Use it directly when you want fine control over GGUF quantisation, KV-cache settings, context length and offload behaviour. It is the right tool when you are squeezing a model onto a tight 24GB card and need to tune exactly how much of the model and cache sit in VRAM.

vLLM is for throughput. If you are serving the model to a small team rather than a single user, vLLM's paged-attention scheduling handles concurrent requests far better than a single-stream runtime. It typically wants more headroom, so this is more comfortable on a 32GB card or a small multi-GPU box. If you want to go deeper on serving choices, our coverage of agentic coding tooling around Cursor Composer 2.5 and the Cursor SDK is a useful companion on the client side of the same workflow.

Wiring Qwen into an agentic coding loop

A model on its own does nothing useful for coding. The value comes from connecting it to an agent that can read files, propose edits, run commands and iterate. Because Ollama and vLLM both expose OpenAI-compatible endpoints, most open coding tools will talk to a local model with nothing more than a base-URL change.

• Cline — a popular open coding agent that runs inside VS Code. Point its custom OpenAI-compatible provider at your local endpoint.
• Continue — an open IDE assistant for VS Code and JetBrains, configured with a local model block.
• Aider — a terminal-native pair-programmer that edits files and makes git commits, happy to drive a local model.
• An open agent stack — for a fuller autonomous loop, plug the local model into a self-hosted stack rather than a single editor plugin.

# Aider against a local Ollama endpoint
export OPENAI_API_BASE=http://localhost:11434/v1
export OPENAI_API_KEY=ollama
aider --model openai/qwen3.6:27b-q4_K_M

# Continue (~/.continue/config.json), minimal local block
# {
#   "models": [
#     { "title": "Qwen 3.6 27B local",
#       "provider": "openai",
#       "model": "qwen3.6:27b-q4_K_M",
#       "apiBase": "http://localhost:11434/v1" }
#   ]
# }

The economics — local versus cloud over a heavy month

The cost case is the one most builders care about. The comparison below sketches a heavy-use month for a single engineer who codes with an agent most of the working day. Treat the cloud figure as illustrative — your actual bill depends entirely on token volume and the model you pick — and remember the GPU is a one-off that amortises across many months.

Approach	Up-front	Monthly running	Notes
Local — RTX 4090, Q4	One-off card cost	Electricity only (a few units of currency)	Marginal cost per token is effectively zero once bought
Local — RTX 5090, Q4/Q8	Higher one-off	Electricity only	More VRAM headroom; longer context, higher throughput
Cloud hosted API	None	Scales with token volume — can be substantial under heavy use	Zero capex; no data control; cost grows with usage

The shape of the trade is what matters. Cloud is pay-as-you-go with no capital outlay, which is right for spiky or occasional use. Local is a fixed up-front cost and then near-free tokens, which wins decisively for sustained heavy use — the breakeven against a heavy cloud habit often lands inside a few months. For builders weighing subsidised compute as a third option, our guide to the IndiaAI Mission's sub-₹100/hour GPU access is worth a read alongside this — rented GPU hours sit neatly between owning a card and paying per token.

Where local still loses to hosted

Honesty matters here, because over-promising local capability helps nobody. A 27B model at Q4 is genuinely capable for everyday coding, but it is not a frontier hosted flagship, and there are tasks where you should still reach for the cloud.

• Long context. The model may advertise a very large maximum context, but at Q4 on a 24GB card your usable KV cache is small. Genuinely long-context work — reasoning across an entire large repository in one pass — is still more reliable on a hosted model with room to breathe.
• The hardest multi-step tool-calling. Reliable, long-horizon agentic loops with many tool calls are where the frontier hosted models still pull ahead. A local model will handle the common cases well and stumble more often on the awkward ones.
• Peak reasoning quality. For the genuinely difficult bug or the subtle architectural refactor, a hosted flagship still has an edge. A pragmatic pattern is split routing — run the everyday loop locally and escalate only the hard tasks to a hosted model.

This is the open-weight story of 2026 in miniature. The field has filled out fast — Kimi K2.6, GLM-5.1 under MIT, DeepSeek V3.2 and R1 also under MIT, Gemma 4 with vision and tool calling, Llama 4 Scout for long context, and gpt-oss:20b as a strong small option that fits in roughly 16GB. The Qwen family sits in the Apache-2.0 camp. For a broader view of how the open-weight leaderboard is moving, see our coverage of DeepSeek V4 Pro topping the open-weight leaderboard and of Cohere Command A+ going Apache 2.0. The headline is that you now have real, shippable choices that never touch a vendor API.

Primary references: the model card on Hugging Face and Alibaba's own Qwen announcement. Read the licence file shipped with the weights before any commercial deployment.