Gemma 4 Runs On Your Laptop: QAT, 1GB Models and Arm's 5.5x Boost

Three things builders should know

• The model now fits the device. Google DeepMind released Gemma 4 12B on 3 June 2026, built to run locally on a consumer laptop with 16GB of RAM, and the first mid-sized Gemma with native audio inputs.
• QAT shrank it to about 1GB. Around 5 June 2026 DeepMind shipped quantisation-aware training checkpoints — a Q4_0 build plus a new mobile-specialised format that cuts the Gemma 4 E2B footprint to roughly 1GB.
• Arm hardware makes it interactive. Arm reports Gemma 4 E2B on Armv9 CPUs hits about a 5.5x average prefill speedup and up to 1.6x faster decode — fast enough for real on-device use, not a demo.

For the past two years the on-device story has been "almost". You could run a small model on a phone, but it was either too dumb to be useful or too slow to feel responsive. Gemma 4 closes that gap. It is not a single model but a family that runs the full range from an Arm handset to a single workstation GPU, and the June 2026 quantisation-aware checkpoints are the piece that turns "technically possible" into "ship it on Monday". This is a meaningful shift in where AI inference happens — and therefore in who pays for it, who controls the data, and how you should architect your next product.

One point up front, because it matters for compliance and it gets misreported constantly. Gemma 4 is open-weight, not open-source. It ships under Google's Gemma licence and terms of use — not Apache-2.0, and not open-source in the OSI sense. You can download the weights, fine-tune them and run them on your own hardware, but the licence carries usage terms you must read before you build a commercial product on top. Throughout this piece we say "open-weight" deliberately.

What Google actually shipped in June 2026

The headline release on 3 June 2026 was Gemma 4 12B: a dense model designed from the start to fit inside 16GB of RAM, which is the floor for most mainstream laptops sold today. It is also the first mid-sized Gemma to take native audio inputs, so you can feed it speech directly rather than bolting on a separate transcription stage. That alone changes the shape of a lot of on-device assistants — voice in, structured answer out, nothing leaving the machine.

Then, around 5 June 2026, came the part that made this practical for phones. DeepMind released quantisation-aware training checkpoints: a standard Q4_0 build for laptops and desktops, and a new mobile-specialised quantisation format that brings the Gemma 4 E2B footprint down to about 1GB (per MarkTechPost). A roughly 1GB multimodal model is small enough to sit comfortably on a mid-range Arm phone alongside everything else the device is doing.

The wider lineup, per Google's release materials, spans edge variants E2B and E4B, the 12B dense model, and a 26B Mixture-of-Experts model that activates only about 4B parameters per token — Google's "A4B" design — which Google says approaches the performance of a dense roughly 31B model at materially lower compute. The MoE variant is the interesting middle ground: near-frontier quality on a single capable GPU, without paying to run 31B dense weights on every token.

Variant	Size class	Active params / token	Target RAM	On-device fit
E2B	Edge (smallest)	~2B dense	~1GB (mobile QAT)	Mid-range Arm phone
E4B	Edge	~4B dense	A few GB	Flagship phone / thin laptop
12B	Dense, mid-sized	12B dense	16GB	Consumer laptop
26B MoE ("A4B")	Mixture-of-Experts	~4B active	Single capable GPU	Workstation / edge server

The figures for the edge variants beyond the published 1GB E2B number are framed as guidance, not vendor-confirmed exact RAM targets — treat the precise byte counts as approximate and benchmark on your own target device before committing.

Why quantisation-aware training is the real story

Quantisation is how you make a model smaller: you store the weights at lower precision, typically 4 bits instead of 16, so the file shrinks by roughly a factor of four and the model uses far less memory and bandwidth at runtime. The catch has always been quality. Squash a model down after it has finished training — naive post-training quantisation — and you lose accuracy, sometimes badly, because the model was never taught to tolerate the rounding errors that low precision introduces.

Quantisation-aware training fixes this by folding the quantisation step into training itself. The model trains while seeing the low-precision representation it will eventually be deployed at, so it learns to be robust to those rounding errors rather than getting blindsided by them at the end. The result is that a 4-bit QAT checkpoint holds quality substantially better than a naive 4-bit post-training squash of the same weights — you get most of the size and speed benefit without most of the accuracy cost.

Approach	When quantisation happens	Size benefit	Quality impact
Naive post-training quant	After training, applied to finished weights	~4x smaller at 4-bit	Noticeable accuracy loss; can degrade sharply
Quantisation-aware training (QAT)	Folded into the training loop	~4x smaller at 4-bit	Quality largely preserved

The Arm numbers that make it interactive

A small file is necessary but not sufficient. The model also has to run fast enough that a person waiting for an answer does not give up. This is where the Arm collaboration matters. Arm reports that Gemma 4 E2B on Armv9 CPUs delivers roughly a 5.5x average prefill speedup and up to 1.6x faster decode (per Arm Newsroom).

Prefill is the phase where the model reads your prompt; decode is where it writes the answer token by token. A 5.5x prefill speedup means the model starts responding far sooner after you hit enter, which is the lag people notice most. The decode gain keeps the words flowing at a readable pace once it starts. Together they are the difference between an on-device assistant that feels usable and one that feels like a science experiment. And crucially, these gains are on the CPU — no dedicated NPU or discrete GPU required — which is exactly the silicon already sitting in the phones and Arm laptops your users own.

What should you actually do?

Here is the builder-facing question: given a family that runs from a 1GB phone model to a 26B MoE, which variant do you pick, and when does running it yourself beat calling a cloud API?

Pick by where the work runs. If the target is a phone and the tasks are routine — classification, extraction, short summarisation, voice commands — start with E2B at the mobile QAT precision and only move up to E4B if your evaluations demand it. If the target is a laptop and you want a genuinely capable local assistant, the 12B dense model at 16GB is the sweet spot. If you are running an edge server or a workstation and want near-frontier quality without frontier cost, the 26B MoE gives you roughly dense-31B behaviour while only activating about 4B parameters per token.

On-device wins on three axes. The first is marginal cost. Once the model is on the device, routine inference is effectively free — you are not paying per token to a cloud API on every request. For a high-volume, low-complexity workload that economics is decisive. The second is privacy and data residency. The third is latency and offline behaviour: no round trip, no dependence on a connection.

Think about it in concrete dual-market terms. An Indian startup shipping an offline-first product to users on patchy connectivity can put Gemma 4 E2B directly on the handset and stop paying per-call cloud fees entirely — the unit economics finally close. A UK team handling regulated data with residency requirements can keep inference inside the device or inside a London region and never send a customer record to a third-party API. Compare that with the old default: routing every request to, say, AWS Mumbai or AWS London, paying per token, and shipping the data off-box. For a large share of routine tasks, that round trip is now optional.

Where this fits in the open-weight landscape

Gemma 4 does not arrive in a vacuum. It sits alongside other open-weight families racing to make capable models small and cheap to run — Meta's Llama 4 pushed multimodal Mixture-of-Experts into open weights, and Gemma's own thinking-mode variants brought structured reasoning to the family. What Gemma 4's June checkpoints add is the device-first packaging: the explicit 16GB laptop target, the native audio inputs, and the QAT work that gets a multimodal model onto a phone.

The strategic read for builders is that the floor has moved. A year ago, "we need an LLM" meant "we need a cloud API budget". Today, a large share of routine tasks can run on hardware your users already own, at near-zero marginal cost, with the data never leaving the device. That does not retire the frontier models — you will still reach for a hosted model on the hard reasoning tail — but it changes the default. The sensible architecture for most products in 2026 is a router: small on-device model for the common case, cloud frontier model for the long tail, and a clear policy for which requests are allowed to leave the device at all.

If you are serving any of these models yourself, the operational details — quantisation choices, batching, KV-cache management and cost-per-token — are where the savings are actually won or lost. We have a full walkthrough of that in our guide to self-hosting LLMs with quantisation and batching. For the wider open-weight context, see our pieces on Gemma 4's thinking modes and Meta's Llama 4 multimodal MoE.

The bottom line

Gemma 4 is the moment the on-device promise stops being a promise. A roughly 1GB multimodal E2B model that runs interactively on Arm CPUs, a 12B model built for a 16GB laptop, and a 26B MoE that approaches dense-31B quality on a single GPU — all open-weight, all available now. The QAT checkpoints are the unlock, because they make the small variants good enough to trust. Read the Gemma licence before you ship, benchmark on your real target device rather than the published figures, and design for a router between on-device and cloud rather than betting everything on one. Primary sources: the Google DeepMind blog and the Arm Newsroom.