On-Device AI: The Proof-of-Work Portfolio That Gets You Hired

Why on-device is a hiring lane now

For most of the past three years, "AI engineer" meant "person who calls a hosted API". That is still the bulk of the work. But a second lane has opened underneath it, and it is hiring quickly: building models that run on the device — a laptop, a phone, a vehicle head-unit, a factory gateway — with no round trip to a data centre.

Three forces are pushing workloads to the edge, and none of them is a fad. First, the silicon arrived. Qualcomm's Snapdragon X2 Elite ships neural processing units rated at roughly 80 TOPS on Qualcomm's own figures, enough to run a quantised small language model at conversational speed without touching the cloud. Apple, AMD and Intel are all shipping comparable NPUs. Second, the models shrank to fit. Google's Gemma 4 runs on a 16GB laptop or a recent phone; in India, Sarvam is building on-device and edge model variants aimed specifically at local, low-bandwidth deployment. Third, the rules changed. Privacy expectations, the UK's data-protection regime and India's DPDP framework all make "the data never leaves the device" a feature you can sell rather than a constraint you tolerate.

For builders in India and the UK chasing remote-global roles, this matters for a practical reason: the supply of engineers who can credibly say "I have shipped a model that runs locally, and here are the numbers" is still small. That gap is exactly where a focused portfolio gets you noticed. This guide gives you five proof-of-work projects, the rubric hiring teams actually use to judge them, and a 30-day plan to ship the first one.

The demand picture across India and the UK

The headline talent shortage is well documented — by various industry estimates the sector faces millions of unfilled AI roles over the next few years, and on-device is a slice of that demand growing faster than the average. The jobs sit in places generalists overlook.

In the UK, edge-AI hiring clusters around regulated and latency-sensitive sectors: healthcare devices that cannot stream patient data, financial endpoints that need offline inference, automotive and industrial firms in the Midlands and the North building intelligence into hardware they already ship. London's consumer-app scene wants on-device features for privacy marketing and for cost — running inference on the user's phone is inference you do not pay a cloud bill for.

In India, the picture is shaped by bandwidth and scale. A feature that assumes a reliable connection fails for a large share of users; one that runs locally does not. The IndiaAI Mission's emphasis on indigenous, deployable on-device models creates demand for engineers who can take a model and make it fit a mid-range Android handset. Add the global remote market — UK and EU companies hiring India-based edge engineers because the skill is scarce everywhere — and you have a lane that rewards a demonstrable, specific portfolio over a generic one.

The recurring theme in both markets: hiring managers are not short of people who have read about on-device AI. They are short of people who can prove they have done it. That is the whole game, and it is winnable with five weekends of focused work.

What edge and on-device hiring rubrics actually test

When a team screens for an edge role, they are checking a fairly consistent set of competencies. Knowing the rubric tells you what to build and, just as importantly, what to measure. Here is the signal they look for versus the weak signal that gets a portfolio quietly filtered out.

What hiring managers check	Strong signal	Weak signal
Quantisation literacy	A table comparing Q4_K_M, Q5_K_M and Q8 on the same model with measured quality loss	"I used a quantised model" with no format, no numbers
Local inference plumbing	A working agent on Ollama / llama.cpp with MCP tools and a reproducible run command	A screenshot of a chat with a hosted API
Hardware reasoning	Tokens-per-second by model size and device, with notes on memory bandwidth and battery	"It runs fast on my machine"
Quality after compression	An eval score before and after quantisation on a fixed task set	"Output looks fine to me"
Privacy and data residency	A feature where data provably never leaves the device, explained in the README	A claim of privacy with the model calling out to an API
Reproducibility	A README a stranger can follow to reproduce every number on their own hardware	Numbers in a slide with no method

The pattern is consistent: the left column is about doing the work, the right column is about talking about the work. A portfolio that lives entirely in the right column reads as marketing, and experienced reviewers discount it within seconds. Everything below is designed to keep you in the left column.

The five proof-of-work projects

These five projects are deliberately sequenced. Each one proves a distinct competency from the rubric above, and they reuse each other's artefacts — the benchmark you build in project two feeds the latency study in project four, and the local agent from project one becomes the shell for the offline RAG app in project three. Build them in order and you finish with a coherent body of work, not five disconnected demos.

Project	What it proves	How to show it
1. Local agent (offline task)	You can wire a model to tools and ship a working agent with no cloud dependency	GitHub repo + 60-second demo GIF of the agent completing a real task with the network off
2. Quantisation benchmark study	You understand the speed-versus-quality trade-off and can measure it	A table of tokens/sec and an eval score across Q4_K_M / Q5_K_M / Q8, plus a short write-up
3. Offline RAG over personal or enterprise docs	You can build retrieval that runs entirely on-device	Repo with a local embedding model + vector store; a demo answering questions with the network disabled
4. NPU / edge latency study	You can reason about hardware: model size, device, memory bandwidth, battery	A chart of tokens/sec versus model size across two or more devices, with method notes
5. Privacy-first on-device feature	You can turn data residency into a shipped feature	A local transcription or summarisation feature; README proving data never leaves the device

Project 1 — A local agent that does a real task offline

Build an agent on Ollama or llama.cpp that completes a genuinely useful task with no network connection. Wire it to tools using MCP so it can read files, query a local database or call a script. A good first task is something a hiring manager recognises: an agent that triages a folder of PDFs, an agent that drafts replies from a local mailbox export, or an agent that runs a maintenance checklist against local logs. What this proves is the whole stack — model loading, tool calling, an agent loop — running on your own machine. Show it with a repo, a clear README, and a 60-second screen recording with the Wi-Fi visibly switched off. The offline demo is the proof; do not skip it.

Project 2 — A quantisation benchmark study

Take one model — Gemma 4 or a comparable 7B–9B — and run it across at least three quantisation levels: Q4_K_M, Q5_K_M and Q8, all in GGUF format. For each, record tokens-per-second on your hardware and a quality score on a fixed task set of twenty to fifty prompts you grade the same way every time. The artefact is a single table that answers the question every edge team asks: how much quality do I lose to get this much speed? This is the highest-leverage project in the set because it teaches the exact vocabulary edge interviews test, and the resulting table is something you reference in every later project. Publish the prompts and the grading method so the numbers are reproducible.

Project 3 — An offline RAG app over personal or enterprise documents

Build retrieval-augmented generation that runs end to end on-device: a local embedding model, a local vector store such as a small on-disk index, and a local generation model. Point it at a corpus that means something — your own notes, an open enterprise dataset, public regulatory documents. The competency this proves is that you can build retrieval without a hosted vector database or an embedding API, which is precisely what a privacy-sensitive or low-bandwidth deployment needs. Demonstrate it answering questions with the network disabled, and document the indexing and retrieval choices in the README.

Project 4 — An NPU or edge latency study

This is the hardware-reasoning project. Measure tokens-per-second across model sizes and across at least two devices — your laptop and a phone, or a laptop with and without NPU acceleration enabled. Plot the results and write up what you learned about memory bandwidth as the real bottleneck, about how battery drain scales, and about where the practical ceiling sits for a given device class. You do not need an X2 Elite reference board to do this well; honest numbers from the hardware you have, with clear method notes, demonstrate exactly the reasoning edge teams want. If you can borrow access to an 80 TOPS-class NPU, note it as a comparison point rather than the whole study.

Project 5 — A privacy-first on-device feature

Finish with a shipped feature, not a benchmark. Local transcription or summarisation is the classic example: audio or text goes in, a useful result comes out, and the data provably never leaves the device. The README should make the privacy claim concrete — which model runs where, what is stored, what is not, and how a sceptic can verify it. This is the project that translates most directly to a job, because it is the shape of work an edge team actually ships. It also closes the loop on the rubric: it forces you to combine local inference, quality measurement and a data-residency story in one artefact.

How to package the proof so it travels

A project only counts if a reviewer can evaluate it in the two minutes they will give you. Packaging is not decoration; it is the difference between work that gets read and work that gets skimmed and closed. Four things do the heavy lifting.

The first is a public GitHub repository with a README a stranger can follow. State what the project does in the first sentence, list the exact commands to reproduce every number, and pin your model versions and quantisation formats. Reproducibility is the trust signal that separates an engineer from a tinkerer. The second is the benchmark itself — the table or chart, with the method beside it, so the numbers carry weight rather than just sitting there. The third is a short write-up: a blog post or a long README section that explains what you did, what surprised you, and what you would change. This is where you show judgement, which is the thing a CV cannot convey. The fourth is a 60-second demo — a screen recording or GIF, ideally with the network visibly off — because a reviewer who sees the thing work trusts the repo far faster than one reading claims about it.

Across all four, one rule governs: numbers beat adjectives. "37 tokens/sec on a 16GB M-series laptop at Q4_K_M, with a 4% drop on my eval set versus Q8" tells a hiring manager more than three paragraphs of prose. Lead with the figure, then explain it.

Making it findable: the Verified Builder profile

Here is the uncomfortable truth that catches out strong builders: a brilliant portfolio that nobody can find does not get you hired. Most on-device roles are filled through search, referral and directory profiles, not cold CV piles. The work of being discoverable is separate from the work of building, and it is the half most engineers neglect.

This is what a Verified Builder profile on AI Tech Connect is for. It gives you one page that hiring teams in India and the UK actually browse, linking your five projects, your benchmarks and your write-ups in a place built for exactly this evaluation. Instead of hoping a recruiter stumbles onto your GitHub, you put a structured, verified profile where the people hiring for edge and on-device roles are already looking. It pairs naturally with building in public — posting each benchmark as you ship it — so that by the time you finish project five, your profile already has an audience.

Interview prep for edge-AI roles

If your portfolio does its job, the interview is mostly a conversation about it — which is the easiest interview to pass, because you have lived the material. Expect questions in four areas, and have a concrete answer drawn from your own projects for each.

• The speed-quality trade-off. Be ready to explain why you would pick Q4_K_M over Q8 for a given device, with the numbers from your benchmark study. This is the single most common edge interview question.
• The bottleneck. Memory bandwidth, not raw compute, is usually what limits on-device inference. Your latency study gives you a real example to reason from.
• The privacy story. Articulate exactly how data stays on the device in your privacy-first feature, and where the boundary is. Vague answers here fail regulated-sector interviews fast.
• The trade-off you made. Pick one decision from a project where you chose speed over quality, or simplicity over completeness, and explain why. Judgement is what they are really testing.

Notice that every one of these maps to a project. That is the point of building the portfolio first: it turns the interview from an interrogation into a guided tour of work you already understand.

A 30-day plan to ship the first project

The fastest way to stall is to try to build all five at once. Sequence them. A focused 30-day plan gets you a hireable artefact and the momentum to keep going.

• Days 1–3: Set up the local stack. Install Ollama or llama.cpp, pull Gemma 4 or a 7B–9B model in GGUF, confirm it runs offline. Get one tokens-per-second number on your hardware.
• Days 4–10: Build project two, the quantisation benchmark. It is the fastest to ship and the most reusable. Produce the table, write the method, publish the repo.
• Days 11–14: Claim your Verified Builder profile and link the benchmark. Start building in public — post the table where edge engineers will see it.
• Days 15–24: Build project one, the local offline agent, reusing your stack. Record the 60-second offline demo.
• Days 25–30: Write the long-form write-up tying both projects together, polish both READMEs, and update your profile. You now have two of the five, packaged and findable.

From there, projects three, four and five each take a focused week or two, and every one of them slots into the profile you have already built. Within a quarter you have a complete, coherent on-device body of work and a place the hiring teams can find it.

The short version

On-device AI is a real, multi-year hiring lane across India and the UK, and the supply of engineers who can prove they have shipped local models is still thin. Five focused projects — a local agent, a quantisation benchmark, an offline RAG app, a latency study and a privacy-first feature — cover the competencies hiring teams test. Package each with a reproducible repo, a benchmark, a write-up and a short demo, and lead with numbers, not adjectives. Then make the work findable: a Verified Builder profile puts it in front of the people hiring, and an early Founding Builder badge is a signal that compounds. Build the proof, then make sure it gets seen.

For more on turning work into a portfolio that gets noticed, see the companion guides on building an AI engineer proof-of-work portfolio, landing your first AI engineer role through open-source proof of work, and building in public for visibility. On the demand side, the AI talent gap and builder visibility piece sets out why being findable matters as much as being skilled.