Ace the AI engineer take-home: what 2026 hiring teams score

What reviewers actually score

• The take-home is the deciding stage, not a formality. As of mid-2026 it is the only round where a hiring team watches you ship working software unsupervised — the closest proxy they have to your first week on the job.
• They scan for production signal, not theory. How you structure data, connect systems, handle failure, and measure quality matters far more than whether you can recite the transformer architecture.
• Retrieval is the gravitational centre. Most 2026 take-homes orbit retrieval-augmented generation, structured output, tool-using agents, evaluation and deployment — and across both markets, retrieval-augmented generation is consistently among the most in-demand AI-engineering skills.
• An eval of your own system is the biggest differentiator. A tiny golden set plus a judge beats a polished demo every time. Most candidates skip it; the ones who do not get the offer.
• Communication is graded. Reviewers live in your README. Trade-offs, known limitations and next steps earn marks that hidden cleverness never will.

Why the take-home decides the offer

Live interviews test how you reason under pressure with someone watching. Useful, but limited: a nervous strong engineer can stumble, and a smooth talker can coast on vocabulary. The take-home removes both distortions. It puts you alone with a problem, a clock and a blank repository, and asks the only question that actually predicts the job — can you ship working software that behaves in production? As of mid-2026, hiring teams across both the Indian and UK markets have converged on this stage as the highest-signal gate in the loop, and they weight it accordingly. A clean take-home can carry a wobbly phone screen; a weak take-home rarely survives a strong one.

What they are reading for is unglamorous and specific. Did you frame the problem before solving it, or did you start coding the first idea that occurred to you? When retrieval returned nothing useful, did your system fall over or degrade gracefully? Is there a test anywhere? Did you notice that one design choice would cost ten times more per query than another? These are operational instincts, and they are visible in a repository in a way they are not in a forty-minute call. The take-home is where a reviewer stops asking "does this person know about RAG?" and starts asking "would I trust this person to own a retrieval pipeline on Monday?"

This is also why the assignment is rarely about the model. The model is a commodity you call over an API. The take-home is about everything around the model — the plumbing, the data, the measurement, the failure handling and the explanation. That framing should shape every decision you make once the brief lands in your inbox.

What they send you, and what it is really testing

The brief usually looks deceptively small: "Build a system that answers questions over this set of documents," or "Build an agent that can look up orders and issue a refund," or "Given this messy dataset, expose a structured query interface." Whatever the wording, the underlying skeleton is almost always one of a handful of shapes, and each shape maps to a known set of signals.

• Retrieval over a corpus. The most common shape. It tests chunking, embedding, hybrid retrieval, re-ranking, and — critically — whether you grounded answers in retrieved context or let the model freewheel. Our guide to LLM-as-judge evaluation in production covers the measurement half of this exactly.
• A tool-using agent. Tests how you decompose a task into tool calls, scope each tool's permissions, defend against prompt injection from tool output, and add a human checkpoint before anything irreversible.
• Structured extraction. Tests schema design, validation, and what you do when the model returns malformed output — do you retry, repair, or fail loudly with a useful error?
• A small end-to-end feature. Tests the whole stack in miniature: ingestion, retrieval or inference, an API or interface, and some evidence that it works.

The brief will usually under-specify on purpose. A line like "make reasonable assumptions" is not laziness from the hiring team — it is the test. Reviewers want to see which assumptions you surface and document, because that is the same judgement you will apply to ambiguous tickets on the job. Treat every gap in the brief as a question to answer explicitly in your README rather than a corner to quietly cut.

The rubric 2026 hiring teams score against

Hiring teams are scoring signals, not features. The seven dimensions below are the ones that recur across AI-engineering take-homes in both the Indian and UK markets as of mid-2026. The table maps each dimension to what a weak submission looks like versus a strong one — read it as a self-audit checklist before you submit.

Dimension	Weak submission	Strong submission
1. Problem framing & scoping	Builds the first idea; no stated assumptions; over-scopes and leaves it half-working	Restates the problem, lists assumptions, scopes to a working vertical slice, and names what was deliberately left out
2. Retrieval / data correctness	Naive single-vector search; arbitrary chunking; answers ungrounded in retrieved context	Sensible chunking, hybrid retrieval where it helps, a re-ranking pass, and answers grounded with citations to source chunks
3. Eval & measurement	No measurement at all — "it seems to work"; a demo with three hand-picked examples	A golden set plus a judge that produces a quality number, run on every change, with the score reported in the README
4. Code quality & tests	One long script; no tests; no types; secrets hard-coded	Readable modules, at least a few meaningful tests, typed boundaries, and configuration via environment variables
5. Failure handling & guardrails	Crashes on empty retrieval or malformed output; no input validation; no fallback	Graceful degradation, retries with backoff, output validation, refusal on out-of-scope queries, and grounding to limit hallucination
6. Cost / latency awareness	No mention of tokens, cost or latency; stuffs the whole corpus into every prompt	A cost-per-query estimate, a note on caching, sensible chunk counts, and p50/p99 latency awareness
7. Communication (README, decisions)	README says only "npm install && npm start"; no trade-offs, no limitations	A decision log, named trade-offs, known limitations, eval results, and a clear "what I would do with more time"

The through-line across all seven is honesty under constraint. Nobody expects a production-grade system in a few evening hours. They expect a candidate who knows what production-grade would require, builds the most load-bearing slice of it, measures it, and tells the truth about the rest. Our companion piece on the AI engineer system-design interview shows the same rubric tested live in the room rather than in a repository.

The differentiator: evaluate your own system

If you do one thing beyond the brief, do this. As of mid-2026 the single biggest differentiator between a strong take-home and an average one is whether the candidate measured their own system's quality. The overwhelming majority of submissions ship a demo — a few cherry-picked queries that happen to work — and stop there. A small minority ship a demo plus an evaluation: a golden set of question-and-answer pairs and a judge that scores the system's answers against them, producing an actual number. That number is the loudest production signal a candidate can send, because measuring quality is exactly the discipline that separates a hobby project from a system someone will pay to run.

It does not need to be elaborate. Ten to twenty hand-written examples are enough to demonstrate the instinct. The harness below is small enough to drop into a take-home, runs offline, and prints a score you can quote in your README. The code stays in US English by convention.

"""eval_harness.py — a minimal golden-set + LLM-judge eval for a take-home.

Run:  python eval_harness.py
Output: per-item pass/fail + an aggregate accuracy score.
"""
import json
from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# 1) Golden set: 10-20 hand-written cases is plenty to show the instinct.
GOLDEN = [
    {
        "question": "What is the refund window for a digital order?",
        "reference": "14 days from the date of purchase.",
    },
    {
        "question": "Does the policy cover items bought on sale?",
        "reference": "Yes, sale items are covered under the same 14-day window.",
    },
    # ... add 8-18 more, drawn from the corpus the take-home gave you.
]

# 2) The system under test. Replace with a call into YOUR pipeline.
def answer(question: str) -> str:
    from my_app.rag import answer_question  # your retrieval + generation
    return answer_question(question)

# 3) An LLM-judge with an explicit rubric. Ask for a strict JSON verdict.
JUDGE_PROMPT = """You are grading an AI system's answer against a reference.
Score 1 if the answer is factually consistent with the reference and does not
invent details, otherwise score 0. Penalise ungrounded claims.

Question: {question}
Reference answer: {reference}
System answer: {system}

Return ONLY JSON: {{"score": 0 or 1, "reason": "one short sentence"}}"""

def judge(question: str, reference: str, system: str) -> dict:
    msg = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                question=question, reference=reference, system=system
            ),
        }],
    )
    return json.loads(msg.content[0].text)

# 4) Run the eval and report an aggregate score.
def main() -> None:
    passed = 0
    for case in GOLDEN:
        system_answer = answer(case["question"])
        verdict = judge(case["question"], case["reference"], system_answer)
        passed += verdict["score"]
        flag = "PASS" if verdict["score"] else "FAIL"
        print(f"[{flag}] {case['question']}  -- {verdict['reason']}")
    score = passed / len(GOLDEN)
    print(f"\nGolden-set accuracy: {passed}/{len(GOLDEN)} = {score:.0%}")

if __name__ == "__main__":
    main()

That is the whole idea: a fixed set of cases, your system under test, and a judge with an explicit rubric that returns a parseable verdict. Quote the resulting score in your README, note which cases failed and why, and you have demonstrated more production maturity than ninety per cent of the pile. If you want the rigorous version — judge calibration, avoiding self-preference bias, and wiring evals into continuous integration — our deep-dive on LLM-as-judge evals in production is the next step.

Time-boxing: ship a vertical slice, do not gold-plate

The most common way strong engineers fail a take-home is by over-building. Given an open brief and professional pride, it is tempting to design the system you would run in production — multi-tenant, observable, horizontally scalable — and then run out of time with three half-finished subsystems and nothing that works end to end. Reviewers cannot score intentions. They score what runs.

The discipline is to ship a vertical slice: the thinnest path through the whole system that actually works. For a retrieval take-home that means ingest a few documents, retrieve, generate a grounded answer, and evaluate it — all of it, however modestly. A complete slice that retrieves over ten documents and scores 80% on a golden set beats an ambitious agent framework that cannot answer a single question because the retrieval layer was never finished. Once the slice works, spend any remaining time on depth where it shows judgement — better chunking, a re-ranker, a couple more tests — not on breadth for its own sake.

Then document the cuts. A short "what I would do with more time" section turns every feature you deliberately skipped into evidence of judgement rather than a gap. Reviewers credit a named, reasoned omission far more than a feature you bolted on but never tested. Scope down, ship the slice, write down the trade-offs — that sequence is the whole game.

The README and build-in-public: where the marks hide

Reviewers spend a surprising amount of their grading time in your README and your decision log — often more than in the code itself. The README is where you control the narrative: it is your chance to explain the trade-offs you made, the things you knew were imperfect, and what you would do next. A repository with a good README reads as the work of someone who communicates; a repository whose README says only "install and run" reads as someone who does not. The skeleton below is a reliable structure to fill in.

# Project: Document QA take-home

## What this is
One-paragraph summary: what the system does and the slice I chose to build.

## How to run
    cp .env.example .env      # add your ANTHROPIC_API_KEY
    pip install -r requirements.txt
    python -m my_app.ingest data/   # build the index
    python -m my_app.serve          # start the API on :8000
    python eval_harness.py          # run the golden-set eval

## Architecture (one diagram or 5 bullet points)
- Ingestion: chunking strategy and why
- Retrieval: hybrid (BM25 + vectors) + re-ranker, top-k = 5
- Generation: grounded answer with citations to source chunks
- Eval: 15-case golden set + LLM-judge, current score below

## Eval results
Golden-set accuracy: 12/15 = 80%. The 3 failures and why (see eval_log.md).

## Decision log (the trade-offs)
- Chose hybrid retrieval over pure vectors because the corpus has exact-match IDs.
- Capped context at 5 chunks to keep cost ~= $0.004/query; quality plateaued there.
- Skipped streaming and auth — out of scope for the slice; see "next steps".

## Known limitations
- No caching yet; cold queries pay full input cost.
- Judge can over-reward verbose answers; spot-checked 5 verdicts by hand.

## What I would do with more time
- Add prompt caching on the system prompt + retrieved context (~10x input cost cut).
- Expand the golden set to 50 cases and wire the eval into CI.
- Add observability: per-query latency and token counts.

That structure does the reviewer's job for them. It surfaces the eval score, names the trade-offs, owns the limitations, and shows forward thinking — the four things the communication dimension of the rubric is grading. Note how it doubles as a decision log: the "Decision log" and "Known limitations" sections are where you demonstrate the judgement that no amount of clean code can convey on its own.

Here is the strategic leap. Everything that makes a take-home strong — a grounded retrieval slice, an eval with a real number, a README that names trade-offs — is exactly what a public portfolio project should contain. If you have already shipped one or two small, production-shaped projects and written them up publicly, a reviewer can pre-vet you before they even send the assignment. A retrieval system over a real corpus and an agent with tool-use plus an eval, each with a clear write-up, signal the same production maturity the take-home is designed to extract. That is why a public Builder profile is the strongest "pre-take-home" signal you can carry: it lets the people hiring see the work before they ask for more of it. Our guides on portfolio projects that get you hired and landing your first AI engineering role through open-source proof-of-work walk through exactly which projects to ship and how to write them up.

Common pitfalls, and the dual-market nuance

Most failed take-homes fail in the same predictable ways. The callout below is the list to check yourself against before you press submit. Every item maps to a dimension of the rubric, and every one is avoidable in the last hour of the exercise.

The rubric travels across both markets unchanged, but the emphasis shifts, and it pays to cover both ends. Take a UK candidate applying to a London fintech and an Indian candidate applying to a Bangalore SaaS company: both are scored on the same seven dimensions, yet the weighting tilts. UK take-homes, especially in regulated sectors like fintech, tend to probe data governance harder — how you handle personally identifiable information, where data is stored, whether you logged anything you should not have, and how you would meet retention and residency requirements. An Indian startup, often shipping fast against a competitive market, frequently weights a clean, working slice and shipping speed more heavily, valuing a candidate who can get a reliable system to production quickly. Neither emphasis is universal and neither is a stereotype to lean on — plenty of Indian enterprises care deeply about governance and plenty of UK startups prize speed. The safe move is to cover both: ship a fast, working slice and add a short note on how you would handle data governance and PII. Do both and your submission reads well regardless of which side of the market is grading it.

Your next steps

The take-home rewards a specific kind of engineer: one who frames before building, ships a working slice, measures their own quality, handles failure gracefully, watches the cost, and writes it all down honestly. None of that is exotic, and all of it is learnable by building. The fastest way to internalise the rubric is to do, this week, exactly what a take-home asks for — but in public.

Build one small retrieval system over a real corpus you care about. Wire in the eval harness from this article, write it up with the README skeleton, and publish it. Then build a second project — an agent with two or three real tools, an eval, and a human checkpoint. Two production-shaped projects, each with a measured eval and an honest write-up, are worth more than any number of certificates, and they convert directly: the same artefacts that win the take-home are the ones that get you found before the take-home is ever sent. Put them on a public Builder profile, and you turn private proof-of-work into something the people hiring can discover on their own.