Pass the LLM System Design Interview: The Round Nobody Preps For

The round that replaced the whiteboard

For a decade, the system design interview had a stable script. Design a URL shortener. Design a news feed. Estimate queries per second, choose a database, add a cache, draw the load balancer, discuss consistency. Candidates prepared from the same handful of books and courses, and interviewers graded against the same handful of trade-offs.

That script is being retired, one loop at a time. Teams building on large language models — which in 2026 means a large share of the teams hiring engineers at all — have replaced or supplemented the classic round with an open design conversation about the systems they actually run: retrieval pipelines, agent architectures, evaluation harnesses and inference cost. The 2026 interview guides now circulating make the shift explicit: preparation resources such as the awesome-llm-system-design collection and the System Design Handbook's LLM guide are organised around end-to-end questions like "design a support agent" and "make our serving cheaper", not around sharding strategies.

Three things follow for anyone interviewing in India or the UK this year.

• The preparation mismatch is the trap. Candidates who prepped classic system design give confident, detailed answers to a question that was not asked. Interviewers read this instantly.
• The round rewards production scars, not vocabulary. Anyone can name a vector database. Far fewer can explain what they would evaluate, what breaks and what a request costs.
• It is learnable in weeks. The question space is narrower than classic system design ever was. One repeatable framework, applied to a handful of prompt types, covers most loops. Our four-week system-design preparation playbook covers the study plan; this guide goes deeper on the round itself — the worked examples and the numbers.

What the round actually is in 2026

The format is consistent across markets: a 35–60 minute open conversation, usually one interviewer, sometimes a shared whiteboard or editor. The prompt is deliberately underspecified — "design an LLM-powered search feature for our product", "design an agent that handles refund requests", "our RAG answers are bad, walk me through how you would fix them". There is no single correct architecture; the interviewer is watching how you navigate.

Four themes recur, and a strong candidate touches all four without being prompted.

Retrieval design. Where does grounding data come from, how is it chunked and indexed, what mix of lexical and vector search do you use, and how do you know retrieval is returning the right material rather than plausible-looking noise. This is where most real-world quality problems live, so it is where interviewers spend the most time.

Agent architecture. When the prompt involves actions rather than answers: how the task is decomposed, what tools the model may call, where state lives, and — the part that separates practitioners from readers — what stops the agent from doing something expensive or destructive.

Evaluation. How you define "good", what your golden set looks like, how you handle non-determinism, and what gates a release. An answer with no evaluation plan is an automatic downgrade in 2026, the way an answer with no capacity estimate was a decade ago.

Cost. Per-request economics under real prices, and the levers that change them: caching, routing, truncation, batching. This is the theme classic prep leaves candidates least equipped for, and we give it its own section below.

A repeatable answer framework

Every strong answer we have seen follows the same five-part arc. Learn it as a sequence you can run on any prompt, out loud, under time pressure.

1. Requirements — but the LLM-specific ones

Before drawing anything, establish the constraints that actually shape an LLM system: What does a wrong answer cost — is this a shopping suggestion or a medical summary? What latency budget do you have — interactive chat tolerates two seconds; type-ahead search does not tolerate two hundred milliseconds of model call. What data may the model see — customer PII, licensed content, cross-tenant leakage? What volume — because a design that is elegant at a thousand requests a day is bankrupting at a million. Two minutes here earns you the right to make opinionated choices later.

2. The retrieval / generation split

Decide, explicitly and aloud, what the model is for. Retrieval finds the material; generation shapes it into an answer. Most quality failures blamed on the model are retrieval failures, so put your engineering effort where the failures are: chunking strategy, hybrid lexical-plus-vector search, a reranking stage, and a threshold below which the system says "I don't have that" instead of hallucinating. Then keep generation boring — a well-constrained prompt over the retrieved context, with citations.

3. The eval plan

Name the golden set (real queries with verified answers, including a deliberate tail of awkward ones), the metrics (retrieval hit rate separately from answer quality, so you can localise failures), the judge (human labels for the seed set, an LLM judge for scale — with its biases named), and the gate (what number must hold before a change ships). Saying "we would A/B test it" is not an eval plan; it is a way of discovering failures in production.

4. The cost model

Sketch cost per request for the naive design, then show the two levers every interviewer expects: prompt caching for the stable prefix, and routing easy requests to a smaller model. Numbers in the section below.

5. Failure modes

Close by attacking your own design: retrieval misses, prompt injection through retrieved documents, the model output failing to parse, the provider having an outage, costs spiking under a retry storm. For each, name the containment — thresholds, output validation, timeouts and circuit breakers, spend alarms. Candidates who volunteer failure modes before being asked are the ones who have run these systems against real users; interviewers know it, and our guide to taking an agent from pilot to production covers the same territory from the shipping side.

Worked example 1 — "Design an LLM-powered search feature"

The most common prompt in current loops, so here is the arc applied end to end.

Requirements. Product search over the company's own content — documentation, help articles, product data. Interactive latency: aim under 1.5 seconds end to end. Wrong answers are embarrassing but not dangerous, so we can accept a small error rate in exchange for coverage — but we must never invent product facts like prices. Volume assumption: one million queries a month, stated aloud so the cost section has a denominator.

Retrieval. Ingest content into chunks of roughly 300–500 tokens split on semantic boundaries (headings, paragraphs), each carrying source metadata. Index twice: a lexical index (BM25) because product names and error codes are exact-match problems, and a vector index for paraphrased intent. At query time, run both and merge with reciprocal rank fusion, then pass the top candidates through a cross-encoder reranker — the single cheapest quality win in most RAG stacks. The generation stage sees only the top handful of reranked chunks.

The path, as pseudo-code you can write on a whiteboard:

def answer(query, user_ctx):
    q = rewrite_query(query, user_ctx)        # small model; stable prompt, cached
    candidates = hybrid_search(q, k=100)      # BM25 + vector, merged with RRF
    top = rerank(q, candidates, k=8)          # cross-encoder over 100 -> 8
    if top.max_score < RELEVANCE_THRESHOLD:
        return no_answer_response(query)      # say "not found"; never guess
    prompt = build_prompt(q, top, cite=True)  # constrained template + citations
    resp = llm.generate(prompt, max_tokens=600, timeout_s=8)
    return validate_and_attach_citations(resp, top)

Trace it aloud for the interviewer: the query is rewritten once by a cheap model (spelling, context from the user's session), retrieval fans out to a hundred candidates because recall is cheap at that stage, the reranker concentrates precision into eight chunks because generation context is expensive, and the threshold guard is the line that prevents the classic RAG failure — a confident answer synthesised from irrelevant chunks. Validation checks the output parses and that every factual claim carries a citation into the retrieved set.

Evals. A golden set of 200 real queries with labelled relevant documents. Track retrieval hit rate at k separately from end-to-end answer quality, because when quality drops you need to know which stage regressed. Gate releases on both.

Failure modes. Threshold too low → hallucinated answers; too high → "not found" on answerable queries — tune against the golden set. Reranker adds latency → run it only when lexical and vector rankings disagree. Provider outage → degrade to plain lexical search results rather than an error page.

Worked example 2 — the agentic design question

The second prompt family: "design an agent that does X" — handles refunds, triages support tickets, files expense reports. The architecture interviewers expect has four named components, and they will probe the safety edges of each.

Planner. The model turn that decomposes the task into steps. Keep it re-entrant: the plan is data, stored outside the model's context, so the agent can resume after a failure rather than starting over.

Tool registry. The set of actions the agent may take, each with a typed schema, a description and — the part candidates forget — an allowlist per task type. A refund agent has no business calling the user-deletion tool. Inputs are validated against the schema before execution, not after.

Memory store. Working state for the current task (steps completed, tool results, running budget) plus, optionally, longer-lived memory across sessions. Distinguish the two aloud; conflating them is a common tell.

Executor. The loop that runs tool calls with timeouts, records results and decides whether to continue, retry or stop. This is where the safety constraints live, and it is where interviewers spend their probing time.

def run_agent(task, allowlist, budget):
    plan = planner.decompose(task)
    state = TaskMemory(task)
    for step in plan:
        if budget.exhausted():
            return state.partial_result(reason="budget")
        tool = registry.select(step, allowlist)   # None -> unsupported step
        if tool is None:
            return state.escalate_to_human(step)
        args = tool.schema.validate(step.args)    # reject bad input pre-call
        result = executor.run(tool, args, timeout_s=step.timeout)
        state.record(step, result)
        if result.is_error and not step.retryable:
            return state.partial_result(reason="tool_failure")
    return state.final_answer()

Trace: the budget check runs before every step, so a looping agent burns bounded money, not unbounded money. Tool selection is constrained by the allowlist, so a prompt-injected instruction to call an unlisted tool dead-ends into human escalation. Schema validation rejects malformed arguments before anything executes. A non-retryable tool failure returns a partial result with a reason rather than pressing on with corrupted state.

The probes to expect, and the answers that land: What can it do without approval? — reversible actions under a value threshold; anything above it, or irreversible, goes to a human queue, and the boundary is enforced in the tool layer and credentials, not in the prompt. What if a tool returns confidently wrong data? — validate at the boundary, cross-check where possible, and admit the class of wrongness you cannot detect, then bound its blast radius. How do you stop it? — a kill switch that revokes the agent's credentials, not a polite instruction in the system prompt.

The cost-estimation question

At some point — often as the closing question — the interviewer asks what your design costs. This is the question candidates bomb most often, and the fix is to carry a small worked table in your head. The one below uses Anthropic's published list prices as of August 2026 (Claude Opus 5 at $5 per million input tokens and $25 per million output; Claude Haiku 4.5 at $1 and $5; cached input reads at roughly a tenth of the fresh price — verify current figures at anthropic.com/pricing). Other providers publish comparable tiers; the shape of the argument is what matters, and you should date-stamp whatever prices you quote, exactly as we have.

Assume the search feature above: roughly 2,000 input tokens per request (system prompt plus retrieved chunks plus query) and 500 output tokens, at one million requests a month.

Serving path	Input cost / request	Output cost / request	Total / request	Per 1M requests / month
Naive frontier call (Opus-class, $5 / $25 per MTok)	$0.0100	$0.0125	~$0.0225	~$22,500
Frontier + prompt caching (1,800 of 2,000 input tokens cached at ~$0.50 per MTok)	$0.0019	$0.0125	~$0.0144	~$14,400
Routed to small model (Haiku-class, $1 / $5 per MTok)	$0.0020	$0.0025	~$0.0045	~$4,500
Blended: 80% routed small, 20% frontier cached	—	—	~$0.0065	~$6,500

The numbers themselves are less important than the reasoning you attach to them. Caching attacks input cost only, so it matters most when the prompt is dominated by a stable prefix — which a RAG system's system-prompt-plus-template usually is. Routing attacks both sides, but it needs a router (a classifier or a confidence signal) and an eval that proves the small model is good enough on the easy traffic. And output tokens are five times the price of input on both tiers, which is why max_tokens discipline and terse answer formats are cost features, not style preferences. A candidate who says "I would cap output, cache the prefix, route the easy 80% down a tier, and re-check quality on the routed traffic" has answered the question completely.

India vs the UK: bands, loops and who is asking what

The round itself is remarkably uniform across the two markets — the same four themes, the same framework works in Bengaluru and in London. What differs is who runs it, how many stages surround it, and what the offer at the end looks like.

India. The 2026 salary guides converge on wide bands driven mainly by employer type. Mid-level AI engineers sit broadly in the ₹12–38 LPA range, with the spread explained less by skill than by segment: services firms anchor the bottom of the band, while GCCs and AI-native product companies pay roughly 40–70% more at equivalent experience (see for instance the cross-market comparison at kaam.work). Engineers with demonstrable GenAI and LLM depth command a premium band of roughly ₹20–70 LPA in 2026 guides, with senior LLM specialists at product companies above that. Loop shape follows the same split: services firms tend to run shorter, more scripted rounds where the LLM design question may be one station among many; GCCs and product companies run the full open-conversation version described here, often with a follow-up deep-dive on whatever you claimed to have built.

The UK. The advertised-salary data is more centralised. ITJobsWatch's most recent published figures put the median advertised salary for a Generative AI Engineer at about £71,250 — notably down around 11% year on year from £80,000, a reminder that title-level medians move — while the more engineering-weighted "Gen AI Software Engineer" title carries a median around £105,000. UK loops at product companies and scale-ups mirror the US pattern: one dedicated LLM system design round plus a practical round. At banks, consultancies and public-sector-adjacent employers, expect the same design conversation with a heavier governance overlay — data residency, auditability and model-risk questions folded into the failure-modes section. Answering those well is a genuine differentiator, and our guide to vetting an AI team before you join doubles as a map of what those organisations worry about.

Two caveats worth stating plainly. Salary figures in careers articles — including this one — age quickly and blend inconsistent samples; treat every number here as indicative, and verify against live sources such as ITJobsWatch for the UK and current India salary guides before you negotiate. And in both markets, the interview loop for senior and staff-level roles adds a second dimension this article does not cover — organisational scope — for which see the senior-to-staff promotion guide.

Conclusion — proof of work is the interview shortcut

Everything above is learnable from the outside. But the candidates who pass this round most comfortably are not the ones with the best-memorised framework; they are the ones for whom every probe lands on something they have actually done. "How would you evaluate it?" is a different question when you can answer "here is the eval harness I published, and here is what surprised me". The interview becomes a walkthrough of your own work — which is the easiest interview there is.

That suggests a preparation plan that doubles as a career asset. Build one small system from this article for real — the search feature with its golden set, or the agent with its tool registry and kill switch. Measure it honestly, including the numbers that disappointed you. Write it up. Then put it somewhere the people running these loops actually look.

That last step is where a verified public profile earns its place. Hiring teams in both markets increasingly shortlist from public, project-structured profiles before a CV is ever opened, because a linkable artefact converts claims into evidence. A Verified Builder profile on AI Tech Connect is built for exactly this: your systems, your write-ups and your numbers, indexed and browsable by the teams hiring across India and the UK — including the freshly funded teams hiring right now. Early profiles carry the Founding Builder badge, and those spots are limited by design. If your proof of work is behind an employer's walls, our guide to building proof of work under NDA shows how to publish the method without the material.

The round nobody preps for is, in the end, the round that is easiest to prepare for honestly: build one real thing, understand why it behaves the way it does, and make the work findable. The framework gets you through the door; the artefact gets you the offer.