Build an AI Evals Portfolio: The Proof of Work Most Engineers Skip

What you need to know

• The demo is no longer the differentiator. A chatbot or RAG app that works on one screenshot is what almost every candidate now ships. It proves you can build. It does not prove you can tell whether what you built is any good.
• The proof of work that is genuinely rare is an evals harness. A real task, a small golden set, defined metrics, a runnable eval script, a regression gate in CI, and a short writeup of baseline → change → result. Almost nobody ships this. The few who do stand out instantly.
• It signals production maturity, not demo-building. The 2026 mental shift is from "model quality" to "system quality, including the model". Thinking in evals answers the question senior and staff rubrics actually ask: how do you know it is good, and how would you catch a regression?
• You do not need a big dataset. Thirty to fifty hand-curated examples beat none. Start with one high-signal metric, establish a baseline, change one thing, re-measure, and write up what moved. That narrative is half the value.

Here is a pattern you can watch repeat in any hiring round. A candidate opens their laptop, shares their screen, and shows a retrieval-augmented chatbot answering a question correctly. The demo works. The retrieval is reasonable. The interface is tidy. And then the hiring manager asks one quiet question that the whole interview turns on: "How do you know it works?" The honest answer, in the overwhelming majority of cases, is a pause, then some version of "well, I tried a few queries and it looked good." That pause is the gap. It is also, if you choose to, your opportunity.

The thesis of this piece is simple. The portfolio everyone builds proves you can assemble an LLM application. The portfolio almost nobody builds proves you can measure one — and measurement is the skill that production teams are short of, the skill that distinguishes a senior engineer from a confident junior, and the skill that is easiest to demonstrate with a small, self-contained piece of work that lives on your profile and your GitHub. This is the companion to our proof-of-work portfolio guide: that piece is about shipping things people can see; this one is about shipping the evidence that the thing actually works.

Why evals convert with hiring managers

To understand why an evals portfolio piece punches above its weight, it helps to see what has changed in how teams think. For a couple of years, the centre of gravity in AI work was model quality. Which model is smartest? Which scores highest on the leaderboard? That framing still matters, but in 2026 the conversation that actually happens inside product teams has moved one level up the stack — to system quality, including the model. The model is one component. The retrieval, the prompt, the parsing, the guardrails, the fallback behaviour and the monitoring are the rest. A team can swap in the cleverest model on the market and still ship something that quietly returns wrong answers, because the system around the model is where reliability lives or dies.

Data scientists and junior engineers tend to over-index on the model and under-index on the system. They will tell you the accuracy of a classifier to two decimal places but go quiet when you ask how they would notice if next week's prompt change made the answers worse. Demonstrating that you think in evals flips that. It tells a hiring manager that you treat an LLM feature the way you would treat any other piece of software: you have a notion of correct, you have a way to measure distance from correct, and you have a tripwire that fires when a change moves you in the wrong direction.

This maps directly onto how senior and staff interview rubrics are written. The questions that separate levels are rarely "can you build it?" — by the time someone is in the room, that is assumed. The differentiating questions are "how do you know it's good?" and "how would you catch a regression before a customer does?" A candidate who can answer those with a real artefact, rather than a hand-wave, is answering at a level above their job title. That is exactly the kind of signal a well-curated GitHub profile and a Verified Builder profile are meant to broadcast.

There is a market angle here too, and it cuts across both of our regions. Whether you are interviewing with a Bengaluru product team or a London scale-up, the roles that pay best are the ones where the company is trusting an AI system with something that matters — money, compliance, customer-facing decisions. Those teams are acutely aware that an unmeasured system is a liability, and they pay for people who reduce that liability. We have written separately about how to read the pay landscape in our AI engineer salary and pay benchmarks guide; the short version relevant here is that "I can measure and de-risk an AI system" is one of the clearest ways to move from the middle of a band toward the top of it.

What an evals portfolio piece actually contains

Let us make this concrete, because "build some evals" is the kind of advice that sounds actionable and is not. An evals portfolio piece is a small, self-contained repository that a stranger can clone, read in ten minutes, and run. It has a specific anatomy. Here is the component checklist — and crucially, the contrast with what most portfolios show instead.

What most portfolios show	What hiring managers actually want
A demo that works on a happy-path screenshot	A real task plus a dataset — even 30 to 50 hand-curated examples
"It gives good answers" (no definition of good)	A golden set with expected outputs you can compare against
A vibe check across a handful of queries	Defined metrics: exact-match or structured for deterministic fields, an LLM judge for fuzzy ones, task metrics like faithfulness for RAG
A screenshot of one good response	A runnable eval script and a results table anyone can reproduce
No way to know if a change made things worse	A regression gate: CI fails if a key metric drops below a threshold
A README that lists the tech stack	A short writeup: baseline → what you changed → what moved → trade-offs
"Works on my machine"	Reproducibility: pinned judge model and version, seeds, and rough cost per run

Read the right-hand column again, because every row is a decision you can make in an afternoon, and together they form something almost no one else in the applicant pool has. None of it requires a research budget or a proprietary dataset. It requires that you take one task you genuinely care about and treat measuring it as a first-class deliverable rather than an afterthought.

The seven components, briefly

A real task and dataset grounds everything; pick a task where you can recognise a good answer from a bad one. A golden set with expected outputs is your definition of correct, written down. Defined metrics turn "good" into a number — use exact-match or structured assertions for fields that have one right answer, an LLM judge for dimensions like helpfulness or tone, and task-specific metrics such as faithfulness and context precision for RAG. A runnable eval script and results table make the work reproducible. A regression gate in continuous integration is the part that signals seniority loudest, because it shows you think about preventing drift, not just observing it. The writeup is the narrative that ties it together. And reproducibility notes — a pinned judge model, fixed seeds, a cost estimate — are the difference between a credible artefact and a screenshot.

Build it in five steps

Here is the sequence I would follow, and the one I would advise a junior engineer in Bengaluru or a career-switcher in Manchester to follow when they ask how to stand out. Each step is small on its own. The compounding comes from doing all five.

Step 1 — Pick a task you actually care about

Motivation matters because evals work is unglamorous and you need to finish it. Choose something concrete: extracting structured fields from invoices, answering questions over a documentation set, classifying support tickets, summarising meeting notes. A RAG question-answering task is an excellent default because it has well-understood metrics and is exactly the kind of system teams are deploying right now. If you want the technical depth on scoring that specific case, our sibling piece on RAG evaluation with Ragas, faithfulness and context precision is the companion to read alongside this one — this guide is the career frame, that one is the metric mechanics.

Step 2 — Curate a small golden set

This is the step people skip, and it is the most important one. Sit down and hand-write 30 to 50 input examples with the output you would consider correct. Include easy cases, a few genuinely hard ones, and at least a couple of adversarial or edge cases — an ambiguous query, an out-of-scope question your system should refuse, an input with a tricky number or date. The act of curating the golden set forces you to define what "correct" even means for your task, which is most of the intellectual work. Store it in a plain, diff-friendly format such as JSON Lines so a reviewer can read it without running anything.

Step 3 — Write the eval, starting with one high-signal metric

Resist the urge to measure everything at once. Pick the single metric that best captures whether your system is doing its job — exact-match for an extraction task, faithfulness for a RAG task, an LLM-judge score for a generation task — and implement that one cleanly. Get the full loop running end to end with one metric before you add a second. For the deterministic fields, structured assertions are cheap and trustworthy. For the fuzzy ones, an LLM-as-judge is the standard tool; just be disciplined about it, which we will come back to. Our LLM evaluation suite guide and LLM-as-a-judge in production guide cover the implementation details that this career-focused piece deliberately keeps light.

Step 4 — Establish a baseline and iterate

Run the eval once and record the number. That is your baseline, and writing it down is what makes everything afterwards a story rather than a snapshot. Now change exactly one thing — swap the model, tighten the prompt, adjust the retrieval chunk size — and re-measure. The discipline of changing one variable at a time is itself a signal of engineering maturity. Keep a short log: what you changed, what the metric did, what it cost. This log becomes your writeup almost verbatim.

Step 5 — Publish the harness and the writeup, and link it

Put the repository on GitHub with a README that opens with your results table and your narrative, not your dependency list. Then make sure the people who hire can actually find it: link it from your Verified Builder profile and reference it in your application materials. Work that nobody can find may as well not exist — a point we make at length in why your AI projects are invisible and how to get discovered. The harness is the proof; the profile and the GitHub link are the distribution.

A concrete shape: the repo and a tiny eval sketch

To make this tangible, here is what a reviewer should see when they open your repository. The structure matters as much as the code — it tells the reader, before they have run anything, that you have thought about the whole loop.

ai-rag-evals/
├── data/
│   └── golden.jsonl          # 30–50 hand-curated {question, expected, context}
├── evals/
│   ├── run_eval.py           # loads golden set, runs system, scores, prints table
│   └── metrics.py            # exact_match(), faithfulness_judge(), etc.
├── .github/
│   └── workflows/
│       └── eval.yml          # CI: run evals, fail if a metric drops below threshold
├── results/
│   └── 2026-06-23.md         # results table + baseline-to-change-to-result writeup
└── README.md                 # opens with the results table, not the tech stack

And here is the heart of it: a small golden-set sample plus a sketch of an LLM-judge call. This is deliberately short — it is the kind of code that fits in a reviewer's head, which is exactly the point.

# data/golden.jsonl (one JSON object per line)
{"id": "q1", "question": "What is the refund window?", "expected": "14 days", "context_id": "policy-3"}
{"id": "q2", "question": "Do you ship to the UK?", "expected": "Yes, 3–5 working days", "context_id": "shipping-1"}
{"id": "q3", "question": "What's the CEO's home address?", "expected": "REFUSE: out of scope", "context_id": null}

# evals/metrics.py — a calibrated LLM-as-judge for the fuzzy dimension
JUDGE_MODEL = "claude-sonnet-4-5"   # pinned: version is part of the result

def faithfulness_judge(answer: str, context: str) -> int:
    prompt = (
        "Score 1 if every claim in the ANSWER is supported by the CONTEXT, "
        "else 0. Reply with only the digit.\n\n"
        f"CONTEXT:\n{context}\n\nANSWER:\n{answer}"
    )
    reply = call_judge(JUDGE_MODEL, prompt, temperature=0, seed=7)
    return int(reply.strip()[0])    # deterministic-ish: temp 0, fixed seed

Notice the small choices that signal care: the golden set includes a refusal case (q3), the judge model is pinned to a named version, temperature is zero and the seed is fixed for reproducibility, and the prompt asks for a single digit so the score is trivial to parse. None of this is hard. All of it reads as the work of someone who has run evals before — which is the impression you are trying to create.

The regression gate is the part that signals seniority

The continuous-integration step is worth dwelling on because it is the component that almost no junior portfolio includes. A regression gate is a few lines in eval.yml that run your eval on every push and fail the build if a key metric drops below a threshold you set from your baseline. Its presence tells a reviewer that you do not just measure quality once and move on — you wire measurement into the development loop so that a future change cannot silently make things worse. That is the exact behaviour production teams want, and seeing it in a personal project is rare enough to be memorable.

The pitfalls that quietly undermine an evals portfolio

An evals piece can go wrong in ways that are worse than having no evals at all, because they project false confidence. A sharp interviewer will spot each of these, so know them and avoid them.

The second pitfall is an LLM-judge you never calibrated. An LLM-as-judge is a useful instrument and an untrustworthy one until you have checked it against human judgement. Score 20 to 30 examples yourself, then measure how often the judge agrees with you, and report that agreement rate openly. A judge with a stated 88 per cent agreement is a credible signal; a judge with no calibration is a vanity metric wearing a lab coat.

The third is a golden set that is too easy, which produces a flattering "100 per cent pass" that means nothing more than that your test is trivial. If everything passes, your test is not discriminating between good and bad systems — add harder cases until something fails, because a test that can fail is the only kind that proves anything. The fourth, and the one that wastes the most potential, is no writeup. The harness without the narrative is half a deliverable. The story — here was my baseline, here is what I changed, here is what moved, and here is the trade-off I accepted — is where you show judgement, and judgement is what is being hired.

How to present it in an interview and on your profile

The artefact is necessary but not sufficient; how you frame it decides whether it lands. The mistake is to lead with the tooling — "I used Ragas and GitHub Actions and an LLM judge." Nobody hires for the tools. Lead instead with the question you were answering and the number that moved: "I wanted to know whether tightening my retrieval would improve answer faithfulness without blowing up latency. It took faithfulness from 0.71 to 0.86 and added about 90 milliseconds at p95, so I shipped it." That single sentence demonstrates a hypothesis, a measurement, a result and a trade-off — the entire arc of senior engineering judgement in fifteen seconds.

Then show the regression gate, because it answers the "how would you catch a regression?" question before it is even asked. And talk openly about the messy parts — the noise in your LLM judge, the cost of running the full eval, the cases your golden set still does not cover. Candour about limitations reads as seniority, not weakness; it is junior engineers who claim everything works. This is the same build-in-public muscle we describe in our build-in-public visibility guide and the same skill-stack signalling covered in the AI engineer skill stack that gets you hired — evals are simply the highest-leverage thing to build in public, because so few people do.

Finally, put it where it will be found. A Bengaluru product team and a London scale-up will not stumble across your repository by accident; they find people through profiles and search. Linking your evals harness from a Verified Builder profile, with a one-line description of the question it answered and the number that moved, turns a buried repo into a discoverable signal that the right people can act on.

Conclusion: ship the proof, not just the demo

The uncomfortable truth of the 2026 market is that the demo has been commoditised. Anyone can wire a model to a vector store and screenshot a good answer, and because anyone can, it no longer differentiates anyone. What remains scarce — and what production teams across India and the UK are actively short of — is people who can answer "how do you know it works, and how would you catch it breaking?" with an artefact instead of a shrug. Building one small evals portfolio piece, end to end, is the single highest-leverage move available to an AI engineer who wants to stand out, precisely because so few people make it.

So pick a task this week. Curate fifty golden examples. Write one metric, establish a baseline, change one thing, re-measure, and write up what moved. Wire in a regression gate. Then publish it and link it from your profile. Shipping evals is exactly the kind of proof-of-work an AI Tech Connect Verified Builder profile is built to showcase — and the engineers who show it, rather than tell it, are the ones who get the call.