The Agent Reliability Engineer: A New AI Role and How to Get Hired

What you need to know

The cost of building an agent that works once has collapsed. A competent engineer can wire a model to a handful of tools, add a loop, and have something demonstrable inside a day. That capability is no longer scarce, and pricing it as though it were is the most common career mistake in this field right now.

What remains difficult is the second half: making that agent behave acceptably on the ten-thousandth request, when a tool is timing out, when a user has phrased something nobody anticipated, when the provider has shifted the default model version underneath you, and when the agent holds credentials that can move money. Somebody has to own that, and increasingly that somebody has a job title of their own. Four things follow.

• The role is real but the title is not settled. Search on responsibilities rather than titles or you will miss most of the openings.
• The core skill is systems discipline, not modelling. Much of the work is recognisable to anyone who has run a production service, which is why the entry route is unusually open.
• It is hired on demonstrated artefacts. Reliability claims are cheap and unverifiable on a CV. A public eval harness with a regression report is neither.
• Demand is visible in India and the UK, for different reasons. One market is growing broadly, the other is splitting sharply. Both favour the same evidence.

What the role actually is

Strip away the naming and the job comes down to six recurring responsibilities. None is exotic. All are things a team discovers it needs only after an agent has embarrassed it in front of a customer.

Owning the eval suite and gating deploys on it. The anchor duty. Somebody decides what "correct" means for a system whose output is open-ended, encodes it into a runnable suite, keeps it honest as the product changes, and has the institutional standing to block a release when the numbers move the wrong way. The last part is the hard part. An eval suite everyone can override is decoration.

Building and reading traces. When an agent does something inexplicable, the only recoverable evidence is the trace: the model calls, tool invocations, retrieved context and intermediate reasoning that produced the outcome. Emitting that in a usable structure is an engineering job in itself, and reading it fluently takes months. The mechanics are covered in our guide to instrumenting agents with OpenTelemetry, and the emerging conventions live in the OpenTelemetry generative AI semantic conventions.

Defining guardrails and kill switches. What is the agent permitted to do, where is that enforced, and how fast can it be stopped. A guardrail written into a system prompt is a request. One enforced by a scoped credential, a tool-layer allow-list or an egress rule is a control. That distinction is how interviewers separate candidates who have operated an agent from those who have built one.

Running incident response. On-call for a non-deterministic system is unlike on-call for a web service. You often cannot reproduce the failure, the fix is frequently a prompt or context change rather than a code change, and the blast radius may already have propagated downstream.

Managing cost and rate limits. Agents fail in an expensive direction. A retry loop that would cost nothing in a conventional service can burn a month's budget in an afternoon when each attempt is a model call over a long context. Somebody owns spend per successful task, concurrency ceilings, caching and the behaviour when a provider throttles you.

Owning model-upgrade rollouts. The responsibility teams underestimate most. An upgrade is a behaviour change across every prompt you own, on the provider's schedule rather than yours. Shadow evaluation, staged rollout, pinned versions and a documented deprecation path separate a routine upgrade from a week of firefighting.

A typical week, honestly

The split below is an illustrative sketch of how time tends to distribute once an agent is live, not survey data. Use it to sanity-check whether a role you are considering is really this one, or a research post with reliability bolted on.

An illustrative week for an engineer owning a live agent. Proportions are a working sketch, not measured data.

Activity	Rough share	What it actually looks like
Eval maintenance and release gating	About a quarter	Adding cases from last week's failures, re-baselining, arguing about a threshold, blocking or clearing a release
Trace review and debugging	About a fifth	Reading traces for reported oddities, finding the tool call that returned malformed data, writing the reproduction
Guardrails, permissions and tool design	About a fifth	Tightening a credential scope, adding a confirmation step, redesigning a tool signature the model keeps misusing
Incident response and on-call	About a tenth	Paging, containment, the post-incident write-up, the eval case that stops it recurring
Cost, capacity and rate limits	About a tenth	Chasing a spend spike, adding caching, setting concurrency ceilings, reporting cost per successful task
Model upgrades and rollouts	About a tenth	Shadow-running a new model version against the suite, staging the rollout, writing the migration note
Everything else	The remainder	Design reviews, onboarding a team onto the harness, answering the risk function's questions

How it differs from the roles next to it

Boundaries determine which team you sit in, what you are measured on and which interview loop you face. Vague answers cost offers: if you cannot articulate why this is not simply MLOps, an interviewer will suspect you have not done it.

Where the agent reliability engineer sits relative to adjacent roles.

Role	Primary artefact	Main failure it prevents	Core skill	Typical background
Agent reliability engineer	The eval suite, the guardrail layer and the rollout process	A live agent behaving unacceptably against real users, money or permissions	Reasoning about non-deterministic systems under production constraints	Backend, SRE or platform engineering, plus deliberate LLM study
ML engineer	A trained or fine-tuned model and its training pipeline	A model that is inaccurate on the task it was built for	Modelling, feature work, offline metric design	Applied machine learning, data science, research engineering
MLOps or platform engineer	The serving stack, registry and deployment pipeline	The system being un-deployable, un-reproducible or un-scalable	Infrastructure automation and pipeline design	DevOps, cloud infrastructure, data engineering
Site reliability engineer	SLOs, alerting and the incident process	The service being unavailable or too slow	Availability engineering and operational rigour	Systems and operations engineering
Eval engineer	Benchmarks, golden sets and judging methodology	Measuring quality incorrectly, or not at all	Measurement design and statistical honesty	Research engineering, data quality, applied science

The clean way to hold the distinction: an SRE keeps the service up, an eval engineer decides whether the output is good, and an agent reliability engineer is accountable for the system doing something acceptable when it is neither down nor obviously wrong. That third category — plausible, confident, permitted and wrong — barely exists in conventional software, which is why it needs an owner.

The skill map

Four blocks. Read them not as a syllabus but as a self-assessment: score yourself against the "enough" column, because the gap you find is your study plan.

Block A — systems fundamentals

Distributed systems behaviour, retry and back-off strategy, idempotency, timeouts, queueing, observability, capacity planning. Nothing here is AI-specific. What changes is that agents violate the assumptions harder: calls are slow, expensive and non-deterministic, so a naive retry is a different animal from a naive retry against a database.

Block B — LLM-specific engineering

Prompt and context engineering as a discipline rather than a knack. Tool design — the signatures, descriptions and error messages you expose materially change how often a model misuses them. Retrieval basics, enough to reason about why the wrong chunk came back. And an empirical sense of how models behave under load and at long context.

Block C — evaluation

Golden set construction, the mechanics and biases of using a model as a judge, sampling repeated runs to characterise variance, and wiring it into continuous integration so a regression is caught by a pipeline rather than a customer. The trap is building an eval the agent can quietly satisfy without doing the work; our guide to building evals your agent cannot game covers that failure mode.

Block D — safety and governance

Prompt injection, including the indirect variety arriving through retrieved documents, least-privilege credentials, blast-radius limitation, kill switches and auditability. The OWASP Top 10 for LLM applications is the working checklist; the NIST AI Risk Management Framework is the vocabulary you need when the risk function asks you to justify a design. Containment patterns are in agents that fail safe.

Self-assessment: what "enough" looks like per block, and where a backend or SRE engineer already stands.

Block	What "enough" looks like	Backend or SRE starting position
A — systems fundamentals	You can explain why a particular retry policy is safe for this operation and unsafe for that one, and you instrument before you guess	Usually already there. This is the block that makes the transition viable
B — LLM-specific	You can look at a failing agent run and form a testable hypothesis about whether the cause is the prompt, the context, the tool schema or the model	The main gap. Closable in weeks of deliberate practice on a real agent, not months
C — evaluation	You can build a golden set from real failures, defend a pass threshold, and name at least two biases in your own judge	Partly there — regression testing instincts transfer; measurement design under non-determinism does not
D — safety and governance	You can enumerate every action the agent can take, say where each boundary is enforced, and describe what stops it in under a minute	Strong on least privilege and network controls; new on injection and model-specific attack surface

Three proof-of-work projects that make the claim checkable

Reliability is the hardest capability to assert credibly on a CV. Everybody writes "built production-grade AI systems"; almost nobody can link to evidence. That asymmetry is available to anyone willing to spend a few weekends on it. All three projects below use open-source agents and public models deliberately, so nothing you publish belongs to an employer.

Project one — a public eval harness for an open-source agent

What to build. Pick a well-known open-source agent. Define a suite of thirty to eighty cases with unambiguous success criteria, including a deliberate tail of awkward ones: ambiguous instructions, tools returning errors, inputs that invite an unsafe action. Run every case multiple times to measure variance rather than reporting a single lucky pass.

What to measure. Pass rate at a fixed number of attempts, variance across repeats, failure taxonomy by category, tokens and wall-clock per case, and the same suite against several model versions so you have a genuine regression report rather than a snapshot.

The public artefact. A repository plus a report a hiring manager can read in five minutes: what you measured, what surprised you, what you would not conclude from the data. The restraint is part of the signal.

What it proves. That you can define correctness for an open-ended system, handle non-determinism honestly, and turn a measurement into a release decision. A worked walkthrough lives in shipping a public agent eval harness.

The gate itself should be boring and mechanical, in whatever CI system you use:

name: agent-eval-gate
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Run eval suite (5 repeats per case)
        run: python -m evals.run --suite golden --repeats 5 --out results.json
      - name: Gate on regression vs baseline
        run: python -m evals.gate --results results.json --baseline baselines/main.json

And the assertion that does the actual gating, kept deliberately explicit so that a reviewer can argue with the thresholds rather than guess at them:

def gate(results, baseline):
    # Absolute floor: never ship below the agreed pass bar.
    assert results.pass_at_5 >= 0.90, f"pass@5 {results.pass_at_5:.3f} below floor 0.90"

    # Relative floor: no silent drift against the last green build.
    drop = baseline.pass_at_5 - results.pass_at_5
    assert drop <= 0.02, f"pass@5 regressed by {drop:.3f} vs baseline"

    # Severity gate: some failures are not tradeable against average quality.
    assert results.count("unsafe_action") == 0, "unsafe action observed; blocking release"

    # Cost gate: quality gains bought with runaway spend are not gains.
    assert results.tokens_per_success <= baseline.tokens_per_success * 1.15, "cost regression"

Four gates, four kinds of failure. Explaining why each exists, and what you do when they conflict, is most of an evals interview round.

Project two — a fault-injection study

What to build. Take an agent and systematically break what it depends on. Make a tool time out. Make it return malformed JSON. Make it return plausible but wrong data. Revoke a credential mid-run. Truncate the context. Then document, for each injected fault, what the agent did — and what you changed so that it failed safely instead.

What to measure. For each fault class: detection rate, whether the agent degraded gracefully or confidently produced nonsense, whether the failure was visible in the trace, and time to containment. The most valuable column records which failures were silent, because silent failures are the ones that reach customers.

The public artefact. A fault catalogue with before-and-after behaviour and the specific mitigation for each. It reads as an engineering document rather than a portfolio piece, which is exactly why it lands. The methodology is covered in pass@k, perturbation and fault injection for agent reliability.

What it proves. That you think in failure modes rather than happy paths — the most reliable predictor of whether someone has run a production system. The lineage here is ordinary reliability engineering, and the Google SRE books remain the best free grounding in it.

Project three — a cost-and-latency teardown

What to build. Take a real agent workload, measure where the money and the milliseconds actually go, then apply three or four specific interventions and measure again. Candidates: caching a stable prefix, routing easy cases to a smaller model, cutting a redundant tool call, tightening retrieval, or replacing a model call with ordinary code where the task was never a language problem.

What to measure. Cost and latency per successful task rather than per call — retries and failures are the whole point — broken down by stage, with a before and after per intervention and an explicit note on whether quality moved. An intervention that halves cost while quietly degrading the pass rate is not an improvement, and saying so is the mark of an honest engineer.

The public artefact. A teardown post with a table of interventions, their measured effect on cost, latency and quality, and a recommendation about which ones generalise.

What it proves. That you can hold an engineering trade-off and a commercial one in the same hand. This is the artefact that gets you taken seriously by people who control budgets — often a more consequential audience than the interview panel.

What the market looks like in India and the UK

Two markets, two different shapes, one common conclusion.

In India, the direction is broad expansion. LinkedIn's AI Labor Market Report 2026 — the title as LinkedIn publishes it — reported that hiring for AI talent in India grew 59.5% year on year, and — the more interesting detail for anyone not living in Bengaluru — that growth is spreading beyond the traditional metros, with Hyderabad and Vijayawada among the fastest-growing locations for AI engineering hiring (as reported in 2026; the underlying research is published through the LinkedIn Economic Graph). Geographic spread matters here specifically, because reliability work is done against a running system rather than a research agenda and travels comfortably to distributed teams.

In the UK, the shape is a split rather than a boom. Indeed's Hiring Lab has described a two-speed labour market within knowledge work itself: broadly weak in volume, but with a fast-growing premium attached to people who can demonstrate genuine capability with AI tools, with postings that reference AI growing even as overall postings in the same categories decline (Indeed Hiring Lab, 2026). The same analysis records graduate postings at their lowest level for this time of year since the pandemic. A two-speed market is not bad news in itself — but it does mean the faster lane is entered on evidence rather than on stated interest, and it is very hard to enter with none.

What that implies for a UK candidate is fairly specific. The organisations most likely to have an agent live against real money or real records — financial services, health, the public sector and the larger consultancies — are also the ones where an agent doing something it should not is an auditable event rather than merely an embarrassing one. That changes which artefact lands first. A fault catalogue with a named mitigation per fault class reads to those buyers as an assurance document rather than a portfolio piece, so it is usually the one to lead with; write the mitigation column as though a risk reviewer will read it, because one plausibly will, and be ready to say which of your three artefacts you would put in front of an auditor. India's product and platform teams are more often reached through the eval harness and the cost teardown, where throughput and unit economics are the live argument. Same three artefacts, different order of presentation — which is why building all three is worth the extra fortnight.

On pay, resist any article that quotes you a band, including this one. What can be said honestly is the shape: reliability and platform work sits on infrastructure bands rather than research bands in both markets. The ceiling is typically lower than a frontier research post; the floor is higher, the employer count far larger, and the skills transfer between companies instead of being welded to one lab's stack. For figures, use our India and UK pay benchmarking guide and verify against live sources before negotiating.

The role is also unusually portable across the two markets: the artefacts are public, the tooling is open, and the failure modes are the same whether the agent serves a bank in London or a logistics platform in Chennai.

The interview loop and how to prepare

Loops vary by employer, but the shape has settled faster than the title has. Four substantive rounds recur, usually in this order.

The systems-design round is framed as "design an agent that does X safely" rather than the classic scaling question. The interviewer listens for whether you ask about permissions, failure handling and rollback before drawing boxes. Candidates who dive into architecture without asking what the agent may touch mark themselves in the first two minutes.

The debugging round puts a trace in front of you and asks what happened. It cannot be crammed, which is the reason to build projects one and two rather than read about them.

The evals round asks you to design a measurement for a task that resists measurement. Expect to be pushed on judge bias, how many repeats you need, and what your threshold should be and why.

The judgement round is about when not to ship. It often decides senior offers, and it rewards candidates who hold a commercial and an engineering view at once rather than retreating into "it depends".

Example questions and what a strong answer contains.

Question	What a strong answer contains
Design an agent that issues customer refunds. What can it do on its own?	An explicit permission boundary with a value threshold, enforcement at the credential and tool layer rather than the prompt, an approval path above the threshold, and an audit trail that survives the agent being wrong
Your pass rate dropped three points overnight and no code changed. What now?	Check whether the provider's model version moved, diff traces between yesterday and today, isolate whether the regression is uniform or concentrated in one case category, and state what you would freeze while investigating
How many times do you run each eval case, and why?	Enough repeats to characterise variance rather than sample luck, a stated reason for the number, and an acknowledgement of the cost trade-off — plus what you do when variance itself is the finding
Your LLM judge agrees with human raters 90% of the time. Is that good enough?	Where the disagreements cluster, whether the judge is biased toward length, formatting or its own family of models, and what you gate on directly rather than through the judge
A tool returns confidently wrong data. How does the agent find out?	Validation at the tool boundary, cross-checks or plausibility bounds where they exist, and honest acknowledgement of the class of wrongness the agent cannot detect — plus what limits the damage in that case
Walk me through the last time you stopped a release.	The specific signal, who you had to convince, what it cost, and — the part that distinguishes a real story — whether you were subsequently proved right or wrong, told either way
The agent works but costs four times the projection. What do you do first?	Measure before optimising, attribute cost by stage, separate retries from first-attempt spend, and name the intervention with the best ratio of saving to quality risk
When would you tell the business not to ship an agent at all?	A clear articulation of when the failure mode is unbounded, unrecoverable or undetectable, and what cheaper non-agent design you would propose instead

The preparation plan is short because the work is the preparation. Four weeks on project one, two on project two, two writing both up properly. Read traces daily. Rehearse the eight answers above out loud against a timer. Then work through the AI engineer interview question clusters, which covers the rounds this role shares with adjacent ones.

Positioning yourself so the work is findable

The structural pattern in how this role gets filled: it leans on demonstrated artefacts and referrals more than on CV keyword matching, because "I make agents reliable" is unverifiable in the abstract and every applicant claims it. Hiring managers compensate by looking for evidence they can open, read and judge for themselves.

So the binding constraint for most capable engineers is not skill. It is that the work is not findable. A private repository, or a project described only in a PDF attachment, does not surface when somebody goes looking for a person who has done exactly this.

The fix is a public, indexable profile organised around projects rather than job titles — the eval harness, the fault catalogue and the teardown as linkable things with a sentence each about what they proved. That is what a Verified Builder profile on AI Tech Connect is for: project-structured, public, and browsed directly by people hiring for agent and platform work in India and the UK.

Profiles created early carry the Founding Builder badge, and those spots are limited by design. That is a plain statement of benefit and scarcity: the profile is worth having on its own terms and the badge is permanent for those who get in early. We make no claim about outcomes — a profile makes your work visible to people looking for it, which is a necessary condition for being found, not a guarantee of a role. If your projects are built, adding them takes about two minutes at /submit-my-profile/. If not, build project one first and come back. For framing that evidence so it survives a skim, see our guide to the proof-of-work portfolio.

Conclusion: a 30-day starting plan

Thirty days is enough to move from "interested in this role" to "has something to link". It is not enough to become senior in it.

1. Days 1–3. Score yourself against the four skill blocks. Write down the two weakest gaps. If you are a backend or SRE engineer, Block B is almost certainly one.
2. Days 4–10. Get an open-source agent running locally with full tracing on. Read fifty traces before writing a line of eval code. You are building the instinct, not the harness.
3. Days 11–20. Build the eval suite. Thirty cases is enough to start, with a deliberate tail of awkward ones. Run each five times. Wire the gate into CI so a regression blocks a pull request.
4. Days 21–25. Inject faults. Break the tools, revoke a credential mid-run, truncate the context. Record which failures were silent, then fix two so they fail loudly instead.
5. Days 26–29. Write it up. Two thousand words, one table, an honest limitations section, and a paragraph on what you would not conclude from your own data.
6. Day 30. Publish the report, link the repository, put both on a public, indexable profile. Then start applying by responsibility keywords rather than job titles.

If the current shape of the work holds, the people who own it in two years will not be the ones with the best models. They will be the ones who were willing to do the unglamorous half — the gates, the traces, the credential scopes, the write-up nobody asked for. That half is learnable, demonstrable, and remarkably few people bother to demonstrate it.