How to Get Hired at an AI Infrastructure Company

What you need to know

• Infrastructure companies sell the substrate, not the feature. If the invoice is denominated in GPU-hours, tokens or requests rather than seats, the hiring rubric changes underneath you.
• The rubric is unusually stable. Prompting fashions turn over every few months; memory bandwidth, queueing theory and the cost of a cache miss do not. Skills learned here keep paying.
• App-layer proof of work reads as off-target. A beautifully engineered retrieval chatbot answers a question the interviewer is not asking. Measurement work answers the one they are.
• Bandwidth, not FLOPS, is usually the binding constraint in decode. Candidates who reason from the memory hierarchy outperform candidates who reason from teraflops.
• Only one of the seven roles needs CUDA. The other six reward systems fundamentals, profiling and honest measurement far more than kernel authorship.
• India and the UK both have real hiring channels — neoclouds and global capability centres in one, GPU cloud operators, silicon and research compute in the other.

AI Tech Connect has published dozens of careers guides, and until now every one of them pointed at the application layer or at research: agents, retrieval, evaluation, fine-tuning, forward-deployed work. That is where most of the hiring volume sits, and the advice is sound. But it leaves a whole class of employer unaddressed, and it is the class whose interviews trip up otherwise strong candidates most reliably. If you have already read our guide to choosing an AI specialisation across agents, evals, infra and FDE and landed on infra, this is the follow-through.

What counts as an AI infrastructure company

The category is broader than "GPU cloud", and the boundary is easier to draw by looking at the invoice than at the marketing. An infrastructure company sells the substrate other people's AI runs on. Its revenue is metered in GPU-hours, tokens, queries, vectors or requests. Its margin is a function of utilisation. Its outages are somebody else's outage too. That shape produces a consistent engineering culture regardless of which segment the company sits in.

Two things follow from that table. First, the skills transfer sideways within it far more readily than they transfer down from the application layer: someone who has run a serving fleet can move to a gateway company or an observability company without retraining, because the underlying questions — queueing, saturation, cost attribution — are the same questions. Second, the segments hire at very different rates over time. Vector database hiring has been quieter since retrieval consolidated into general-purpose databases; serving and capacity hiring has not. Pick the segment on the work, but check the hiring signal before you commit six months of preparation to one company's stack.

The role map, and what each interview tests

"Infrastructure engineer" is not a job. Seven reasonably distinct roles sit under it, and they test different things. Knowing which one you are interviewing for is worth more than another week of revision, because the same answer that impresses a fleet engineer will bore a kernel engineer.

If you are coming from application work, the honest ranking of accessibility runs: distributed systems and reliability first, then capacity and fleet, then inference and serving, then developer experience and solutions, with kernel engineering last by a considerable margin. Kernel roles are the ones people fixate on and the ones with the narrowest funnel. The serving roles have far more headcount and reward the same curiosity.

Why the infra rubric is not the app-layer rubric

This is the crux, and it is worth being blunt about. Application-layer AI interviews are, at bottom, quality interviews: can you make the model produce a good answer, and can you tell whether it did? Infrastructure interviews are efficiency and reliability interviews: given that the model is what it is, can you serve it predictably and profitably at load? The two rubrics barely overlap.

The fundamentals that carry an infra interview

The good news about this rubric is that it is finite and it ages slowly. Ten concepts cover most of what a serving or capacity interview will reach for, and none of them has changed materially in the last two years. Learn them properly once.

If any of those are unfamiliar, the fastest route in is to serve a model yourself and watch the numbers move. Our vLLM throughput and latency playbook covers the batching and KV mechanics hands-on, and the prefill-decode disaggregation guide explains the one architectural split that most sharply separates people who have run serving from people who have read about it.

Proof of work an infra hiring manager actually believes

This is the part that decides whether you get the interview at all, and it is where most candidates lose before they start. The general advice in our proof-of-work portfolio guide holds here, but the specific artefacts are different, and the difference is not cosmetic.

Infrastructure hiring managers are professional sceptics about numbers, because their job is to be lied to by benchmarks. What earns their attention is not an impressive result. It is a result you have made falsifiable: methodology stated, environment pinned, limitations declared, raw data published. An artefact they can argue with is an artefact they will read.

The benchmark harness, and why the methodology is the artefact

Take the top entry seriously, because it is the most accessible of the high-signal options and it costs less than people assume: a few hours of rented GPU time on a serverless platform will produce a defensible sweep. The code below is a sketch, not a framework. What matters is the order of operations.

#!/usr/bin/env bash
# bench/run.sh — one sweep, one machine, one recorded environment.
set -euo pipefail

MODEL="${MODEL:?set MODEL}"
STACK="${STACK:?vllm|sglang|trtllm}"
OUT="results/${STACK}-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"

# 1. Record the environment BEFORE any numbers exist. Reviewers read this first,
#    and an unrecorded environment makes every downstream figure unciteable.
nvidia-smi -q                                        > "$OUT/gpu.txt"
nvidia-smi --query-gpu=name,memory.total,persistence_mode,clocks.max.sm \
           --format=csv                              > "$OUT/gpu-summary.csv"
pip freeze                                           > "$OUT/python-env.txt"
git rev-parse HEAD                                   > "$OUT/harness-commit.txt"
uname -a                                             > "$OUT/host.txt"

# 2. Warm up, then throw the warm-up away. Cold-start numbers are a different
#    experiment and mixing them in is the commonest way to publish a lie.
python -m bench.load --stack "$STACK" --model "$MODEL" \
       --concurrency 1 --requests 50 --discard

# 3. Sweep concurrency with FIXED input and output lengths, so the x-axis
#    means one thing. Variable-length prompts belong in a separate run.
for C in 1 2 4 8 16 32 64 128 256; do
  python -m bench.load \
    --stack "$STACK" --model "$MODEL" \
    --concurrency "$C" \
    --input-tokens 1024 --output-tokens 256 \
    --requests $(( C * 20 )) \
    --repeat 3 --seed 1234 \
    --json "$OUT/c${C}.json"
done

# 4. One row per concurrency point: TTFT p50/p95/p99, inter-token latency p99,
#    aggregate tokens/s, and GPU-hours per million tokens. Median of 3 runs,
#    with the spread printed beside it.
python -m bench.report "$OUT" --markdown > "$OUT/RESULTS.md"

The results file is where the credibility lives, and it should read like a lab notebook rather than a launch post. A methodology header of this shape does more for you than another ten percent on the headline number:

## Methodology

Hardware      : 1x NVIDIA H200 SXM (141 GB HBM3e), persistence mode on, driver pinned
Host          : CPU model, RAM, kernel version, container image digest sha256:...
Stacks        : vLLM <version> (commit abc1234), SGLang <version> (commit def5678)
Model         : <exact repo>@<commit sha>, FP8 weights, quantised with the exact
                command in scripts/quantise.sh — not a third-party checkpoint
Workload      : 1,024 input / 256 output tokens, fixed seed, prompts in data/
Warm-up       : 50 requests discarded before every measured point
Runs          : 3 per point; reported value is the median, spread in brackets

## Limitations

- Single node only. No multi-node results; collectives are untested here.
- Long context (>8k) not measured. The KV curve will look different there.
- Speculative decoding and LoRA adapters disabled in both stacks.

## Known bias

vLLM ran with chunked prefill enabled; SGLang did not, because the equivalent
flag behaved differently in the version tested. This favours vLLM on TTFT at
high concurrency. Raw JSON is in results/ — re-run and disagree.

That "known bias" section is the whole trick. Declaring the flaw in your own experiment is the strongest available signal that you are not selling anything, and it is the sentence a hiring manager quotes back to you in the interview. Candidates who publish clean, flattering benchmarks with no limitations section get read as marketing. Candidates who publish an awkward result with a careful caveat get read as colleagues.

The postmortem, the quantisation study and the cost teardown

The saturation postmortem is the cheapest high-signal artefact available to anyone who has ever operated a service, because you probably already have the incident. Write up the time your queue depth outran your drain rate: the graph, the moment you noticed, the wrong hypothesis you chased first, the actual cause, the fix, and the guardrail you added so it cannot recur silently. Name your own error plainly. Infrastructure teams live inside incidents, and they are hiring for how you behave in one.

The quantisation study works because it forces two variables to stay in view. Take one open-weight model, quantise it to two or three formats with commands you publish, and report both throughput and quality on a task you can defend — not a leaderboard score, but something concrete like exact-match on a structured extraction set. The interesting output is rarely "FP8 is fine". It is the specific place where quality fell off and the specific batch size where the throughput gain stopped paying.

The cost teardown is pure arithmetic and reads as seniority. Model self-hosting against a hosted API at three load levels — a trickle, a steady business-hours load, and a saturated fleet — and show where the crossover sits. Nearly everyone underestimates how much load is needed before self-hosting wins, because nearly everyone forgets that a GPU you rent at night is billed whether or not anyone is using it. The method for building that denominator properly is in our LLM unit economics guide, and if you are comparing hardware vendors, the mixed-vendor GPU fleet guide covers the portability traps that make naive cost comparisons wrong.

The loop, and a worked capacity calculation

Getting the interview is mostly a function of visibility. Infrastructure people congregate in a small number of places: the issue trackers and Slack or Discord channels of serving projects, the mailing lists and forums around specific hardware, conference talks about production incidents, and the comment threads under other people's benchmarks. Reviewing somebody else's benchmark carefully and publicly is an underrated way in — it costs an evening, it demonstrates exactly the skill being hired for, and the author usually remembers you. Cold applications to GPU cloud companies convert poorly; a thoughtful issue comment on their open-source runtime converts surprisingly well.

What the rounds usually contain

Loops vary, but four rounds recur across the segment. A systems design round framed as a constraint rather than a product: serve this model to this many concurrent users under a p99 budget and a cost ceiling, and justify every choice. A debugging round where you are handed a saturated fleet — climbing queue depth, flat GPU utilisation, rising p99 — and asked what you check first. A reading round where you interpret a flame graph, a GPU trace or a latency histogram out loud, which is really a test of whether you have ever profiled anything. And a back-of-envelope round, which is the one candidates most often fumble, because it asks you to produce numbers from first principles while someone watches.

The back-of-envelope, worked

Here is the calculation in full, with every assumption stated. The hardware figures are NVIDIA's published H200 specifications, checked in August 2026: 141 GB of HBM3e at approximately 4.8 TB/s. The model shape is Llama-3-70B as described in Meta's Llama 3 paper: 80 layers, 8 key-value heads under grouped-query attention, head dimension 128. Everything else is an assumption I am choosing openly, which is exactly what the interviewer wants to hear you do.

# capacity.py — a decode-bound estimate you can defend out loud.
# Every input below is an ASSUMPTION. Say them aloud in the interview;
# you are graded on the reasoning, not on the digits.

# --- Hardware: NVIDIA H200 published specs, checked August 2026 --------------
HBM_CAPACITY_GB   = 141.0      # HBM3e per GPU
HBM_BANDWIDTH_GBS = 4_800.0    # ~4.8 TB/s peak
ACHIEVED_FRACTION = 0.70       # you never get datasheet bandwidth; 60-80% is honest

# --- Model: 70B dense decoder served at 8-bit weights ------------------------
PARAMS_B        = 70.0
BYTES_PER_PARAM = 1.0          # FP8 / INT8
LAYERS          = 80           # Llama-3-70B: 80 layers,
KV_HEADS        = 8            #   8 KV heads (GQA),
HEAD_DIM        = 128          #   head_dim 128
KV_BYTES        = 2.0          # KV cache retained in FP16

# --- Workload ----------------------------------------------------------------
CTX_TOKENS   = 4_000           # average live sequence length
GPU_HOUR_USD = 2.50            # ASSUMPTION: mid-range on-demand rate, Aug 2026

weights_gb         = PARAMS_B * BYTES_PER_PARAM                    # 70 GB
kv_per_token_bytes = 2 * KV_HEADS * HEAD_DIM * KV_BYTES * LAYERS   # 327,680 B
kv_per_seq_gb      = kv_per_token_bytes * CTX_TOKENS / 1e9         # 1.31 GB

# 1. Concurrency ceiling. Reserve ~10% for activations and fragmentation.
usable_gb = HBM_CAPACITY_GB * 0.90 - weights_gb                    # 56.9 GB
max_seqs  = int(usable_gb / kv_per_seq_gb)                         # 43

# 2. Single-stream decode ceiling: one full sweep of the weights per token.
eff_bw            = HBM_BANDWIDTH_GBS * ACHIEVED_FRACTION          # 3,360 GB/s
sec_per_step      = weights_gb / eff_bw                            # 0.0208 s
single_stream_tps = 1 / sec_per_step                               # 48 tok/s

# 3. Batching amortises that weight read across the batch, so AGGREGATE
#    throughput rises roughly linearly until the compute roof or KV roof bites.
batch         = min(max_seqs, 32)
aggregate_tps = single_stream_tps * batch                          # ~1,536 tok/s

# 4. Cost per million output tokens on ONE GPU at full batch.
usd_per_mtok = (1e6 / aggregate_tps) / 3600 * GPU_HOUR_USD         # ~$0.45

Read the outputs back in plain English, because that is what the round is really testing. KV cache costs 320 KiB per token, so a 4,000-token conversation costs 1.31 GB of HBM just to stay alive. After the weights, one GPU has roughly 57 GB left, which is about 43 concurrent conversations — and note that the constraint is memory, not arithmetic. Single-stream generation cannot exceed about 48 tokens per second, because generating one token means reading all 70 GB of weights out of memory once. Batch thirty-two of those together and the same weight read serves thirty-two users, so aggregate throughput approaches 1,500 tokens per second while each individual user still sees roughly the same speed. At an assumed $2.50 per GPU-hour, that is about $0.45 per million output tokens before any margin, overhead or idle time.

On the GPU-hour assumption: rented accelerator pricing moves quickly, and published on-demand rates vary widely by provider, region and commitment. IntuitionLabs' August 2026 comparison across more than fifteen providers put H100 on-demand rates between $1.49 and $6.98 per GPU-hour, which is a factor of nearly five for nominally the same chip. Use whatever the provider publishes on the day, state it as an assumption, and show the calculation's sensitivity to it — that is a better answer than a precise number you cannot source.

India and the UK: where the jobs are

Both markets have real infrastructure hiring, and the channels differ enough to be worth naming separately.

In India, the largest driver has been the compute build-out. The Press Information Bureau announced that the IndiaAI Mission's common compute capacity crossed 34,000 GPUs during 2025, and reporting through mid-2026 put the empanelled total above 38,000 across a group of private providers including Jio, Tata Communications, Yotta and E2E Networks. Capacity at that scale creates operational roles rather than research ones: fleet engineers, serving engineers, support and solutions engineers who onboard startups onto subsidised capacity. Separately, global capability centres in Bengaluru, Hyderabad, Pune and Chennai have moved well past the support-function stereotype and now run genuine platform and inference teams for their parent organisations — often the most accessible entry point for an experienced backend engineer, because they hire on systems fundamentals and train the GPU specifics. And silicon design work has arrived in earnest: Graphcore, the Bristol-headquartered accelerator company acquired by SoftBank in July 2024, announced an AI engineering campus in Bengaluru in October 2025, committing up to £1 billion over a decade and 500 semiconductor engineering roles, with the first hundred in logical design, physical design, verification and bring-up.

In the UK, the shape is different: fewer very large fleets, more silicon and more research compute. London-headquartered Nscale operates AI data-centre and GPU cloud capacity; Fractile, which raised a $220 million round announced in 2026, is building inference-specific silicon out of London and Bristol and recruiting from the Graphcore, NVIDIA and Imagination talent pools. Arm in Cambridge and Graphcore in Bristol anchor a genuine hardware and compiler community that has no real equivalent elsewhere in Europe. And research compute is a serious and often overlooked employer: Isambard-AI at the University of Bristol, part of the national AI Research Resource, runs 5,448 NVIDIA GH200 Grace Hopper superchips across 1,362 nodes, and clusters of that size need scheduler, storage, networking and user-support engineers continuously. Research-compute roles pay less than a well-funded startup and offer unusually good access to hardware most engineers never touch — a reasonable trade early in an infrastructure career.

On compensation, treat the published numbers with suspicion. Aggregator and recruiter data for infrastructure specialisms is thin, inconsistently levelled and heavily skewed by a small number of US outliers, so quoting a band would mislead more than it helps. The reliable observations are structural rather than numerical: infrastructure roles at a given level tend to sit at or above equivalent application-engineering roles at the same company; the spread within the specialism is very wide because scarce skills like kernel engineering clear at a different price from platform work; and equity composition varies enormously between a listed silicon company, a venture-funded neocloud and a university cluster. Ask for the band in the first call, and compare offers on total structure rather than on a headline figure from a salary blog.

Common pitfalls, and when this path is not for you

• Leading with application-layer work. Covered above, and it is the single most common own goal. Reorder your portfolio so the measurement work is first, even if the application work is more polished.
• Quoting benchmark numbers you did not measure. Repeating a vendor's throughput claim in an interview is a trap, because the person opposite has probably tried to reproduce it and knows the caveats you do not. If you must cite someone else's figure, name the source and the date, and say what you would check before believing it.
• Ignoring cost entirely. An engineer who optimises latency without reference to what it costs is optimising half the problem. Every design answer should end with the price of the design.
• Publishing a benchmark with no methodology section. Without pinned versions, a recorded environment and a stated workload, your numbers are unciteable — and the absence reads as either inexperience or salesmanship.
• Reporting the mean. An average latency that improved while p99 doubled is a regression that your graph is hiding. Report the distribution, always.
• Benchmarking a stack you have not tuned against one you have. Comparing your carefully configured favourite against a default-flags competitor is not a comparison. Either tune both or declare the asymmetry.
• Assuming kernel engineering is the way in. It is the narrowest door in the building. Six other roles are hiring more people on more transferable skills.

And the honest case against. Infrastructure work is unglamorous in a specific way: much of it is measurement, much of it is on-call, and the wins are invisible to everyone outside the team — nobody thanks you for a p99 that did not move. The feedback loop is slower than application work, the hardware is expensive enough that experimenting on a whim is not always possible, and a meaningful share of the job is arguing about numbers. If you are energised by shipping user-visible features quickly, the application layer is a better fit and there is no shame in that. If you are the sort of person who reads a flat scaling curve and cannot leave it alone until you know why, this is the right building.

One last thing worth saying plainly, because it is the reason this article exists. Infrastructure proof of work is unusually invisible. A benchmark harness in a private repository, a postmortem in an internal wiki, a patch merged under a handle nobody connects to your name — none of it reaches the people hiring unless you put it somewhere they look. The artefacts described here are worth building, and they are worth publishing under your own name, with a profile attached that says what you work on and where you are. Early Verified Builder profiles on AI Tech Connect carry the Founding Builder badge, and there are only so many of those. It takes two minutes and no CV, and it is the difference between having done the work and being findable for having done it.

Sources

• NVIDIA — H100 Tensor Core GPU datasheet
• NVIDIA — H200 Tensor Core GPU specifications
• Kwon et al. — Efficient Memory Management for LLM Serving with PagedAttention (SOSP 2023)
• Meta — The Llama 3 Herd of Models (model configuration)
• Press Information Bureau — India's common compute capacity crosses 34,000 GPUs
• University of Bristol — Isambard-AI system specifications
• Graphcore — £1bn India investment and Bengaluru AI engineering campus
• IntuitionLabs — H100 rental prices compared across cloud providers (August 2026)