Air-Gapped LLM Deployment for Banks, Insurers and the NHS

Who actually needs this, and who only thinks they do

Somebody in a procurement meeting says the words "it has to be air-gapped", everybody nods, and eighteen months later a team is maintaining an isolated GPU cluster that nobody can patch quickly, running a model two generations behind, to satisfy a requirement that was never written down. This is the single most expensive misunderstanding in regulated AI delivery, and it is almost always avoidable by asking one question early: which specific obligation are we satisfying, and what does it actually demand?

Full air-gap — no network path in or out, artefacts moved by a controlled physical or diode-based transfer — is rare, costly and slow. The far more common real requirement is an egress-restricted deployment: your own virtual private cloud or on-premise cluster, in an approved region, with no route to the public internet except through an explicitly allow-listed and logged path, and often not even that. That configuration satisfies the overwhelming majority of what banks, insurers, NHS organisations and Indian regulated entities are actually being asked for, and it costs a fraction of the alternative to run.

Be honest about the difference, because the cost is mostly recurring. An air gap does not have a build price and then a maintenance price; it has a permanent tax on every dependency update, every model upgrade, every CVE, every engineer onboarding and every incident. Budget for the tax, not the build.

Match the driver to the level it genuinely demands

The table below is a starting point for the conversation with your compliance and information-governance colleagues. It is not legal advice, it does not substitute for their assessment, and the right-hand column is the one that saves money.

What is driving the requirement	What it typically demands in architecture	What it usually does not demand
Personal data under UK GDPR and the Data Protection Act	Lawful basis, processor contracts, documented transfers, security proportionate to risk, deletion on request	An air gap. Regional processing with contractual and technical controls is the normal answer
Personal data under India's Digital Personal Data Protection Act	Notice and consent handling, reasonable security safeguards, breach handling, defined processor obligations	An air gap. In-country processing in an approved region is the usual architectural consequence
NHS patient data and information-governance assurance	Documented data flows, supplier assurance, access control, retention discipline, assessed security posture	An air gap. Most trusts accept assured hosting inside a controlled boundary
Supervisory expectations on IT outsourcing and operational resilience	Governance, inspection and audit access, concentration-risk analysis, contingency and exit planning	An air gap. It demands that you can leave a provider, not that you never use one
Segregated payment, card or industrial-control zones	Genuine network segregation from corporate and internet zones, tightly controlled artefact ingress	Nothing — this is a real air-gap case, and should be treated as one
Classification levels or defence and national-security work	Accredited enclaves, cleared personnel, approved transfer mechanisms, formal accreditation cycles	Nothing — the accreditation regime dictates the design, not your architecture preferences
Sites with no reliable connectivity (ships, remote clinics, plant floors)	Fully local inference, local model store, offline update path, degraded-mode behaviour	Nothing — here the driver is availability rather than compliance, but the engineering is the same

For grounding on the named obligations rather than paraphrase: India's Act sits with the Ministry of Electronics and Information Technology; the RBI's Master Direction on Outsourcing of Information Technology Services sets the governance, inspection-access and exit expectations for RBI-supervised entities; NHS organisations and their suppliers work through the Data Security and Protection Toolkit; and in UK financial services the Bank of England, PRA and FCA operate a critical third parties regime, under which HM Treasury made its first designations in July 2026. Read them with your compliance function; do not let an engineer, including this one, tell you what they require of your specific institution.

The isolation ladder

There are four rungs, and moving up one is roughly an order of magnitude more operational effort than the rung below. Pick the lowest rung that satisfies the written requirement.

Rung	What leaves your boundary	Capability available	Ongoing operational load	Audit burden
0. Public API over the internet	Prompts, completions, metadata — to a third party over public routes	Everything, immediately, including the newest frontier models	Minimal — a client library and a key	Highest per-request scrutiny; hardest to justify for regulated data
1. Private endpoint to a managed service	Nothing over public routes; traffic stays on the provider backbone within your chosen region	Near-frontier — whatever your provider offers in London, UK South, Mumbai or Central India	Low — networking, private DNS, key management	Moderate. Concentration risk and exit planning become the live questions
2. In-VPC self-host, egress allow-listed	Nothing at inference time; controlled egress for package and image pulls only	Best available open-weight models; you choose the version and the upgrade date	Moderate to high — you now own capacity, upgrades and reliability	Moderate. You must evidence the allow-list and prove it is enforced
3. Fully air-gapped on-premise	Nothing, ever. Artefacts enter through an approved transfer mechanism	Whatever you imported at the last window — typically a release or two behind	High and permanent — every update becomes a staffed ritual	Highest evidentiary volume, but conceptually simplest: the boundary is the control

Rung 1 deserves more attention than it usually gets in these conversations. If your only real constraint is where processing happens and who can see the traffic, a private endpoint into a managed service in a London or Mumbai region gives you frontier capability with a fraction of the operational cost of rung 2, and the request-routing patterns for it — how you decide which region a given request belongs in — are covered in our DPDP and GDPR routing guide. Rung 2 is where most serious regulated deployments land. Rung 3 is where this article earns its keep.

Getting the weights across the boundary

Inside an air gap, a model is not a name. llama-3.3-70b-instruct is a label that can point at different tensors on different days. What you are actually deploying is a specific set of files with a specific content hash, and your entire evidentiary position rests on being able to name that hash months later.

The import is a three-stage pipeline: a connected staging environment that fetches and records, a signing step that makes the bundle tamper-evident, and an isolated ingest that verifies before it trusts. Every stage produces a record.

# === STAGE 1: connected staging zone (has internet, has nothing sensitive) ===
# Pin the revision. A tag or a branch name is not a version.
REPO="Qwen/Qwen3-32B"
REV="a1b2c3d4e5f60718293a4b5c6d7e8f9012345678"   # commit SHA from the model repo
DEST="/staging/weights/qwen3-32b-${REV:0:12}"

hf download "$REPO" --revision "$REV" --local-dir "$DEST"
# (huggingface_hub < 0.34 spells this: huggingface-cli download ...)

# Record where it came from, when, and who ran it. This is the provenance record.
cat > "$DEST/PROVENANCE.json" <<EOF
{
  "source_repo":   "$REPO",
  "source_revision": "$REV",
  "fetched_at_utc": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  "fetched_by":    "${USER}",
  "licence_file":  "LICENSE",
  "import_ticket": "CHG-2026-04417"
}
EOF

# Content manifest over every file, deterministic order.
( cd "$DEST" && find . -type f ! -name 'MANIFEST.sha256' -print0 \
    | sort -z | xargs -0 sha256sum > MANIFEST.sha256 )

# === STAGE 2: sign the manifest with the model-signing key ===
cosign sign-blob --key /keys/model-signing.key \
  --output-signature "$DEST/MANIFEST.sha256.sig" \
  "$DEST/MANIFEST.sha256"

# === STAGE 3: inside the boundary, verify BEFORE anything else touches it ===
cosign verify-blob --key /keys/model-signing.pub \
  --signature MANIFEST.sha256.sig MANIFEST.sha256
( cd /models/qwen3-32b-a1b2c3d4e5f6 && sha256sum -c MANIFEST.sha256 )

Note what is being signed: the manifest, not each file. That gives you one signature to manage and a cheap, complete integrity check on ingest. Note also that PROVENANCE.json is inside the manifest, so the record of where the weights came from is itself tamper-evident.

Open-weight and open-source are not the same thing, and legal will notice

This distinction costs teams weeks when it surfaces late. A genuinely open-source model ships under a recognised licence — Apache 2.0 or MIT, typically — with no restriction on field of use. An open-weight model ships weights you can download and run, under a bespoke community or custom licence that may carry acceptable-use conditions, redistribution terms, naming or attribution requirements, or thresholds that change the terms above a certain scale of use.

Neither is better; they are different objects for the purpose of sign-off. A bank's legal team can approve an Apache 2.0 artefact in an afternoon. A bespoke licence needs a read, and it needs re-reading when the model publisher updates it. Three practical rules:

• Import the licence file with the weights and hash it into the manifest. The licence that applies is the one that shipped with the artefact you deployed, not the one on the website today.
• Get sign-off before the weights cross the boundary, not before they go to production. Removing an artefact from an air-gapped environment cleanly is far more work than not importing it.
• Record the approval reference in the registry entry. When an auditor asks who approved this model for use with customer data, you want a ticket number, not a recollection.

The internal model registry

Once you have more than two models — and you will, because you need a judge model and probably an embedding model — you need a registry. It does not need to be a product. An object-store prefix plus a small metadata table is enough, and in an air gap the boring option is the right one.

The registry entry for every model should carry: the content digest, the upstream repository and revision, the import ticket, the licence identifier and approval reference, the approved use classes, the environments it is promoted to, the evaluation run that justified promotion, and the date it was last verified against its manifest. That last field matters more than it sounds — a periodic re-verification job that walks the model store and re-checks hashes catches silent corruption and, more importantly, produces the evidence that you check. The broader discipline of pinning model identity and detecting drift is covered in our guide to verifying model provenance and cutoff drift.

The dependency problem nobody budgets for

Ask a team how long an air-gapped deployment will take and they will estimate the model serving. The model serving is the easy part. The hard part is that the modern Python and CUDA ecosystem assumes an internet connection at a dozen points you have never thought about, and each one fails differently: some loudly at start-up, some silently at first use, some only under a specific code path that runs once a week.

This is the table to hand to whoever is doing the estimate.

What breaks	Why	The fix
pip install	No route to PyPI	Internal mirror (Nexus, Artifactory, devpi) or a pre-built wheelhouse plus --no-index
Torch and CUDA wheels	Large, platform- and CUDA-version-specific; often on a separate index	pip download with explicit platform, ABI and index URL on the connected side
Base container images	Docker Hub, GHCR and NGC are all unreachable	Internal registry seeded by skopeo copy, digest-pinned in every manifest
Tokenizer and config auto-download	from_pretrained("org/model") resolves against the Hub even when weights are local	HF_HUB_OFFLINE=1 plus absolute local paths everywhere — never a repo ID
Chat template resolution	The template lives in tokenizer config; a missing one silently changes prompt formatting	Import the full repo, not just *.safetensors; assert the template hash at start-up
Framework telemetry	Serving engines and SDKs phone home with usage statistics by default	VLLM_NO_USAGE_STATS=1, DO_NOT_TRACK=1, and audit every library for its own opt-out
Encoder and NLP data files	tiktoken, NLTK and spaCy fetch data on first call, not at install	Pre-seed the cache directories (TIKTOKEN_CACHE_DIR and equivalents) and ship them in the image
TLS certificate revocation checks	OCSP and CRL endpoints are unreachable; clients hang or soft-fail unpredictably	Internal CA with a defined revocation approach agreed with your security team
Time synchronisation	No public NTP; clock drift breaks TLS validation and corrupts audit ordering	Internal stratum-2 NTP, monitored. Audit trails are worthless if timestamps disagree
Vulnerability scanning	Scanners quietly report "clean" when their database cannot update	Import the vulnerability DB on the same cadence as everything else; alert on DB age
Dashboard and plugin installs	Grafana plugins, dashboards and datasource definitions are fetched at runtime	Bake plugins into the image; provision dashboards as files in version control

The wheelhouse workflow is worth spelling out, because getting the platform tags wrong is the most common first-day failure — you build the bundle on a developer laptop, and it silently resolves to wheels that will not install on the target.

# === Connected side: build a wheelhouse for the EXACT target platform ===
# Do not run this on a laptop and hope. Match the target's Python and glibc.
pip download \
  --platform manylinux_2_28_x86_64 \
  --python-version 312 \
  --implementation cp \
  --abi cp312 \
  --only-binary=:all: \
  --extra-index-url https://download.pytorch.org/whl/cu128 \
  --dest ./wheelhouse \
  -r requirements.lock

# Hash-lock the bundle so ingest can verify it like any other artefact.
( cd wheelhouse && sha256sum *.whl > ../wheelhouse.sha256 )
tar czf wheelhouse-2026-08.tar.gz wheelhouse wheelhouse.sha256

# === Isolated side: install with the index disabled entirely ===
tar xzf wheelhouse-2026-08.tar.gz
( cd wheelhouse && sha256sum -c ../wheelhouse.sha256 )
pip install --no-index --find-links=./wheelhouse -r requirements.lock

# === Or, if you run an internal mirror instead of a wheelhouse ===
# /etc/pip.conf on every build and runtime host
cat > /etc/pip.conf <<'EOF'
[global]
index-url = https://artifacts.internal.example.net/repository/pypi/simple
cert = /etc/pki/tls/certs/internal-root-ca.pem
require-hashes = true
EOF

require-hashes = true is the line that turns a mirror from a convenience into a control: every requirement must carry a hash, so a compromised or substituted mirror artefact fails the install rather than shipping. The same reasoning applies to any agent tooling you bring inside the boundary — the supply-chain checks in our guide to vetting MCP servers and agent skills do not stop being relevant just because the network is closed. An air gap protects you from remote exploitation. It does nothing about a malicious artefact you imported yourself.

Container images travel the same way, and the reason to use an OCI layout rather than a simple tarball is digest preservation — you want the image inside the boundary to be verifiably the image you approved outside it.

# Connected side — copy all architectures, preserving digests
skopeo copy --all \
  docker://vllm/vllm-openai:v0.11.0 \
  oci:/transfer/vllm-openai:v0.11.0

skopeo inspect --raw oci:/transfer/vllm-openai:v0.11.0 | sha256sum \
  > /transfer/vllm-openai-v0.11.0.digest

# Isolated side — push into the internal registry, then pin by digest forever
skopeo copy --all \
  oci:/transfer/vllm-openai:v0.11.0 \
  docker://registry.internal.example.net/ai/vllm-openai:v0.11.0

# In every deployment manifest, reference the digest, never the tag:
#   image: registry.internal.example.net/ai/vllm-openai@sha256:9f2c...

Standing up the serving stack inside the boundary

With the artefacts in place, serving is comparatively conventional. vLLM is the usual choice and the launch differs from a connected deployment in three respects: every path is absolute and local, offline mode is asserted rather than assumed, and telemetry is explicitly disabled.

#!/usr/bin/env bash
set -euo pipefail

# --- Assert isolation. Fail fast rather than degrade quietly. ---
export HF_HUB_OFFLINE=1          # no Hub HTTP calls at all
export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HOME=/models/.hf-home  # writable, pre-seeded, inside the boundary
export TIKTOKEN_CACHE_DIR=/models/.tiktoken
export VLLM_NO_USAGE_STATS=1     # also honours DO_NOT_TRACK
export DO_NOT_TRACK=1
unset HTTP_PROXY HTTPS_PROXY      # a proxy is a route; there must not be one

MODEL_DIR=/models/qwen3-32b-a1b2c3d4e5f6

# --- Verify integrity on every start, not just on import. ---
( cd "$MODEL_DIR" && sha256sum -c MANIFEST.sha256 --quiet )

vllm serve "$MODEL_DIR" \
  --served-model-name internal-general-32b \
  --tokenizer "$MODEL_DIR" \
  --download-dir /models/.cache \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --quantization fp8 \
  --host 0.0.0.0 --port 8000

# Request logging is handled at the gateway, not the engine — prompts are
# records (see below). Check your vLLM version's flag for engine-side request
# logging: the name and the default have changed between releases.

Two details in there are load-bearing. Passing a directory to vllm serve rather than a repository identifier is what guarantees no resolution attempt against a remote hub. Re-verifying the manifest on every start costs seconds on a warm page cache and converts silent corruption into a refusal to boot, which in a regulated environment is the behaviour you want.

GPU sizing by model class

The following is arithmetic, not benchmarking. Weight memory follows directly from parameter count and precision: roughly two bytes per parameter at bf16, one at fp8, and around 0.5 to 0.6 bytes at 4-bit once scales and zero-points are counted. On top of that, allow a further 20 to 40 per cent of device memory for KV cache, activations, CUDA graphs and fragmentation. Deliberately absent from this table are throughput figures, because they depend on your batch profile, context lengths and interconnect, and a fabricated tokens-per-second number is worse than none. Measure on your own traffic — the method is in our vLLM throughput and latency playbook.

Model class	Weights at bf16	Weights at fp8	Weights at 4-bit	Sensible single-node starting point	What it is good for
7–9B dense	~14–18 GB	~7–9 GB	~4–5 GB	1 × 48–80 GB accelerator	Classification, extraction, routing, redaction, guardrails
24–32B dense	~48–64 GB	~24–32 GB	~13–18 GB	2 × 80 GB (TP=2); 1 × 80 GB at 4-bit, tight on context	Summarisation, retrieval-grounded answering, drafting
70B dense	~140 GB	~70 GB	~38–45 GB	4 × 80 GB (TP=4)	Multi-step reasoning, adjudication support, complex drafting
100B+ dense or large MoE	200 GB+	100 GB+	55 GB+	8 × 80 GB node, sometimes multi-node	The expensive rung. Justify it with an eval, not an instinct

Two caveats that catch people out. For mixture-of-experts models, total parameters determine memory while active parameters determine compute, so an MoE can be cheap to run and expensive to fit — exactly the wrong way round for an air gap where memory is a capital purchase. And at long contexts with high concurrency, KV cache can exceed weight memory outright; if your use case is document-heavy, size from the KV cache first and treat the weights as the smaller term.

On quantisation: fp8 is the low-risk default on hardware that supports it natively, and 4-bit schemes buy real headroom at a quality cost that is task-dependent and cannot be predicted from published averages. In a regulated environment there is a further point that engineering teams routinely miss — a quantised build is a different model for audit purposes. It has a different digest, it needs its own registry entry, and it needs its own evaluation run. Promoting a quantised variant on the strength of the full-precision model's evals is a finding waiting to happen.

High availability with nothing to scale into

In a cloud deployment, a failed node is an inconvenience. Behind an air gap there is no autoscaling group and no spot capacity; there is the hardware in the rack. Three consequences follow.

• Size for N+1 at peak, not at average. The spare capacity is the availability strategy. If a four-GPU node is the serving unit, the second node is not optional.
• Build admission control, not just a queue. When capacity is fixed, the failure mode under load is unbounded latency, which is worse for a clinician or an underwriter than a clear refusal. Cap the queue, shed load with a meaningful error, and define the degraded mode in advance — usually a smaller model or a deterministic non-AI fallback.
• Treat hardware lead time as an availability parameter. Accelerator procurement into a secure data centre is measured in months, not hours. A cold spare on the shelf is genuinely cheaper than the recovery-time objective it protects. Write that trade-off down where the finance conversation happens, because it will otherwise be cut as "unused kit".

Running evaluations with no internet

Evaluation is where air-gapped programmes most often quietly give up, because the standard playbook assumes a hosted judge model and a downloadable benchmark. Neither is available. What you build instead is more work up front and better afterwards, because it is grounded in your own data rather than a public leaderboard.

The judge lives inside. Run a second model as the evaluator, served from a separate endpoint, with its own registry entry and its own pinned digest. Two rules make this trustworthy. The judge must not be the model under test, or you are measuring self-consistency and calling it quality. And the judge must not be upgraded in the same change window as the candidate, because when the score moves you will have no way to attribute the movement. In practice teams keep a judge on a slow release train — perhaps twice a year — and treat a judge upgrade as its own change with its own re-baselining exercise.

Golden sets are artefacts. They come through the same signed pipeline as the weights, with the same provenance record and the same digest discipline. Build them from three sources: real cases from your own environment, redacted or synthesised where the data itself cannot sit in an evaluation store; adversarial cases written by the domain experts, which in a bank means the credit and financial-crime teams and in an NHS trust means the clinical safety officer; and regression cases harvested from every production incident, which is the set that grows fastest and matters most.

Reproducibility is a manifest, not an intention. An evaluation result is only meaningful if you can say precisely what produced it.

# Every eval run pins six things. Anything unpinned is a future argument.
python -m evals.run \
  --candidate-endpoint     http://vllm-candidate.ai.svc.cluster.local:8000/v1 \
  --candidate-digest       sha256:9f2c41d0b7a3e5c81f6d94ab0327e5f18c4d2b7a09e6c531 \
  --judge-endpoint         http://vllm-judge.ai.svc.cluster.local:8000/v1 \
  --judge-digest           sha256:41ab7c920de5f3418b06c2ad975e0f3c8b1d4e6a72f95c08 \
  --golden-set             /evals/goldensets/retail-credit-adverse-action-v4 \
  --golden-set-digest      sha256:7d10ea45c9b3082f614d7ea3b58c096d2f4a81e07b3c9d52 \
  --harness-version        3.2.1 \
  --temperature 0 --seed 1729 --repeats 3 \
  --baseline /evals/runs/incumbent-2026-05-19.json \
  --out      /evals/runs/candidate-2026-08-19.json

# The gate, run by CI inside the boundary:
#   promote only if candidate is non-inferior to the incumbent on every
#   safety dimension, and no individual regression case flips from pass to fail.
python -m evals.gate \
  --candidate /evals/runs/candidate-2026-08-19.json \
  --incumbent /evals/runs/incumbent-2026-05-19.json \
  --non-inferiority-margin 0.02 \
  --fail-on-regression-case-flip

The harness itself is a dependency like any other — whether you use a house runner as above or import something like an open evaluation harness, it arrives through the wheelhouse and it gets a pinned version in the manifest. Note --repeats 3: even at temperature zero, batching and kernel non-determinism produce run-to-run variation, and a single run is not a measurement.

Observability when the telemetry cannot leave

Every observability vendor's default configuration sends data to the vendor. Inside the boundary you run the collection and the backend yourself, which is well-trodden — an OpenTelemetry collector into a self-hosted trace store, metrics store and log store, all from images you imported and pinned. The engineering is ordinary. The governance is not.

Prompts and completions are records. This is the sentence to put on the first slide. A prompt sent by a claims handler may contain a policyholder's medical history. A prompt sent by a clinician contains patient data by definition. A prompt sent by a relationship manager at an RBI-supervised institution contains customer financial data. The moment you capture that text into a trace, it is in scope for everything your organisation already does about records: retention schedules, access control, subject access and erasure, discovery, and breach handling. Teams that treat LLM traces as application debug logs create a large, unclassified, over-permissioned store of regulated data and do not realise it for a year.

# otel-collector.yaml — inside the boundary, no external exporter, ever.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Payloads are records. Keep a hash for correlation; drop the text.
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(attributes["gen_ai.prompt.sha256"], SHA256(attributes["gen_ai.prompt"]))
              where attributes["gen_ai.prompt"] != nil
          - set(attributes["gen_ai.completion.sha256"], SHA256(attributes["gen_ai.completion"]))
              where attributes["gen_ai.completion"] != nil
          - delete_key(attributes, "gen_ai.prompt")
          - delete_key(attributes, "gen_ai.completion")
  batch: {}

exporters:
  otlp/internal:
    endpoint: tempo.observability.svc.cluster.local:4317
    tls:
      ca_file: /etc/pki/tls/certs/internal-root-ca.pem

# CI ASSERTION: this file must contain no exporter endpoint outside
# *.svc.cluster.local or the approved internal CIDR ranges. The check runs
# on every merge; a failing check blocks the pipeline.

service:
  telemetry:
    metrics:
      level: basic
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [transform/redact, batch]
      exporters:  [otlp/internal]

Hashing rather than dropping outright is the compromise that keeps the system debuggable: you can still tell that the same prompt recurred, correlate a complaint with a trace, and count duplicates, without holding the text. Where you genuinely need the text — and for incident investigation you sometimes do — write it to a separate, explicitly classified store with a shorter retention period, a named owner, break-glass access and its own audit log. Do not let it default into the same index as your web-server logs. The wider question of what to capture and how to respond when a model misbehaves is covered in our guide to LLM incident response runbooks.

One more control worth the twenty lines it takes: make the no-egress property testable rather than assumed. A scheduled job inside the boundary that attempts a handful of outbound connections and alerts if any succeeds converts your network policy from a configuration into a monitored control, which is exactly the distinction an auditor is trained to probe.

The import ritual: patching and model upgrades

An air gap replaces continuous delivery with a batched, staffed import. This is not a failure of maturity; it is the trade you accepted. The failure is running it ad hoc, because then it happens when someone remembers, which is after the incident.

Put it on a calendar, staff it with two named people, and give each artefact class its own cadence and its own gate.

Artefact class	Cadence	Trigger for an out-of-band import	Gate before promotion
Host OS and GPU driver	Quarterly	Critical vendor advisory affecting the driver or kernel	Smoke test plus one full eval run — drivers change numerics
Container base images	Monthly	High or critical CVE in a base layer	Offline scan clean at your severity threshold, digest re-pinned
Serving engine (vLLM et al.)	Every 1–2 releases	Security fix, or a feature you have a written need for	Full eval, load test at peak concurrency, rollback rehearsed
Python dependencies	Monthly, as a locked set	Advisory in a direct or transitive dependency	Lockfile diff reviewed, hashes verified, eval run
Model weights	Quarterly at most	A measured capability gap, never novelty	Licence approval, full eval versus incumbent, canary inside the boundary
Judge model	Twice a year	Demonstrated judge failure mode	Re-baseline every golden set; never in the same window as a candidate
Golden sets	Continuously, promoted monthly	Any production incident adds a regression case immediately	Domain-expert review; signed and digested like any artefact
Vulnerability database	Weekly — more often than anything else	Never skipped. Alert if the DB is older than the threshold	None. Import it; the scan is the gate for everything else

Scanning deserves its own note because the failure is silent. A scanner with a stale database produces a green report, and a green report is what gets attached to the change record.

# === Connected side: pull the vulnerability DBs, on the weekly cadence ===
trivy image --download-db-only      --cache-dir ./trivy-cache
trivy image --download-java-db-only --cache-dir ./trivy-cache
tar czf trivy-db-2026-08-19.tar.gz -C ./trivy-cache .
sha256sum trivy-db-2026-08-19.tar.gz > trivy-db-2026-08-19.sha256

# === Isolated side: seed the cache, then scan with updates disabled ===
sha256sum -c trivy-db-2026-08-19.sha256
tar xzf trivy-db-2026-08-19.tar.gz -C /var/cache/trivy

# Fail the build on anything at or above your agreed threshold.
trivy image \
  --offline-scan \
  --skip-db-update --skip-java-db-update \
  --cache-dir /var/cache/trivy \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  registry.internal.example.net/ai/vllm-openai@sha256:9f2c41d0b7a3e5c8

# Guard against a stale database silently reporting clean:
DB_AGE_DAYS=$(( ( $(date +%s) - $(stat -c %Y /var/cache/trivy/db/trivy.db) ) / 86400 ))
[ "$DB_AGE_DAYS" -le 10 ] || { echo "Vulnerability DB is ${DB_AGE_DAYS}d old"; exit 2; }

Access control and the audit trail

An air gap answers "can this data leave?" It answers nothing about "who asked this question, and what did the system tell them?" That second question is the one that comes up in a complaint, a subject access request, a clinical safety review or a supervisory visit.

The single most common architectural mistake is the shared service account: an application authenticates to the inference gateway with one credential and every request looks identical. It is invisible in an architecture diagram and fatal in an audit, because you can no longer attribute an output to a person. Carry end-user identity from your internal identity provider through the gateway to the inference record, and make the per-user identity a required field rather than an optional one. The token-handling patterns are the same ones we describe in least-privilege credentials for AI agents; being on an isolated network changes none of them.

Each inference record should be able to answer, on its own: who asked, under what role and business justification, at what time by a synchronised clock, against which model digest and which serving-engine digest, with which decoding parameters, producing an output identified by hash, and reaching which downstream decision. That is a wide row, and it is worth every column.

Here is what auditors have actually asked for, in roughly the order they ask:

1. Identify the model that produced this specific output on this date. Answerable in seconds from the digest on the record; effectively unanswerable from a version tag.
2. Show the licence for those weights and who approved its use for this data class. The registry entry, with the approval reference.
3. Show the change record for the last model upgrade, including what you tested. The eval run manifest and the gate result, linked from the change ticket.
4. Show who has queried the system in the last quarter and under what identity. The per-user attribution, which is why the shared service account is fatal.
5. Show the egress rules, and evidence that they were enforced rather than merely configured. The policy in version control, plus the output of the scheduled egress test.
6. Show your retention schedule for prompts and outputs, and evidence of deletion. Deletion job logs, and the ability to delete by subject key rather than only by date.
7. Show how you would exit this arrangement. Increasingly the sharpest question, and the one that connects an isolated deployment back to third-party and operational-resilience expectations in both markets. Your answer is stronger here than for a managed service: you hold the weights and the runtime. Write it down anyway.

Anti-patterns we keep seeing

• The air gap with a hole in it. A proxy allow-list containing wildcard entries for a cloud object-store domain is not an allow-list; it is a route to most of the internet. If the allow-list has wildcards, you are on rung 2 — which may be fine, but say so in the design document rather than claiming rung 3.
• Weights on removable media with no manifest. If the artefact arrives without a signed manifest, you have imported something you cannot describe, and you have no way to detect substitution. This is the control the whole boundary exists to support.
• One shared service account for the whole application. Cheap to build, impossible to audit, and the finding will require re-architecture rather than a configuration change.
• Promoting a quantised build on the full-precision model's evaluations. Different digest, different behaviour, different model. It needs its own run.
• Prompts and completions in the general application log index. Regulated data, over-permissioned, retained under the wrong schedule, and discovered by someone else.
• Treating the weights as the only artefact. The tokenizer, chat template and generation config change outputs materially. Import and hash the whole repository.
• Letting the candidate model judge itself. You are measuring self-consistency and reporting it as quality.
• Assuming isolation equals security. An air gap removes remote exploitation from your threat model. It leaves insider risk, physical access and the supply chain of everything you imported — and it makes patching slower, which raises the value of the vulnerabilities you carry.
• No rebuild plan. If the cluster were lost, could you reconstitute it from artefacts you hold, without internet access, within your recovery objective? If nobody has tested that, you do not have a recovery objective; you have a hope.
• Building rung 3 when the written requirement was rung 1. The most expensive item on this list, and the reason the first section of this article exists.

A staged rollout that works

Sequence matters here more than in a connected deployment, because backtracking means a fresh import window. Five stages, and the first two involve no GPUs at all.

1. Write down the boundary and the driver. One page, agreed with compliance and information governance, naming the obligation, the isolation rung, the transfer mechanism, the exception process and the sign-off authority. Nothing else starts until this is signed. This is also the moment to argue for rung 1 or 2 if the driver does not genuinely demand rung 3.
2. Build the artefact pipeline before you need it. Staging zone, signing keys, manifest format, internal registry, wheelhouse or mirror, transfer procedure. Prove it end to end by importing something trivial. Teams that build this after the model is chosen import the model by hand "just this once", and that becomes the process.
3. Stand up the smallest useful model and the full observability stack together. A 7 to 9 billion parameter model on one accelerator, behind the real gateway, with the real identity propagation and the real redacted tracing. The goal is not capability; it is to prove the boundary, the audit trail and the black-hole test with something cheap to redeploy.
4. Bring the evaluation harness up before the production model. Judge model, golden sets, harness, gate, baseline run. Without a baseline you have nothing to compare the real model against, and you will end up promoting on impressions.
5. Then size and deploy the production model, and rehearse the exits. Full capacity with N+1, canary within the boundary, and — before go-live — a rehearsed rollback, a rehearsed rebuild from artefacts, and a rehearsed audit request.

Readiness check	You are ready when
Boundary definition signed	Compliance and IG have signed a page naming the driver, the rung and the exception route
Licence approval recorded	Every model in the registry has an approval reference and a hashed licence file
Signed import path proven	A non-trivial artefact has gone end to end and failed correctly when the signature was tampered with
Black-hole test passed	Every code path runs under default-deny egress with no hangs and no silent retries
Manifest verified on boot	A deliberately corrupted weight file prevents the service from starting
Per-user attribution live	You can name the individual behind any inference record from the last 90 days
Payload handling agreed	Prompts and completions have a named owner, a classification, a retention period and delete-by-subject
Egress monitored, not assumed	A scheduled outbound-connection test runs and alerts, with evidence retained
Evaluation baseline exists	A pinned run against pinned golden sets with a pinned judge, reproducible on demand
Import calendar staffed	Named owners, dated windows, and a vulnerability DB age alert that has fired at least once in test
Rollback rehearsed	The previous model digest has been restored under time pressure, not just in theory
Rebuild rehearsed	The cluster has been reconstituted from held artefacts with no external network

None of this is exotic engineering. It is ordinary engineering with the network removed, which turns every convenient default into a decision you have to make explicitly and record. That is the real character of the work, and it is why the teams who do it well are usually the ones who wrote the boundary definition first and reached for the GPUs last. If you are staffing for it, the sector-specific roles and the skills that actually get hired are covered in our guide to AI roles at banks, insurers and the NHS, and the policy backdrop across both markets is tracked in our policy news and infrastructure news sections.