Write a System Card for Your AI Feature: A Working Template

Model card, system card, technical file: three documents, three jobs

The model card is the one everyone has read. It was introduced by Mitchell and colleagues in 2019, at Google, and its stated lineage is deliberately unglamorous: nutrition labels on food, datasheets for electronic components, disclosure documents in finance. The argument was that a trained model is a component with characteristics, and that shipping a component without stating its characteristics is a practice the rest of engineering abandoned decades ago. The original nine sections were model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats and recommendations. Seven years on, that skeleton has aged remarkably well.

What has changed is what people build. In 2019 the artefact you shipped was often the model. In 2026 the artefact you ship is almost never the model: it is a model you did not train, behind a prompt you wrote, in front of a retrieval index you built, wired to four tools, filtered by two guardrails, with a human review queue behind it and a fallback path when the provider returns a 529. Every one of those layers can fail independently, and most of the failures your users actually experience live in the assembly rather than in the weights. A model card written by a frontier lab describes none of it.

That is the gap a system card fills. And it is worth separating it cleanly from the third document in the family, the regulatory technical file, because conflating the two is the fastest route to either over-promising to a reviewer or writing three hundred pages nobody reads.

What the EU AI Act actually says, stated precisely

Two provisions matter for this discussion, and it is worth quoting their substance carefully rather than paraphrasing loosely, because loose paraphrase is how teams end up asserting things in a sales call that a buyer's counsel can correct them on.

Article 11, read together with Annex IV, covers technical documentation for high-risk AI systems. The documentation must be prepared before the system is placed on the market or put into service, and it must be kept up to date across the system's lifecycle. Annex IV specifies the categories of technical documentation that must be maintained. Note the shape of that obligation: it is a file with specified categories, maintained continuously, not a document you produce once for a launch review.

Article 13 covers transparency and provision of information to deployers. High-risk systems must be designed so that their operation is sufficiently transparent for deployers to interpret output and use it appropriately, and they must be accompanied by instructions for use that are concise, complete, correct and clear. The point that gets reversed in internal discussion more often than any other: Article 13 is an obligation on the provider, not on the deployer. If you build and supply the feature, the duty to make it interpretable and to write usable instructions is yours, and it does not transfer to the customer who switches it on.

Beyond the EU, the practical pressure is the same even where the statutory position differs. UK public-sector buyers ask about intended use, oversight and failure behaviour as a matter of course in their assurance questions. Indian financial-services procurement asks who processes what data, where it goes and who is accountable when the model is wrong. On the Indian side the DPDP Act's rules are being phased in through to May 2027, so what you will meet in a vendor questionnaire for the next while is a buyer's risk appetite rather than a settled statutory checklist — which, if anything, makes a clear document more useful, not less, because you are answering a question nobody has standardised yet.

The template you can copy

Here is the skeleton. It is deliberately Markdown, deliberately short, and deliberately ordered so that the sections a sceptical reader wants first come first. Copy it into a file called SYSTEM_CARD.md next to your feature's code, so that it moves through review with the code rather than living in a wiki nobody has opened since the launch.

# System Card: {Feature name}

## 1. Identity and version
- System name, internal ID, and the version this card describes
- Card version + date issued; link to the previous version
- Status: pilot / limited release / general availability / deprecated

## 2. Owner and contacts
- Accountable owner (a named person, not a team inbox)
- Engineering contact, security contact, escalation path for users

## 3. Intended use
- What the system is for, in one paragraph a non-engineer can read
- The specific decisions it supports and who makes the final call
- Deployment context: which markets, which user population, which channel

## 4. Out-of-scope use
- Uses the system is NOT validated for, stated as explicit prohibitions
- Populations, languages or data types it has not been evaluated against
- What happens if someone uses it out of scope (blocked / flagged / undetectable)

## 5. System architecture
- Models: provider, model ID, version, where hosted, fallback model
- Prompts: what is in the system prompt, where the template lives, who can change it
- Retrieval: sources, index build cadence, embedding model, chunking, freshness
- Tools: every tool the system can call, and what each one can write to
- Guardrails: input filters, output filters, refusal policy, rate limits
- Human review: which actions route to a human, and the queue behind them

## 6. Data
- Input data: what the system receives, from where, retained for how long
- Personal data: categories processed, lawful basis or contractual basis, residency
- Evaluation data: how the set was built, size, how it is refreshed
- Training / fine-tuning data, if you trained anything, and its provenance
- What is sent to third-party model providers, and under what terms

## 7. Evaluation
- Methodology: how the set was built, who labelled it, agreement rate
- Headline metrics, with the date measured and the system version measured
- Per-slice results: by language, region, customer segment, input length, channel
- Baseline comparison: what the system is better than, and by how much
- Known measurement limitations

## 8. Known failure modes
- For each: the condition that triggers it, observed frequency, user impact,
  current mitigation, residual risk after mitigation
- Failures found in production, not only those found in testing

## 9. Human oversight and escalation
- Which outputs a human sees before they take effect
- How a reviewer can tell the system is wrong (what the UI shows them)
- How a user or reviewer overrides, reverses or escalates a decision
- Who is on the rota, and the service level for the queue

## 10. Limitations
- Capability limits: what it cannot do regardless of configuration
- Operating envelope: volume, latency, input size, supported languages
- Dependencies that can take the feature down

## 11. Monitoring in production
- What is logged, for how long, and who can read it
- Live metrics and the thresholds that page someone
- Drift detection: what is sampled, how often, reviewed by whom
- Incident history summary and link to post-incident reviews

## 12. Change log
- Date | Version | What changed | Which layer | Evaluation re-run? | Author

A worked example: ticket triage at a services firm

A skeleton is easy to nod at and hard to fill. So here is a realistic one, filled — a support triage agent of the kind dozens of teams in Bengaluru, Pune, Manchester and London have shipped in the last eighteen months. It classifies inbound tickets, routes them, and drafts a first reply for an agent to edit. The numbers below are illustrative, not measured: they are shaped like real results from a system of this kind, and their job here is to show you what an honest metrics block looks like, not to tell you what yours will say.

# System Card: Support Triage Assistant

## 1. Identity and version
System: support-triage-assistant | Internal ID: SUP-TRIAGE
System version: 2.4.0 | Card version: 2026-09-04 | Previous: 2.3.1 (2026-07-18)
Status: general availability (IN + UK support desks)

## 2. Owner and contacts
Accountable owner: Head of Support Engineering (named in the internal copy)
Escalation: support-eng on-call rota, 24x5 across Bengaluru and Manchester

## 3. Intended use
Classifies inbound support email and web-form tickets into 14 categories,
assigns a priority, routes to a queue, and drafts a suggested first reply.
A human support agent reviews and sends every reply. The system does not
close tickets, issue refunds, or communicate with customers unassisted.

## 4. Out-of-scope use
- Not validated for voice or chat transcripts (email + web form only)
- Not validated for languages other than English and Hindi
- Not to be used for complaints that reference regulatory escalation,
  bereavement, or vulnerable-customer indicators: these bypass drafting
- Not a decision system for account closure, credit, or fraud outcomes

## 5. System architecture
Primary model: hosted LLM, provider version pinned, EU + IN regions
Fallback: smaller model on classification only; drafting disabled on fallback
Prompts: one system prompt per task; versioned in repo, change requires review
Retrieval: help-centre articles + last 90 days of resolved tickets;
  index rebuilt nightly at 02:15 IST; embeddings model pinned
Tools: read-only CRM lookup; queue-assignment write; draft-save write.
  No send, no refund, no account-mutation tool is exposed.
Guardrails: PII redaction on input; profanity and PII filter on output;
  vulnerability-keyword classifier routes straight to a senior agent
Human review: 100% of drafts reviewed before send; routing is automatic
  but reversible by any agent in one click

## 6. Data
Input: ticket body, subject, customer tier, last 3 ticket summaries
Personal data: name, email, account ID. Payment data is redacted pre-model.
Retention: prompts and completions 30 days, then deleted; labels retained
Evaluation set: 1,400 historical tickets, stratified by category, language,
  region and length; hand-labelled by two senior agents; refreshed quarterly
Third parties: ticket text (post-redaction) is sent to the model provider
  under the enterprise agreement; no training on our data

## 7. Evaluation (measured 2026-08-29 on v2.4.0 — see table in this card)
Methodology: 1,400-ticket held-out set, two labellers, adjudicated
  disagreements; inter-labeller agreement 0.89 before adjudication
Headline: routing accuracy 91.2%; draft accepted without edit 46%;
  draft accepted with minor edit 38%; draft discarded 16%
Baseline: previous keyword-rule router routed 71.4% correctly

## 8. Known failure modes
FM-1 Multi-issue tickets. A ticket raising two unrelated problems is routed
  on the first issue only. Observed in ~6% of tickets. Mitigation: agents
  are prompted to split. Residual risk: second issue delayed by one hop.
FM-2 Hinglish and code-mixed input. Classification accuracy drops on
  code-mixed Hindi-English. Mitigation: confidence threshold routes these
  to manual triage. Residual risk: higher manual load on the IN desk.
FM-3 Confident drafts on stale policy. Retrieval can surface a superseded
  help-centre article. Mitigation: nightly rebuild, article expiry dates
  surfaced in the draft UI. Residual risk: same-day policy changes.
FM-4 Long forwarded threads. Above ~8,000 tokens the draft starts replying
  to the oldest message. Mitigation: truncate to the newest 3 messages.
FM-5 Tone on angry tickets. Drafts are over-formal when sentiment is
  strongly negative. Mitigation: sentiment flag switches the draft template.

## 9. Human oversight and escalation
Every outbound reply is sent by a human. The draft UI shows the retrieved
sources, the confidence band, and a diff against the template. Any agent can
reject a routing decision; rejections are logged with a reason code and
reviewed weekly. Vulnerability-flagged tickets skip drafting entirely.

## 10. Limitations
English and Hindi only. Email and web form only. Degrades above 8,000 tokens.
Hard dependency on the model provider; on provider outage the system falls
back to classification-only and drafting is disabled, not degraded.

## 11. Monitoring in production
Logged: input hash, category, confidence, route, agent action, edit distance.
Alerts: routing accuracy on the daily audit sample below 85%; draft rejection
rate above 25%; retrieval index age above 36 hours.
Drift: 50 tickets a day sampled and hand-checked by a rotating senior agent.

## 12. Change log
2026-09-04 | 2.4.0 | Sentiment-switched draft template | prompt | re-run: yes
2026-07-18 | 2.3.1 | Index rebuild moved to nightly    | retrieval | yes
2026-06-02 | 2.3.0 | Added vulnerability classifier    | guardrail | yes

Read that back and notice what it does. It tells a buyer in ninety seconds that no money moves, no email sends itself, and a human is in the loop on every outbound message. It tells an engineer joining the team which layer to look at when a complaint arrives. And it tells an auditor that the team knows its own weak spots by name and number. None of that is marketing copy — and that restraint is the point. A system card is a technical disclosure document, and the moment it starts selling, it stops being evidence.

The evaluation section: report slices, not averages

This is the section most teams write defensively, and defensive writing is what makes it useless. The instinct is to find the largest plausible number and put it at the top. The discipline that makes the section worth reading is the opposite: report the number that a reader would be annoyed to discover later.

The mechanism to internalise is that aggregate accuracy hides exactly the failures that matter. A slice is a subset of your traffic that shares a property which could plausibly change performance: a language, a region, a customer segment, an input length band, a channel, a time of day, a document type. If your system is 91% accurate overall, that number is an average over slices, and averages are dominated by whichever slice supplies the most volume. The slice that supplies the least volume is very often the one attached to the most risk — a minority language, a regulated customer type, the long tail of unusual documents.

The headline says 91.2%. The table says that one in twenty-five of your tickets is handled by a system that is right three times in four. Those are the same system. A reader who is given only the first number and later discovers the second will not conclude that you were unlucky; they will conclude that you knew. Publish the slices and the conversation becomes about the mitigation, which is a conversation you can win.

Three further rules make the section credible. State the date and the system version every number was measured on, because an undated metric is an assertion rather than a measurement. Describe how the evaluation set was built and who labelled it, including the agreement rate between labellers, since a set built by one engineer in an afternoon and one adjudicated by two domain experts deserve different amounts of trust. And name the baseline you are better than — a rules engine, a previous model version, an untrained human — because a number with no comparator communicates nothing. If your evaluation set is still a handful of examples in a notebook, the groundwork is worth doing first: building a golden set and an LLM-judge suite is the prerequisite, and mining your production logs for error analysis is how you find the slices that deserve a row in the table.

The failure modes section: write down what breaks

This is the section that separates a card written by a team that runs the system from a card written by a team that launched it. It is also the section people are most reluctant to write, on the theory that naming your weaknesses arms your critics.

The theory is wrong, and the reason is worth being explicit about. The reviewer on the other side of the table has read a number of these documents. They are not assessing whether your system has failure modes; they assume it does, because every system does. They are assessing whether you know what yours are. A card that names five failure modes, each with a triggering condition, an observed frequency, a mitigation and a residual risk, reads as a team that has instrumented its own product and looked at the results. A card with a blank failure modes section reads as a team that has not looked — and the reviewer's next move is a much longer questionnaire designed to find out what you missed.

Write each entry in four parts: the condition that triggers it, how often you see it, what the user experiences, and what you have done about it. Add the residual risk that remains after the mitigation, because a mitigation with no residual risk is usually a mitigation that has not been tested. And include failures found in production, not only failures found in testing. A card whose failure modes are all the tidy kind you would discover in a test harness tells an experienced reader that the production ones have not been catalogued yet.

Versioning: the engineering events that force a reissue

Documentation must be refreshed when the system is retrained, fine-tuned or modified, and when the operational context, the datasets or the performance characteristics change. That is the principle. The practical problem is that in an assembled system, "modified" covers a dozen things that do not look like modifications to the person doing them — nobody thinks of a prompt tweak as a system change until the numbers move. So convert the principle into a list of concrete engineering events, and put that list where the engineers are.

A quarterly review on top of the event triggers is worth having, because some drift belongs to no single event — the index quietly ages, the traffic mix shifts, a downstream system starts sending a document type nobody anticipated. But the events are what keep the card honest between reviews. If you already publish provenance or content-marking signals alongside outputs, the same discipline applies there: machine-readable content marking is a claim about your pipeline, and a claim about your pipeline goes stale the moment the pipeline changes.

Who reads it, and what each reader wants

One document, four audiences, and they want genuinely different things. Writing for all four at once is what produces the bland, unreadable card — the fix is to accept that different sections are load-bearing for different readers and to make each one strong rather than to average them into mush.

Your own engineers want sections 5, 10 and 11: the architecture, the limits and what is logged. For them the card is the fastest onboarding document in the repository and the first thing to open at 2am when something is wrong. If your card is not useful to the person debugging, it is a compliance artefact rather than an engineering one, and it will rot.

Procurement and security reviewers want sections 3, 4, 6 and 9: intended use, out-of-scope use, data, and human oversight. UK public-sector and enterprise reviews tend to open with what the system is for, what it must not be used for, and who is accountable when it is wrong. Indian BFSI vendor due diligence tends to open with the data: which categories are processed, which third parties receive them, where they are stored, how long they are kept. Both sets of questions are answerable from the same card if you wrote section 6 properly. The teams that suffer in these reviews are not the ones with weak systems — they are the ones re-deriving the answers from memory in every cycle, with a slightly different story each time.

End users and deployers want intended use, limitations and the escalation path, in language that does not require an ML background. This is the constituency Article 13 is written for, and it is the one most often skipped: a card written entirely in the register of an internal design document does not make a system's operation sufficiently transparent for the person operating it, however complete it is.

Auditors want the change log and the evidence trail — dates, versions, what was re-measured and when. The change log is the cheapest section to maintain and the one that does the most work under scrutiny, because it converts a set of claims into a history. If you are the person in the organisation who ends up owning all of this, it is becoming a recognised job in its own right: the AI governance engineer career path is what that role looks like when it is done by an engineer rather than a lawyer.

Five ways system cards go wrong — and where to publish yours

Marketing language. The single most common failure. "Industry-leading accuracy", "enterprise-grade safety", "state of the art" — none of these are measurements, and a reviewer reading them concludes, correctly, that the document was written by the wrong department. A system card is a technical disclosure document. If a sentence would survive unchanged in a landing page, delete it.

Aggregate-only metrics. One headline number, no slices, no dates, no baseline. It is not that the number is wrong; it is that it is unfalsifiable, and an unfalsifiable claim does not build trust with anybody whose job is verification.

The stale card. Written for launch, never touched again, describing a version of the system that stopped existing four prompt changes ago. This is worse than having no card, because a card with a date on it is a representation about the current system, and a representation that is no longer true is a liability rather than an asset.

Copying the lab's model card. Pasting a frontier lab's model card into your documentation and calling it yours. It documents a component you did not build and says nothing about the prompts, retrieval, tools or oversight that constitute your actual product. Cite it in section 5 and write your own.

Omitting the human review path. A card that describes what the model outputs but not who checks it, how they can tell it is wrong, or how a user overrides it. Section 9 is the section buyers read most closely, because "a human reviews this" is the mitigation that most often makes a use case acceptable to them in the first place. If you are designing that path rather than documenting an existing one, the queue behind the escalation is where the real engineering sits — and if the system calls tools, the permission boundary around those tools belongs in section 5 in detail.

Where the card should live, and whether to publish it

Keep the source of truth in the repository, next to the code, in version control, reviewed in the same pull requests. That is what keeps it current. Then decide separately what to expose: an internal copy with named individuals, infrastructure detail and unmitigated risks, and a public or buyer-facing copy that redacts the personnel and the attack surface but keeps the intended use, the limitations, the slice metrics and the failure modes. Those redactions are legitimate; deleting the failure modes for the public version is not, and a reader who sees a public card with no weaknesses learns something about you that you did not intend to teach.

On whether to publish at all: for a small team selling into enterprise or public-sector buyers, publishing is close to free credibility. Large vendors have compliance departments and reference customers; you have neither, and a specific, dated, honest document is the cheapest substitute either market accepts. It also changes the shape of your sales conversations, because the buyer's technical reviewer arrives having already read the answers to their first twenty questions and can spend the call on the twenty-first. And where a system is subject to the EU transparency obligations for certain interactions, the card is a natural companion to the user-facing disclosure work described in implementing Article 50 transparency — one document for the buyer, one signal for the user, both saying the same thing.

Start with the template. Fill it for the feature you already shipped rather than the one you are planning, because the shipped one is where the gaps are, and the gaps are the point. My own rule of thumb — and it is a rule of thumb, not a benchmark — is that a first pass on an existing feature takes a focused day for someone who knows the system, and that the evaluation section is where the day actually goes, because writing it honestly usually surfaces a slice nobody had measured. That surfacing is worth more than the document.