Prompting for Indic Languages and British English in One App

Two problems that look like one

Teams building for both markets usually file this work under "internationalisation" and hand it to whoever is least busy. That framing hides the fact that the two halves have almost nothing in common.

Serving Hindi, Tamil or Bengali is an economics and capability problem. The model can probably do the task, but the same meaning consumes several times more tokens, which raises your cost, eats your context window and lengthens your latency. No amount of prompt engineering fixes a tokeniser.

Serving British English is a control problem. The model can produce perfect en-GB output, and will still drift back towards American spelling over a long generation because the underlying distribution pulls that way. It costs nothing extra and fails constantly.

They need different solutions, different measurements and different tests. Treating them as one workstream is why so many dual-market products end up with an expensive Hindi experience and a subtly American British one.

Part one: the tokeniser tax

Subword tokenisers are trained on a corpus. If that corpus was overwhelmingly English, the learned vocabulary contains few pieces that correspond to Indic morphemes. When Devanagari, Tamil or Malayalam text arrives, byte-pair merges fail, and the text fragments into very short units — in the worst case, individual bytes.

The scale of this has been quantified. Work on the cross-lingual cost of subword tokenisation for Indian languages reports that under cl100k_base, the vocabulary used by several widely-deployed models, Indian languages carry an average tokenisation tax of roughly 8x relative to English, rising to around 13x for Malayalam. Other analyses put Hindi and Bengali nearer 5x. The spread between those figures is not a contradiction; it is a function of which tokeniser, which corpus and which measurement, which is exactly why you must measure your own.

Two things follow immediately, and both are more serious than the headline cost figure.

• Your context window shrinks by the same multiple. A 200,000-token window holds a great deal of English and a much smaller quantity of Malayalam. Retrieval systems tuned on English chunk sizes will silently under-retrieve for Indic content.
• Your latency rises. Output tokens are generated sequentially. A response that is 5x more tokens takes roughly 5x longer to stream, which is a user-experience problem long before it is a cost problem. If you have a latency budget, Indic output will blow through it.

Measure it on your own corpus, in ten minutes

Do not take any published multiplier and apply it to your product. Run this against a representative sample of your actual traffic.

import tiktoken

# Use the encoding your production model actually uses.
enc = tiktoken.get_encoding("o200k_base")

SAMPLES = {
    "en": "Please confirm your appointment for Tuesday at half past three.",
    "hi": "कृपया मंगलवार को साढ़े तीन बजे अपनी नियुक्ति की पुष्टि करें।",
    "ta": "செவ்வாய்க்கிழமை மூன்றரை மணிக்கு உங்கள் சந்திப்பை உறுதிப்படுத்தவும்.",
    "bn": "অনুগ্রহ করে মঙ্গলবার সাড়ে তিনটায় আপনার অ্যাপয়েন্টমেন্ট নিশ্চিত করুন।",
}

base = len(enc.encode(SAMPLES["en"]))
for lang, text in SAMPLES.items():
    n = len(enc.encode(text))
    words = len(text.split())
    print(f"{lang}: {n:4d} tokens  "
          f"{n/base:5.2f}x English  "
          f"{n/max(words,1):5.2f} tokens/word")

The tokens per word column is the number to track over time. It is called fertility, and it is the tokeniser-independent way to compare. English on a modern vocabulary sits around 1.3 to 1.5. Purpose-built Indic tokenisers such as the one in Sarvam-1 have been reported at roughly 1.4 to 2.1 for Indic scripts, which is close to English parity. An English-first vocabulary handling Devanagari can be several times that.

Five things that actually reduce the tax

1. Choose the model partly on its tokeniser. Newer multilingual vocabularies materially reduce the Indic tax — one reported figure is around a 73 per cent reduction in the average tax when moving from an English-first vocabulary to a multilingual one. This is a bigger lever than any prompt change, and it is a procurement decision rather than an engineering one.
2. Keep instructions in English, content in the user's language. Your system prompt is static and repeated on every single call. Writing it in Hindi multiplies your fixed overhead on every request for no capability gain. Put the instructions in English and let the user's text and your output be in the target language.
3. Cache aggressively. A high-fertility prompt is exactly the case where prompt caching pays for itself fastest, because the cached prefix is where the token bloat lives. If you serve Indic content and are not caching, that is the first fix.
4. Budget per script, not per request. If your unit economics assume a flat cost per conversation, Tamil and Malayalam users will be structurally unprofitable and you will not see it in the aggregate. Break your cost per task down by language.
5. Consider a specialist model for Indic-heavy paths. Indian foundation models trained with Indic-first tokenisers — the family we covered in Sarvam's multilingual stack is the obvious example — can be dramatically cheaper per unit of meaning even where a frontier model scores higher on a benchmark. Cost per correct answer is the metric, not accuracy.

Part two: keeping British English British

The second problem has no cost attached and irritates users far more than teams expect. Research triangulating variety bias in foundation models finds that orthographic variants cluster towards American-preferred spellings, while vocabulary-level differences are somewhat more balanced but still lean the same way. In plain terms: ask for British English and you will mostly get it, until the model has generated a few hundred tokens and the prior reasserts itself.

For a UK-facing product this matters commercially. A financial services firm in Leeds sending customers a letter that says "authorized" and "check your balance" reads as offshore and unbranded, and in regulated communications it can trigger review cycles that cost more than the AI feature saves.

The pattern that holds: a dedicated, redundant style block

A single sentence buried in a paragraph of instructions does not survive a long generation. What works is a separate, clearly delimited block at system level, with the hallmark spellings shown rather than described.

<output_style>
language: en-GB
variety: British English (Oxford spelling: NO — use -ise, not -ize)

REQUIRED SPELLINGS (use these forms, never the American ones):
  organise, optimise, realise, analyse, apologise, recognise
  colour, behaviour, favour, labour, honour
  centre, metre, theatre, litre
  licence (noun) / license (verb)
  practice (noun) / practise (verb)
  programme (a broadcast or scheme) / program (software only)
  travelled, cancelled, modelling, labelled
  whilst is acceptable; gotten is not

VOCABULARY:
  mobile (not cell), lift (not elevator), post (not mail),
  autumn (not fall), CV (not resume), holiday (not vacation)

FORMATS:
  dates: 22 August 2026  (never 08/22/2026)
  currency: £1,250.00
  time: 24-hour for schedules, 12-hour with "pm" in prose
  spelling of numbers: one to ten in words, 11+ in numerals
</output_style>

Three design choices in that block are doing the work, and each is worth stating explicitly.

First, it is delimited. A tagged block is materially more robust than prose instructions, because it survives being surrounded by other content and is easy for the model to re-attend to during a long generation.

Second, it shows rather than tells. "Use British spelling" is an abstraction the model must resolve on every word. A list of the specific forms is a set of concrete anchors, and the guidance from prompt-engineering practice is consistent on this point: include the hallmark spellings themselves.

Third, it disambiguates the genuinely tricky pairs. Licence/license and practice/practise are the two that native British writers get wrong, and models get them wrong constantly. Programme versus program is the one that catches technical products, because software is the exception.

Then verify deterministically, because prompting alone is not enough

The style block reduces the failure rate substantially. It does not eliminate it, and for regulated or brand-sensitive output you need a check that does not involve a model at all.

US_TO_GB = {
    "organize": "organise", "organized": "organised",
    "optimize": "optimise",  "optimized": "optimised",
    "analyze": "analyse",    "analyzed": "analysed",
    "recognize": "recognise","realize": "realise",
    "color": "colour",       "behavior": "behaviour",
    "favor": "favour",       "labor": "labour",
    "center": "centre",      "meter": "metre",
    "theater": "theatre",    "liter": "litre",
    "traveled": "travelled", "canceled": "cancelled",
    "modeling": "modelling", "labeled": "labelled",
    "defense": "defence",    "offense": "offence",
    "gotten": "got",
}

def gb_violations(text: str) -> list[tuple[str, str]]:
    """Return (found, expected) pairs. Word-boundary matched, case-insensitive."""
    import re
    hits = []
    for us, gb in US_TO_GB.items():
        if re.search(rf"\b{us}\b", text, flags=re.IGNORECASE):
            hits.append((us, gb))
    return hits

Wire this into two places. In CI, run it across your golden outputs so a prompt change that weakens the style block fails the build — the pattern from our guide to evals in CI. And at runtime, for any output that reaches a customer unedited, use it as a gate: log a violation, and either repair it mechanically or route the generation for a second pass.

The case nobody plans for: code-mixing

Real Indian users do not write in clean Devanagari or clean English. They write Hinglish, Tanglish and everything between — Romanised Hindi with English nouns, English syntax with Hindi discourse markers, single messages that switch script mid-sentence. If your product has a text input in India, you are receiving this already, whether or not you have designed for it.

The common failure is a well-intentioned normalisation step that transliterates Romanised Hindi into Devanagari before the model sees it. This is almost always worse than doing nothing. Transliteration is lossy, ambiguous, and the frontier models generally handle Romanised Hindi perfectly well as-is. The pipeline destroys information the model could have used.

Retrieval is where code-mixing does real damage, because a Romanised query will not match a Devanagari document under most embedding models, and your recall silently collapses for exactly the users who most need it. If retrieval is in your path, our guides to multilingual RAG and cross-lingual retrieval and choosing an embedding model on your own benchmark cover the mechanics.

The three code-mixing failures worth testing for

Code-mixing does not fail in one way, and a single test case will not catch all of it. Three distinct failure modes recur often enough to deserve dedicated cases in your evaluation set.

Script-flipping in the output. The user writes in Hinglish; the model replies in Devanagari, or begins in English and switches mid-response. This is jarring rather than incorrect, and users read it as the product being broken. The fix is to state the output script explicitly rather than leaving the model to infer it from the input, because the input is by definition ambiguous.

Entity mangling across scripts. Names, place names and product names frequently survive a Devanagari response in Roman script, and vice versa. If your pipeline does anything downstream with those entities — a database lookup, a match against a customer record — the script mismatch breaks it silently. Test with real names from your own data rather than with dictionary words.

Numeric and date convention drift. Indian users write lakh and crore, use the two-two-three digit grouping (12,34,567 rather than 1,234,567), and expect DD/MM/YYYY. Models trained predominantly on Western text default away from all three, and will sometimes convert a figure the user wrote in lakhs into millions without saying so. That is a correctness bug with financial consequences, and it belongs in your evaluation set as an explicit case, not as an assumption.

Each of these takes about ten minutes to add as a test case and none of them appears in a translated evaluation set, because a translation of an English case still has Western conventions baked into it. This is the concrete argument for hand-building locale cases rather than machine-translating your existing ones.

One template, small locale blocks

The maintainable architecture is a single prompt template with a compact injected locale block. Parallel full prompts per market always drift apart, usually within a quarter, and the drift is invisible until a customer reports it.

Layer	Shared across markets	Locale-specific
Task instructions	Yes — one copy, in English	No
Tool and schema definitions	Yes	No
Output style block	Structure only	Variety, spellings, date and currency format
Few-shot examples	One shared set for task shape	One or two in-locale examples appended
Evaluation set	Shared task cases	Per-locale cases, scored separately
Model choice	Default tier	May differ where the tokeniser justifies it

Evaluating this properly

The trap here is aggregate scoring. A product that is 94 per cent accurate overall can be 97 per cent on English and 71 per cent on Tamil, and the aggregate number will never tell you. Two rules prevent this.

Score every locale separately, always. Never report a single accuracy figure across languages. Your dashboard should have a row per locale, and a regression in one should fail the build even if the mean improves.

Build locale-specific failure cases, not translations. Translating your English evaluation set into Hindi tests translation, not your product. The cases that matter are the ones that only occur in that locale: Indian date formats and the lakh and crore number system, mixed-script input, honorific registers in Tamil, UK postcodes and National Insurance formats, dates written 22/08/2026 rather than 08/22/2026. A hundred hand-built locale cases beat a thousand machine-translated ones.

Add a third, cheap check specifically for the British English problem: run the deterministic spelling scanner over every generated output in your evaluation run and report the violation rate as a first-class metric alongside accuracy. It costs nothing to compute and it is the only way to notice that a model upgrade has quietly made your en-GB output more American.

A checklist to work through

1. Measure token fertility for every language you serve, against every model you route to.
2. Break your cost-per-task reporting down by language and check that no locale is structurally unprofitable.
3. Move instructions to English; keep only content in the target language.
4. Turn on prompt caching — the high-fertility prefix is where it pays best.
5. Add a delimited output-style block with hallmark spellings shown, not described.
6. Add a deterministic spelling scanner to CI and to your runtime gate.
7. Declare code-mixed input as expected; stop any transliteration pre-processing.
8. Split every evaluation metric by locale and build genuine per-locale failure cases.
9. Re-run the whole checklist after any model change. Tokenisers and priors both move.

None of this is difficult. Almost all of it is skipped, because both problems are invisible from the inside of an English-speaking engineering team looking at one aggregate number. Making them visible is most of the work.

Further reading: The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages and Which English Do LLMs Prefer? Triangulating Structural Bias Towards American English in Foundation Models.