Prompts as Code: Versioning, Testing, Shipping

A prompt is a deployed artifact with none of the usual controls

The pattern is consistent across teams. A prompt starts as a string in a pull request, gets a fix, gets another, and by month four it is four hundred lines that nobody dares touch. Somebody pastes a variant into a support channel and it becomes the real one. A product manager edits it in a vendor console at half past four on a Friday. Then a customer reports that the tone changed, and there is no answer to when, or by whom, or whether it can be put back.

None of that is a discipline failure. It happens because a prompt does not look like software. It is prose, anyone can read it, and editing it feels like editing a document rather than changing a production system. The fix is to make the tooling match the reality rather than the appearance.

Terms, used precisely

Prompt bundle

The complete set of inputs that determine a model's behaviour on a call: the instruction text, the pinned model build, the decoding parameters, the output schema, the context assembly rules and the eval suite that certified them. Versioning any one of these alone is not versioning.

Model pin

Referring to a specific dated model build rather than a floating alias, so that the artifact you tested is the artifact that serves traffic until you deliberately change it.

Eval gate

A threshold in continuous integration that blocks a prompt change from merging unless a fixed test set still passes at or above an agreed score, compared against the version currently in production.

Prompt drift

The slow accumulation of clauses added to fix individual incidents, each reasonable alone, which together produce an instruction nobody understands as a whole and nobody can safely shorten.

Canary release

Routing a small share of live traffic to a new prompt version while the rest stays on the current one, so a regression that the eval suite missed is discovered on a fraction of users rather than all of them.

Where prompts actually live, and why most of those places fail

Look at where the text physically sits before designing anything. Four of the five common locations cannot support a rollback, and two of them cannot even tell you what is currently running.

The last row is the target, and the phrase that carries the weight is without a deploy. Prompts change more often than code and usually for operational reasons rather than product ones. If every change rides a release train, people will route around the train, and the vendor console is where they route to.

Version the bundle, not the string

This is the change that makes everything else work. A prompt version that identifies only the instruction text is a version number attached to a fraction of the thing that determines behaviour, which is how teams end up with an eval result from March defending a system whose model build changed in May.

The payoff is operational. With six components pinned, rollback is switching a pointer to a previous bundle, reproduction is loading a bundle by version, and an incident review is a diff between two bundles rather than an archaeology exercise across four systems. The file below is the shape we use.

# prompts/triage/classify_ticket.prompt.yaml
# The unit of versioning is this whole file, never the instruction alone.

id: triage.classify_ticket
version: 7                       # bump on ANY change below, including whitespace
owner: "@rina"                   # a person, not a team
status: production               # draft | canary | production | retired

model:
  id: "provider/model-name-2026-05-14"   # a dated build, never a floating alias
  fallback_id: "provider/model-name-2026-02-02"
  params:
    temperature: 0
    top_p: 1
    max_output_tokens: 400       # part of the contract: raise it and outputs change

context_contract:                # what the caller MUST supply, and the cap on each
  ticket_body:   { required: true,  max_tokens: 3000, truncate: "tail" }
  customer_tier: { required: true,  enum: [free, pro, enterprise] }
  recent_tickets:{ required: false, max_items: 3, max_tokens: 600 }
  retrieval:     { index: "kb_v14", top_k: 4, max_tokens: 1800 }

output_contract:
  schema_ref: "schemas/ticket_triage.v3.json"   # schema version is part of THIS version
  on_invalid: retry_once_then_dead_letter

instruction: |
  You classify inbound support tickets. You do not answer them.
  ...
  Rule 9 was added on 2026-03-02 after ticket 88412: a refund request inside a
  bug report is a refund request. Do not remove this rule during a tidy-up.

evals:
  suite: "evals/triage.v5"       # pinned. A new suite is a new prompt version.
  gate:
    overall_pass: ">= 0.92"
    per_class_recall: { billing: ">= 0.90", security: ">= 0.98" }
    p95_latency_ms: "<= 2500"
    cost_per_1k_calls: "<= 4.00"

cache:
  stable_prefix_ends_after: "instruction"   # nothing volatile above this line
  forbidden_in_prefix: ["timestamp", "user_name", "request_id", "session_id"]

changelog:
  - v7: "added rule 9, refund inside bug report. eval 0.918 -> 0.941"
  - v6: "model pin moved to 2026-05-14 build. eval 0.907 -> 0.918, p95 -1.1s"
  - v5: "removed rule 4 as redundant. eval fell to 0.881, REVERTED in v5.1"

Testing something that is not deterministic

You cannot prove a prompt correct, and chasing proof is what stops teams from testing at all. You can bound it. An eval suite is a fixed set of cases with assertions attached, run repeatedly, producing a score with an error bar. The job is to make that error bar smaller than the regression you care about.

1. Fix the case set and version ita week, once

   Real inputs, including the ones that caused incidents, with the expected outcome recorded and a note on why. Cover the ordinary path, the ambiguous middle and the cases that should be refused or escalated. Building a golden dataset covers assembly, and the discipline that matters most is that a case is never edited to make a failing version pass.

2. Choose the weakest assertion that still catches the bugper case

   Exact match where the answer is a code or a class. Schema conformance where the output is structured. Contains or does-not-contain for a required disclaimer or a forbidden phrase. A model judge only for open text, and only with its own rubric under version control, since a judge is a prompt too and inherits every problem on this page.

3. Repeat each case and record the spreadcompute

   Even at temperature zero, results vary across provider infrastructure, so a single run per case gives you a number with an unknown error bar. Three repeats is a workable default. Record the variance, because a case that flips between runs is telling you something a mean hides.

4. Set the gate against production, not against zeropolicy

   The useful gate is a maximum allowed drop from the version currently serving traffic, plus floors on the classes where being wrong is expensive. An absolute threshold alone lets a suite decay gradually while every individual change passes.

5. Run only the suites the change touchedCI budget

   Full-suite runs on every commit get switched off within a month for cost and time. Map bundles to suites, run the affected ones on the pull request, and run everything nightly. A gate people disable is worse than no gate, because it produces the belief that changes were tested.

The size of the case set is not a matter of taste. It sets the smallest regression you are able to see at all, and that is arithmetic you can run in a minute.

Run it with your own numbers and the usual result is uncomfortable: a suite of thirty cases cannot reliably see a five point drop, which means most of the eval work being celebrated is measuring noise. The two levers are more cases and more repeats, and cases are worth more because they add coverage as well as precision. Writing evals for LLM systems goes into the assertion design, and catching model regressions covers the nightly comparison that catches what a pull request gate cannot.

Shipping a prompt change without a deploy

A prompt release is a config release. The sequence below takes minutes rather than a release cycle, and every stage produces evidence somebody can read later.

# .github/workflows/prompt-gate.yaml  (or the equivalent in your CI)
# A prompt change is a code change. It gets the same gate.

name: prompt-gate
on:
  pull_request:
    paths: ["prompts/**", "schemas/**", "evals/**"]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Step 1, structure. This fails in seconds, before you spend a single token.
      - name: Lint the bundle
        run: |
          promptctl lint prompts/                 # schema of the .prompt.yaml itself
          promptctl assert-pinned prompts/        # no floating model aliases
          promptctl assert-prefix-clean prompts/  # no volatile tokens in the cached prefix
          promptctl diff-report --base origin/main   # posts the rendered diff on the PR

      # Step 2, behaviour. Only the prompts this PR actually touched.
      - name: Run pinned eval suites
        env: { PROVIDER_KEY: "${{ secrets.PROVIDER_KEY }}" }
        run: |
          promptctl eval --changed-only \
                        --repeats 3 \
                        --report evals/out/report.json \
                        --fail-under-gate          # gate values live in the .prompt.yaml

      # Step 3, regression rather than just an absolute score.
      - name: Compare against the version on main
        run: |
          promptctl compare evals/out/report.json \
                            --against main \
                            --max-drop 0.01 \
                            --per-class-max-drop 0.02 \
                            --report-cost --report-p95

      # Step 4, evidence. The PR carries the numbers, so review is about the change.
      - name: Comment the result
        run: promptctl comment --report evals/out/report.json

Who is allowed to change a prompt

Someone who is not an engineer will need to change a prompt, and refusing that is what creates the shadow copy in the vendor console. The workable answer is not to restrict who edits, but to make the gate the same for everybody.

Give non-engineers an editing surface that writes to a branch rather than to production. The eval suite runs on their change exactly as it does on an engineer's, the result appears in plain language, and merge is blocked if the gate fails. This is more work to build than a text box, and it removes the entire category of a Friday afternoon edit nobody knew about.

One rule belongs with this. Every clause in a mature prompt was added in response to something that happened, and the clause that looks redundant is usually the one holding back a failure the newcomer has never seen. Require a comment naming the incident beside any rule added for one, and require an eval case to accompany it. The comment survives the tidy-up. The eval case survives the person.

The failure modes that appear around version twelve

Early prompt problems are quality problems. Later ones are systems problems, and they arrive in a predictable order.

• Accretion. Every incident adds a clause and nothing ever removes one. The instruction reaches a length where later rules contradict earlier ones and the model resolves the contradiction however it likes. The remedy is a periodic rewrite that keeps the eval score, not a slow trim of clauses whose purpose has been forgotten.
• The silent model swap. A floating alias resolves to a new build and behaviour moves without a commit. This is the single most common cause of a system that was fine on Friday and strange on Monday, and pinning removes it entirely.
• Cache-breaking edits. Someone moves a timestamp or a user name into the top of the system prompt. Behaviour is unchanged, the prefix cache stops matching for every request, and cost and latency rise with no code change to blame. Keep volatile values below the stable prefix and assert on it in CI.
• Copy-paste divergence. The same prompt exists in three services and gets fixed in one. Six weeks later they disagree about a policy and nobody knows which is right. One registry, referenced by id, with no local copies.
• Eval set rot. The suite stops representing production because the traffic shifted, so the gate keeps passing while real quality falls. Sample fresh production cases monthly, add them, and version the suite when you do.
• The judge that drifts. A model-graded eval is itself a prompt on an unpinned model. Pin the judge, version its rubric, and check it against human labels periodically, or you will spend a week debugging a regression that only ever existed in the grader.

Putting it in place on a system that already exists

None of this needs a rewrite. On a running system the order below reaches the useful state in about two weeks, and each step is worth having even if you stop after it.

The order matters more than the tooling. Logging the version comes before evals because without it you cannot interpret any result you gather. Pinning the model comes first because everything else is measured against a moving target until it is done. This is the practice underneath our prompt systems work, and it pairs directly with the schema discipline in structured output from LLMs, since a schema version is part of the same bundle.