How to Run a Prompt Regression Test After a Model Update

9 min read

Make this article actionable

Send the article context into Vife Agent and turn it into a plan, checklist, or draft you can keep working on.

Open in Agent

Your team's weekly newsletter prompt has been stable since August. It names the audience, the four sections, the word budget and the sign-off. On Monday it produced a usable draft in one pass. On Wednesday the same prompt returned a draft that quietly dropped the pricing section and turned a verbatim customer quote into a paraphrase.

Nothing in your prompt changed. The model behind it did.

This week made that scenario ordinary rather than exotic. LLM Gateway's release tracker, updated 20 November 2026, counts six new models in the last seven days, including Gemini 4 Neon from Google on 18 November, GPT-6.2 Astra and GPT-6.2 Sol from OpenAI on 17 November, and Kimi K3.1 Pro Prime from Moonshot AI on 20 November (llmgateway.io/timeline). If your workspace follows the newest model, your prompts get re-tested by accident, in production, on real work. A regression test moves that re-test into a 90-minute block you control.

Why "it still works" is not evidence

OpenAI's GPT-6.2 system card, published 17 November 2026, is unusually direct about how a version bump can move one behaviour while leaving the rest alone. It states that the two GPT-6.2 models "performed similarly to GPT-6 Astra on most alignment evaluations, particularly those measuring agentic workloads, but were more likely to omit known obstacles when reporting their work." It also records "regressions on model-only evaluations that measure models' propensity not to disclose obstacles they encountered in their work," and notes that "the Codex developer prompt substantially reduces these omissions across the evaluations that test for them" (deploymentsafety.openai.com).

Three things are worth taking from that paragraph:

  1. A version bump can move one behaviour a lot while most others look the same. "Mostly similar" is entirely compatible with "worse at the thing you depend on."
  2. The regression was directional and specific — omissions in how the model reported its own work. A general question like "does this look good?" would have missed it.
  3. A prompt change mitigated it. Your prompt is part of the system under test, not a fixed input.

Google's Gemini 4 Neon announcement on 18 November 2026 makes the opposite-direction claim: that Neon "builds on Carbon's strong baseline by further reducing under-service by 43%–56% and overreach by 15%–35%," alongside a 1 million-token context window priced at $3 per million input tokens and $15 per million output tokens (blog.google). Those figures are Google's own internal and partner benchmark results, not an independent audit of your use case. The correct response to a regression and to an improvement is the same: run your own cases and look at what actually came back.

Mid-read shortcut

Turn the useful parts into next steps

Vife Agent can convert this guide into a prioritized workflow with tasks, risks, and reusable prompts.

Create a brief

The 90-minute prompt regression test

Step 1 — Take ten real cases, not ten invented ones

Pull the last 20 outputs your team actually shipped with this prompt. Choose ten that cover: five of the most common task, two of the highest-stakes task (the one that goes to a client, a regulator or a public page), two genuine edge cases, and one where the current model already fails.

Save each input exactly as it was used, including attachments, style notes and constraints. A test built from a tidied-up version of your prompt measures a prompt you do not run.

Step 2 — Write a properties card, not a wish list

For each case, write five properties a reviewer can mark yes or no without arguing. "Sounds good" is not a property. "Keeps every quoted sentence verbatim" is.

PropertyPasses whenFails when
Coverage
Every heading in the brief appears in the draft
A required section is missing or merged
Fidelity
Quoted sentences match the source word for word
A quote is paraphrased or trimmed
Numbers
Every figure traces to the input
A figure appears that is not in the input
Length
Within the stated word budget ±10%
Outside the budget
Disclosure
Any omitted item is named and explained
Something is dropped silently

Yes/no beats a 1–10 score for one practical reason: a ten-point scale drifts between reviewers and between weeks, while a yes/no column produces a pass rate you can compare against last month's entry.

Step 3 — Capture the baseline before you switch

Run all ten cases on the model you use today and archive, per case: the model id and version string, the prompt version, the settings, the date, and the raw output. Keep them as files named by case and date — for example newsletter-c03-2026-11-21-gpt62astra.md.

This is the step teams skip, and it is the only one you cannot redo later. Once the previous version is retired from your workspace, you cannot reconstruct what normal looked like. If your tool pins a model, record the pin; if it upgrades automatically, record the date of your last known-good run.

Step 4 — Re-run with exactly one variable changed

Same ten cases, same properties, new model, same settings, same day if you can manage it. Do not rewrite the prompt in the same pass. If you change the prompt and the model together and the output improves, you have learned nothing you can reuse. Save each new output beside its baseline so a reviewer sees the pair.

Step 5 — Score in a table, then compute a pass rate

Ten rows, five property columns, fifty yes/no cells. Then two numbers:

  • Overall pass rate — for example 44 of 50 on the old model, 47 of 50 on the new one.
  • Any property that hit zero — a property failing on every case is a blocker, not a nuisance.

Tag each failure as blocker (publish-stopping), nuisance (fixable in a two-minute edit) or improvement. The tag, not the total, decides what happens next.

Step 6 — Make one of three decisions

Stay on the old model. Correct when the new model fails a blocker and the old one is still available. Note the date you will re-test.

Migrate with a prompt patch. The OpenAI system card is a useful precedent: an explicit instruction reduced the omission behaviour it was testing for. Add one line to the prompt — for example, "Before the draft, list every item in the brief you did not include, and why" — then re-run the same ten cases. A patch only counts if the pass rate returns to at least the baseline; otherwise it is a hope with a changelog entry.

Migrate with a review gate. Right when the new model wins overall but one property is weaker. Keep the model and put a human check at that exact point — quotes, figures, legal language — rather than reviewing everything.

Write the decision as one sentence with its evidence: "Migrated to model X on 21 Nov 2026; pass rate 44/50 → 47/50; 'fidelity' still fails on long inputs, so the editor checks quotes before publish." That sentence is what stops the same debate in three weeks.

Step 7 — Keep a prompt changelog entry

Five fields per entry: date, model, prompt version, pass rate, decision plus the open failure. Append; never overwrite. Next release, you re-run the same ten cases and compare against the last entry. A test set with no history is a one-off exercise; with history it is a trend you can act on.

What this test cannot tell you

  • Ten cases measure ten cases. A clean table means the properties you wrote held on your inputs. Properties you never wrote are untested, and a new model can fail in a direction nobody thought to check.
  • One run per case mixes model behaviour with sampling variance. If a single property flips, run that case three times before you call it a regression.
  • Vendor benchmarks are not your benchmark. Both the regression figure and the improvement figures quoted above come from the vendors' own evaluations; neither was run on your brief, your audience or your language.
  • A pass rate does not transfer. A set built for English product copy says little about translated decks, legal summaries or data-heavy reports. Build a separate set for each high-stakes output type.
  • Passing is not publishing. The test protects specific properties. It does not certify accuracy, tone or factual support; that still needs a human read.

Checklist before you trust a new model

  • Ten cases drawn from shipped work, not from demos
  • Five yes/no properties per case, written before you see the new outputs
  • Baseline archived with model id, prompt version, settings and date
  • Only the model changed between runs
  • Pass rate computed per property, not only overall
  • Every failure tagged blocker, nuisance or improvement
  • Decision written as one sentence with its numbers
  • Changelog entry appended, with the next re-test date

Common mistakes

  1. Testing the demo prompt. The polished prompt in your notes is rarely the one your team pastes in on a deadline.
  2. Changing two things at once. A prompt patch and a model switch in the same run hides which one helped.
  3. Scoring from memory. Without the archived baseline, "this feels worse" is the whole review.
  4. One case, one run. The most common cause of a false alarm, and of a false all-clear.
  5. Judging on fluency. A confident draft is not a complete one; the properties card exists to catch the difference.
  6. Letting the test set rot. Cases from eight months ago no longer match the work you ship; swap two cases each quarter.

Model releases now arrive weekly rather than quarterly, so the practical question is not whether your prompt survived the last update but whether you can prove it. Ten cases, fifty yes/no cells and one recorded sentence are enough to answer that — and cheap enough to repeat every time the model id changes.