How to Build an AI Review Rubric With Yes/No Questions
Make this article actionable
Send the article context into Vife Agent and turn it into a plan, checklist, or draft you can keep working on.
A bakery ships 40 product photos, 12 short clips and 8 one-page menus every month. The review checklist says the set must look “clean, on brand and readable.” Two people open every file. By the fortieth photo nobody is reading the checklist any more — they are pattern-matching on fatigue.
That is the gap a decision-style model is meant to close. BenchLM's release tracker lists HeyGen Decision 1 on November 13, 2026, and the model documentation describes an endpoint that takes named questions about text, images or video and returns a probability for each answer (BenchLM, HeyGen docs). Google's Gemini 4 Neon, published November 18, 2026, is described as reading uploads such as spreadsheets, PDFs and photos and turning them into structured deliverables (Google blog).
Neither of those facts makes your checklist work. A model that can answer hundreds of named questions is only as good as the questions you write. Most review checklists are written as adjectives, and adjectives produce probabilities that sit near the middle forever. This guide turns a human checklist into a set of yes/no questions an AI can answer, with a banding policy and a calibration step so you know when to trust the answer and when to open the file yourself.
Step 1 — Start from the decision, not the adjective
Before rewriting a single line, write down what you actually do when a check fails. Every checklist line should map to one of three actions: ship it, fix it, reject it.
- “Looks clean” maps to nothing. Delete it or replace it.
- “Contact shadow is missing” maps to “fix it.”
- “Label is unreadable at thumbnail size” maps to “reject it, re-crop.”
If a line cannot change a decision, it is decoration. This filter usually cuts a 25-line checklist to 8–10 lines, which matters: every question you keep costs attention when you read the results.
Turn the useful parts into next steps
Vife Agent can convert this guide into a prioritized workflow with tasks, risks, and reusable prompts.
Step 2 — Make each question atomic
A question an AI answers well has one property, one subject and one frame of reference.
| Weak | Why it fails | Atomic rewrite |
|---|---|---|
Is the photo clean and on brand? | Two properties, no evidence rule | Is the product the only object in the frame apart from the surface it sits on? |
Is the shadow right? | “Right” has no reference | Is there a contact shadow under the product, on the same side as the key light? |
Is the label readable? | No size or distance | Can the brand name on the label be read in a 400-pixel-wide version of this image? |
Three rules keep questions atomic:
- No “and.” Split compound checks. If both must pass, ask both.
- One reference per question. “Same as the reference frame” is answerable; “consistent” is not.
- A real “no.” If every asset in the batch would answer yes, the question is not discriminating. Drop it.
Vendor mechanics reward this discipline. HeyGen's documentation describes three question types: a yes/no statement scored from 0 to 1, a choice from a list you define, and an ordered score of up to 10 levels, with 1–400 named questions in a single request (HeyGen docs). Keep yes/no for pass/fail checks. Reserve the score type for genuinely gradable properties — label legibility, crop tightness — where you need a 1–5 and not a verdict.
Step 3 — Write the scope line for every question
A yes/no question without a scope line invites the model to infer. Two sentences fix it:
- Evidence: “Answer from the pixels in this frame only.”
- Missing evidence: “If the label is not visible, answer no.”
The second line is the one people skip, and it is the one that makes a rubric repeatable. It also exposes checks your photography cannot support: if the label is out of frame in 30% of the set, that is a shooting problem, not a review problem.
Scope also has to respect input limits. HeyGen's documentation states up to 50 images or 10 videos per request, with each video capped at 60 seconds and read from sampled frames (HeyGen docs). For a 45-second clip, write the question against a time window — “in the first three seconds, is the product fully in frame?” — instead of assuming the model watched every frame.
Step 4 — Set three bands, not one threshold
A single cutoff turns a probability into a coin flip with extra steps. Use three bands instead:
- 0.85 and above: pass, no human read.
- 0.15 and below: fail, route to the fix queue.
- Between: a human opens the file.
The numbers are your policy, not a vendor guarantee, and they should reflect which error is more expensive. For a public storefront, a false pass (a bad photo ships) costs more than a false fail (someone re-checks a good photo), so widen the review band to 0.2–0.9 in the first month. Tighten it only after calibration shows the model agrees with you.
Step 5 — Calibrate on 20 items you have already judged
Before a batch runs unattended, take 20 assets whose verdicts you already know and run the rubric against them.
- Score each question separately, not the asset as a whole.
- Record agreement per question: how often the model's pass/fail matched yours.
- Rewrite or delete any question under roughly 70% agreement. That threshold is yours to set.
- Re-run the rewritten question on the same 20 items before trusting it on 2,000.
Do not read the raw probability as a correctness score. HeyGen's documentation is explicit that the returned confidence is “not a calibrated probability that the answer is correct” (HeyGen docs). Calibration against your own labels is the only thing that tells you what 0.7 means in your workflow.
Step 6 — Route the uncertain band to a stronger model
The economics of review are volume, not difficulty. Most assets are obviously fine or obviously broken; a small share needs judgment. Split the work along that line instead of sending everything to the largest model.
Microsoft's November 17, 2026 announcement of MAI Code 2 and Code 2 Flash is a useful illustration of how vendors position models by task shape rather than by a single quality ranking: Code 2 Flash is described as built for “high-volume work such as code explanations, targeted edits, routine fixes, and rapid iterations,” while the company states the models are “explicitly not yet tuned for harder problems, like fundamental science research and very long-horizon agentic work” (Microsoft AI). Read that as a routing hint, not a scoreboard: high-volume, narrow checks belong on a cheap fast model; anything that needs cross-asset reasoning does not belong in the batch at all.
Cross-document questions are their own category. Gemini 4 Neon is described with a 1 million-token context window at $3 per million input tokens and $15 per million output tokens, and Google says it reads spreadsheets, PDFs and photos “more precisely” (Google blog). That makes questions like “does the price on page 3 contradict the price on page 9?” practical in one pass — but the pricing and capability statements are Google's, published on the launch day, and have not been independently verified here.
Step 7 — Version the rubric and log every answer
A rubric that changes silently is not a measurement. Log five fields per asset: question id, asset id, probability, band, and the human decision. Add a rubric version, and bump it whenever a question's wording changes.
Two rules follow: never compare pass rates across rubric versions, because a wording change moves numbers more than a process improvement does; and keep the human overrides, since twenty of them are a better source of new questions than any brainstorm.
A worked example: 12-photo bakery set
Six questions, run per image:
- Is the product the only object in the frame, apart from the surface it rests on?
- Is the product's contact shadow visible, and on the same side as the key light?
- Can the brand name on the label be read at 400 pixels wide?
- Is the background free of text and logos?
- Is the product's color within one shade of the approved sample in the brand folder?
- Is the frame free of hands, packaging tape and price stickers?
Questions 1, 2, 4 and 6 are pure yes/no. Question 3 is the borderline case — run it as yes/no first, and only add a 1–5 score if you find yourself wanting to distinguish “readable” from “comfortably readable.” Question 5 needs a reference image attached, otherwise it is an adjective in disguise.
Checklist before you run a batch
- Every question maps to a ship / fix / reject action.
- No question contains “and.”
- Each question has an evidence line and a missing-evidence line.
- The batch fits the input limits: 50 images or 10 videos per request, 60 seconds per video.
- Time-windowed questions are written for clips, not for whole videos.
- Three bands are set, with the review band wide in month one.
- 20 known items were scored and per-question agreement was recorded.
- The rubric has a version number and the log captures human overrides.
Common mistakes
Asking for a verdict instead of a fact. “Is this good enough to publish?” invites the model to guess your standards. Ask the observable property and apply your own standard to the probability.
Mixing subjects in one question. “Is the product sharp and the background clean?” fails twice as often and tells you half as much.
Treating the middle band as a nuisance. The 0.2–0.85 band is where the rubric is honest about its own limits. Route it to a person, and use those files to improve the questions.
Calibrating on the batch you are shipping. Twenty labeled items from last month are a test set. The current batch is not.
Forgetting the cost of the review itself. A 40-question rubric on 2,000 assets is 80,000 judgments. Ten atomic questions you act on beat forty nobody reads — the paper checklist's failure mode, just faster.
What the sources do and do not claim
The dates, question types, limits and quotes above come from the linked pages read on November 19, 2026. Benchmark and pricing statements from Google and Microsoft are the companies' own launch-day claims and were not independently verified. HeyGen's documentation does not state a public price for Decision 1, so treat cost planning as an open item. What the sources do support is narrower and more useful: these systems return per-question probabilities, the confidence value is not a calibrated accuracy score, and the quality of the output depends on how precisely you wrote the question. That is the part you control.

