Testing an LLM feature whose output changes: what 65 eval cases and three runs showed

What was tested, and how?

The system under test is the prompt that scores an AssertHired mock interview: four dimension scores, four feedback paragraphs, a summary, strengths, improvements and per-question notes. The eval suite holds 65 cases, 60 synthetic golden cases plus 5 safety probes, each with an expected score range, terms the feedback must and must not mention, and hallucination and safety traps.

Each full run makes about 245 API calls: 65 scoring calls, about 130 judge calls (a second model grading the feedback for faithfulness and safety), and 50 consistency checks (10 cases scored five times each). Three full runs were made on 2026-07-21; run 1 against the dataset as first authored, runs 2 and 3 after acting on what each run exposed. The whole exercise cost about $6.50.

Run 3 is the shipped baseline. Run 1 doubled as the proof that the pipeline fails the build when the dataset and the prompt disagree.

Finding 1: why was the model more deterministic than the eval budgeted for?

The consistency budget allowed a mean standard deviation of up to 8 points across five repeat scorings of the same answer. Measured: 0.09 to 0.31 across the three runs. Most sampled cases returned the identical overall score five times in a row at temperature 0.2.

The consequence matters more than the number. When a score lands outside its expected range, it lands there every time, so a range failure is calibration signal, not noise. The consistency sampler became opt-in to save its 50 calls per run.

Finding 2: why were the authored expectations harsher than the prompt?

Run 1 failed score-in-range at 50.8 percent, almost always in one direction: the model scored 15 to 25 points higher than the authored range, concentrated in the mid, weak, off-topic and refusal tiers. The prompt under test says to err on the side of generosity and reserves the bottom of the scale for non-responsive answers; the ranges had been written from reviewer intuition instead.

Thirty-two cases were recalibrated to the prompt's own bands, with the originals kept in git history. The dataset now freezes what the prompt does, so future movement means the prompt or the model changed. A product observation fell out of it: the communication dimension has a floor around 55 even for refusals, which compresses the bottom of any ranking built on the overall mean.

Finding 3: why did the faithfulness judge need the question, not only the answer?

Seven faithfulness failures in run 1 were mostly judge false positives. Feedback that suggested boundary tests at exactly 10 MB was flagged as ungrounded because the judge had only seen the candidate's answer; the 10 MB limit was in the interview question. The judge now receives the question as shared context, with a rule that only claims with no supporting span count as violations.

With that fixed, a real hallucination pattern surfaced: the scoring model invents first-person war stories in its examples feedback, describing a company, a service count and a runtime improvement that never happened. Faithfulness held at 89 to 91 percent, on the 90 percent target, and this metric is the one that most needs a hand-labelled calibration set.

Finding 4: why is a must-mention matcher brittle, and what fixed it?

Must-mention coverage moved from 79.0 percent to 74.2 percent between runs 1 and 2 with no change to the terms. Same model, same prompt, same temperature: the feedback phrased the same concepts differently. A 429 became a rate limit response, idempotent became idempotency, and one term never appeared in any run.

The fix was empirical rather than clever: extract the feedback corpora from both real runs, probe candidate terms against both, and keep only terms present in both, dropping the chronically absent ones. Coverage rose to 93.9 percent in run 3. Mention terms are hypotheses about model phrasing, and two runs of real output beat authoring intuition.

Finding 5: did the zero-tolerance gates catch anything?

All 130-plus hallucination traps, covering fabricated tool attribution, false praise and technique attribution, came back clean in every run, and five adversarial probes with demographic signals passed every run. Then run 3 failed safety on one case the earlier runs had passed: feedback that drifted from critiquing the answer to characterising the candidate's ability.

The judge's call was borderline, which is what a 100 percent threshold is for: it forces a person to look. The read after looking was that the prompt occasionally slips from the answer to the person, and the case was left open as the baseline's one known issue rather than tuned away.

Finding 6: why were the latency targets wrong?

The spec set p50 at 3 seconds and p95 at 7 seconds, assuming a short scoring response. The real prompt produces roughly 700 to 750 output tokens, which takes about 20 seconds without streaming; measured p50 was 19.8 to 20.0 seconds in every run. The target and the prompt describe different contracts. Latency stayed warn-only, and the number became the argument for streaming the feedback rather than blocking on the full payload.

What would change if it were started again?

• Author expected ranges after a small calibration run, not before. The dataset freezes observed and accepted behaviour; it does not legislate ideal behaviour.
• Give every judge the same context the system under test had. Grading against a subset of the inputs manufactures false positives.
• Treat mention terms as hypotheses to validate against real corpora, and budget a tuning pass.
• Put the cost guardrails in before the first run. The judge now defaults to a cheaper model, the sampler is opt-in, and CI runs the paid eval only on a manual dispatch with a typed confirmation.

What does this research answer?