What is Golden Dataset?

A golden dataset plays the role that fixed test data and expected results play for deterministic code, adapted to a system whose output varies. Each case holds an input and a description of an acceptable output: a score range, terms the answer must mention, terms it must not, and traps for known failure modes such as invented facts or unsafe content. The eval harness runs every case, grades each output against its expectations, and fails the build when a metric drops below its threshold.

The cases should cover the range of inputs the feature will meet, not only the happy path. In the public eval suite behind AssertHired's interview scoring, 65 golden cases span several answer-quality tiers, including weak, off-topic and refusal answers, and those were the tiers where the prompt's scores and the authored expectations disagreed most.

The most common mistake is writing the expected outcomes before looking at what the system does. That suite's first run failed score-in-range at 50.8 percent: the ranges came from reviewer intuition, and the model scored 15 to 25 points higher than authored. Thirty-two cases were recalibrated to the prompt's own bands. The lesson that generalises: author expectations after a small calibration run, then treat the dataset as a freeze of accepted behaviour, so any future movement means the prompt or the model changed.

A golden dataset grows over time. Every production failure worth fixing becomes a new case, ids stay stable so each case keeps its history, and changes to an expectation are reviewed like code, because loosening a range silently is the eval equivalent of deleting a failing assertion.