Here is the concrete problem. This assertion is correct and useless:
expect(response).toBe("The refund window is 30 days.")It fails on "You can request a refund within 30 days.", which is the same answer. It passes on nothing else. Run it against a real system and it fails constantly for reasons that are not defects.
So what replaces it?
Assert on properties, not on strings
Stop asserting the output and start asserting things that must be true of the output. This is the core idea of property-based testing, applied to prose.
For the refund example, what actually matters?
- It mentions 30 days.
- It does not state a different number of days.
- It does not promise a refund unconditionally.
- It is under 100 words.
- It is in English.
Every one of those is checkable without knowing the exact wording. Together they are a stronger assertion than string equality was, because string equality only ever covered one phrasing.
The four layers, cheapest first
Order matters, and it comes back in module 8 as a cost strategy.
1. Structural checks. Does it parse? Are required fields present? Is it within length limits? Free, instant, deterministic. Run these on everything.
2. Keyword and pattern checks. Must-mention, must-not-mention, regex for formats like dates and currency. Nearly free. Covers a surprising amount of real risk, especially the must-not-mention cases: no competitor names, no unconditional promises, no leaked internal terms.
3. Similarity checks. Compare against a reference answer by embedding similarity rather than exact match. Useful when you have a known-good answer and want to detect "this drifted a long way from it" without demanding identical phrasing. Moderate cost.
4. Judged checks. Ask a model whether the output meets a rubric. Handles the genuinely subjective parts. Most expensive and least deterministic, so it goes last. Module 5 is entirely about doing this properly.
The mistake almost everyone makes is starting at layer 4. It is the most interesting layer, so it gets built first, and then the suite is slow, expensive, and itself non-deterministic. Build 1 and 2 first. They catch more real defects than you expect.
Thinking in rates
Combine layered assertions with the distribution mindset from the previous lesson and you get the actual unit of an eval suite:
Case refund-window-basic, run 10 times: 10 of 10 structural, 10 of 10 keyword, 9 of 10 judged. Last week it was 10, 10, 10.
That is a result you can act on. It says the case is broadly healthy and something may have moved on the subjective dimension, which is worth a look but is not a page-someone regression.
A single pass or fail could not have told you any of that.
What to take away
Exact-match assertions are not merely inconvenient here, they are actively harmful, because failing them constantly trains the team to ignore red.
Replace them with a stack of cheap checks that encode what actually matters, run each case enough times to see a rate rather than a sample, and reserve the expensive subjective check for the part that genuinely needs judgement.