AI Quality Engineering
How to test LLM and agent systems as a QA engineer. 31 lessons across 10 modules: non-determinism, golden datasets, assertions for fuzzy output, LLM-as-judge and validating the judge, red teaming for prompt injection, CI regression gating, cost control, and an honest walkthrough of a real eval run whose headline metric came back at 50.8 percent. Anchored to a public, production-grade eval suite you can clone, run, and point an interviewer at.
See It In Action
Preview
Course curriculum
10 modules. Every lesson is a guided, narrated video.
01What Breaks in AI Systems3 lessons
- Welcome, and who this is for
- The failure modes that have no analogue in normal software
- Why this job is going to QA people
02Non-Determinism3 lessons
- The assumption your entire test strategy is built on
- Where the variance actually comes from
- Testing when you cannot compare to a fixed string
03Building a Golden Dataset3 lessons
- What a golden dataset is, and what it is not
- Building your first one from real traffic
- Curating it so it keeps finding things
04Assertions for Fuzzy Outputs3 lessons
- Must-mention and must-not-mention
- Score ranges, thresholds, and why a single number lies
- Writing assertions that survive a model upgrade
05LLM as Judge3 lessons
- When a judge is the right tool, and when it is a cop-out
- Writing a judge prompt that is not just vibes
- Testing the judge itself
06Red Teaming4 lessons
- Prompt injection, direct and indirect
- System prompt exfiltration
- Output format hijacks and why they are a security bug
- Running a red team suite you can rerun
07Drift Detection and Regression Gating3 lessons
- What drift actually looks like in production
- Regression gating in CI
- Choosing what blocks and what only warns
08Cost Control3 lessons
- Why the expensive eval should never run automatically
- Cheap deterministic checks first
- Budgeting an eval suite you can afford to keep
09Reading a Failed Eval Run Honestly3 lessons
- The run that came back at 50.8 percent
- What the number actually meant
- What changed after, and what did not
10Portfolio and Interviews3 lessons
- Putting an eval suite in your portfolio
- Talking about it in an interview
- What the AI QE role looks like right now
Finish and earn a credential you can prove
Complete every module to earn your AI Quality Engineering certificate, with a unique credential ID (AH-AQE-XXXXX) that anyone can confirm at /verify, no login. It lands on your public QA Passport automatically, one link a recruiter can check in seconds. Share it on LinkedIn in one click.
Want more weight? Pass the timed Professional certification exam to earn the Professional credential for this course.
Ready to get started?
Instant access after purchase. Lifetime access and free updates included.