AI Quality Engineering
How to test LLM and agent systems as a QA engineer. 31 lessons across 10 modules: non-determinism, golden datasets, assertions for fuzzy output, LLM-as-judge and validating the judge, red teaming for prompt injection, CI regression gating, cost control, and an honest walkthrough of a real eval run whose headline metric came back at 50.8 percent. Anchored to a public, production-grade eval suite you can clone, run, and point an interviewer at.
WHAT YOU GET
10 Modules, 31 Lessons
WHO IT IS FOR
QA and SDET engineers whose company has just shipped an LLM feature and asked them to test it
INTERVIEW READY
Every module ends with a quiz that frames the concept the way an interviewer asks it, not the way a textbook states it.
Need more than one? Get all 7 courses for $77 and save 52%.
Preview
What You Get
Who Is This For?
Course curriculum
10 modules. Every lesson is a guided, narrated video.
01What Breaks in AI Systems3 lessons
- Welcome, and who this is for
- The failure modes that have no analogue in normal software
- Why this job is going to QA people
02Non-Determinism3 lessons
- The assumption your entire test strategy is built on
- Where the variance actually comes from
- Testing when you cannot compare to a fixed string
03Building a Golden Dataset3 lessons
- What a golden dataset is, and what it is not
- Building your first one from real traffic
- Curating it so it keeps finding things
04Assertions for Fuzzy Outputs3 lessons
- Must-mention and must-not-mention
- Score ranges, thresholds, and why a single number lies
- Writing assertions that survive a model upgrade
05LLM as Judge3 lessons
- When a judge is the right tool, and when it is a cop-out
- Writing a judge prompt that is not just vibes
- Testing the judge itself
06Red Teaming4 lessons
- Prompt injection, direct and indirect
- System prompt exfiltration
- Output format hijacks and why they are a security bug
- Running a red team suite you can rerun
07Drift Detection and Regression Gating3 lessons
- What drift actually looks like in production
- Regression gating in CI
- Choosing what blocks and what only warns
08Cost Control3 lessons
- Why the expensive eval should never run automatically
- Cheap deterministic checks first
- Budgeting an eval suite you can afford to keep
09Reading a Failed Eval Run Honestly3 lessons
- The run that came back at 50.8 percent
- What the number actually meant
- What changed after, and what did not
10Portfolio and Interviews3 lessons
- Putting an eval suite in your portfolio
- Talking about it in an interview
- What the AI QE role looks like right now
Finish and earn a credential you can prove
Complete every module to earn your AI Quality Engineering certificate, with a unique credential ID (AH-AQE-XXXXX) that anyone can confirm at /verify, no login. It lands on your public QA Passport automatically, one link a recruiter can check in seconds. Share it on LinkedIn in one click.
Want more weight? Pass the timed Professional certification exam to earn the Professional credential for this course.
Common questions
Do I need to know machine learning to take this course?
No. This is written for QA and SDET engineers, not ML engineers. You are testing the behaviour of a system you did not train, which is the job you already do, applied to outputs that change between runs. There is no model training, no maths, and no research-paper reading anywhere in the course.
How do you test something that gives a different answer every time?
That is the central question of the course and it takes four modules to answer properly. You build a golden dataset of cases with known-good characteristics, write assertions on properties rather than exact strings (must-mention checks, score ranges, structural rules), use a second model as a judge where a property is genuinely fuzzy, and then test the judge itself so you are not trusting an unvalidated grader.
What is the eval suite, and is it really public?
Yes. It is at github.com/aston-cook/asserthired-evals and you can clone and run it. It is a reference implementation over synthetic cases and a sanitized prompt snapshot, not a suite running against live production traffic. Module 9 walks through one of its runs, which scored 50.8 percent against a CI floor of 80 percent, and what was changed to bring it to 98.5 percent. The write-up is in FINDINGS.md in that repository, so every number in the module can be checked against its source.
What programming language does it use?
The eval suite and the code examples are JavaScript and TypeScript, run with Node. The concepts (golden datasets, judge validation, red teaming, CI gating) are language-neutral, so the patterns transfer if your stack is Python or Java.
Will this help me get an AI quality role?
The final module is about exactly that: what to put in a portfolio, how to talk about eval work in an interview, and what the role actually looks like today. The strongest thing you leave with is not the certificate, it is a working eval suite you can demonstrate and a failed run you can explain honestly, which is a much rarer interview answer than a course completion.
How long does it take to finish?
About four hours of video across 31 lessons, plus the time you spend running the suite yourself. There is no deadline and no expiry: you get lifetime access and every future update at no extra cost.
Related Resources
Ready to get started?
Instant access after purchase. Lifetime access and free updates included.