How long is Evaluating AI Systems?

7 hours of recorded sessions across 5 modules, taken at your own pace. 5 of the modules end in work Nova grades.

Is Evaluating AI Systems live or self-paced?

Self-paced. The sessions are recorded by an instructor who does this work, and Nova, Square 1's AI tutor, grades every checkpoint and project against a rubric you can read.

How much does Evaluating AI Systems cost?

A$45 for the founding intake, paid once; the standard price is A$90. It is the same price in every country. Founding students keep their founding price on any second programme.

What will I build in Evaluating AI Systems?

3 deployed projects: A golden set and rubric, A calibrated judge and A CI eval pipeline. Each is graded against a published rubric and kept on a record an employer can open at /verify.

What skills does Evaluating AI Systems teach?

evals, LLM judges, CI and labelling. It prepares you for roles such as AI engineer and QA engineer (AI).

Does Evaluating AI Systems guarantee a job?

No. No placement rate is published because there are no graduates to count yet. What you leave with is graded, deployed work on one record an employer can open and run, and a hiring plan you keep with the career agent.

On demandRecorded by an instructor, graded by Nova

Evaluating AI Systems

Name: Evaluating AI Systems
Price: 45.00 AUD
Availability: PreOrder

LLM judges, rubrics and regression suites you can trust.

Evaluating AI Systems is a self-paced online course: 7 hours of recorded sessions across 5 modules, with 5 graded by Nova, Square 1's AI tutor. The founding price is A$45, down from A$90. Start any time and keep your own pace.

Taught with

PythonClaudeChatGPTGitHub

Not sure it is for you?Take the free Generative AI skill check

Module by module

Module 1Defining quality60 min · graded
Module 2Golden sets90 min · graded
Module 3LLM judges, calibrated90 min · graded
Module 4Regression suites in CI90 min · graded
Module 5Project: evals for a real feature90 min · graded

recorded hours: 7
modules, in order: 5
deployed projects: 3
graded checkpoints: 5

Who it is for, and what you receive.

Developers in Python who have shipped or tested an LLM feature and need to say how good it is, with evidence.

Before you start

Python; you have shipped or tested an LLM feature.

Skills

evalsLLM judgesCIlabelling

1
The recorded sessions
Taught by an instructor, taken in order at your own pace, yours for twelve months.
2
Graded work after every module
Exercises and projects marked by Nova against a rubric you can read, with what you did well and what to fix.
3
The projects
Deployed and graded, each one on your record with its repository.
4
Your hiring plan
The same six-part plan the bootcamps use, run with the career agent.
5
The record and the certificate
Every grade at square1ai.com/u/{handle}; a credential ID that resolves at square1ai.com/verify.
6
Nova as your tutor
Help that is about your actual work, because it has read all of it.

The content plan, module by module.

7 recorded hours from an instructor, taken in order at your own pace. Nova grades the work at the end of each module.

Rubrics
Agreement
What to measure

The projects.

Each is deployed and graded by Nova against a rubric you can read before you start.

Project 1
A golden set and rubric
Pick an LLM feature you have shipped or tested, such as a support reply drafter for a fictional software company. Write a rubric, sample and label 200 real cases, and have a second person label a subset to measure agreement.
You hand in
- Written rubric
- Golden set of 200 cases
- Agreement report
The rubric requires
Agreement between labellers is above the stated threshold.
Project 2
A calibrated judge
Write an LLM judge prompt that scores outputs against your rubric, then calibrate it against your human labels until they agree. Report where the judge and humans disagree and how you will watch for drift.
You hand in
- Judge prompt
- Calibration report
- Drift check plan
The rubric requires
Agreement between the judge and human labels is measured on the golden set.
Project 3
A CI eval pipeline
Wire the golden set and judge into continuous integration with thresholds, so a change that makes the feature worse cannot merge. Prove it by seeding a regression, such as a weakened prompt, and watching the build fail.
You hand in
- CI pipeline config
- Thresholds with rationale
- Failed build from seeded regression
The rubric requires
A seeded regression is caught and blocks the merge.

What you can do at the end.

You can say how good an AI feature is, with evidence, and three graded projects on your record.

1
Define quality as a rubric two people agree on.
2
Build a labelled golden set.
3
Calibrate an LLM judge against human labels.
4
Run regression suites in CI that block a bad change.
5
Build evals for a real feature.

Roles this prepares you for

AI engineerQA engineer (AI)

No placement rate is shown, because there are no graduates to count yet. The roles above are what the projects are built for.

Your record at the end.

Every exercise and project is graded by Nova against a rubric you can read, and every grade is kept on one page an employer can open and run. This is what the programme writes to it.

Graded, line by line: Nova reads every submission against the brief and the rubric and returns a score, what you did well and what to fix.
Module by module: Each module ends in graded work; the next opens when you are ready, on your own schedule.
Nova remembers: Help is about your actual work, because the tutor has every submission and every failed exercise of yours.
One page an employer can run: Every grade and project at /verify. An employer opens it and runs the code.

Record, Evaluating AI SystemsExample

Project
A golden set and rubric
Agreement between labellers is above the stated threshold.
Graded
Project
A calibrated judge
Agreement between the judge and human labels is measured on the golden set.
Graded
Project
A CI eval pipeline
A seeded regression is caught and blocks the merge.
Graded
Every module
5 graded checkpoints
Each module ends in work Nova grades line by line against a rubric you can read.
Graded

The entries, not the grades: those are yours to earn. The page lives at /verify and an employer needs no account to open it.

5 entries, at your own pace. Reserving a spot holds the founding price.

Pre-order

How we help you find a job.

Proof, not a certificate: the projects you deployed are the thing you show, and the tools below are yours to use.

The course booklet

The whole plan on a few pages: the parts, the projects, the gates and the price. We email it to you.

Get the booklet

1
Your hiring plan
The same six-part plan the bootcamps use: target roles, the gap map, the proof to send, weekly actions, an interview log, the outcome. You run it with the career agent.
2
A record an employer can run
Your graded projects on /verify. An employer opens it and runs the code.
3
The career agent
Paste a real job posting at /career and it maps the role to your graded work and what to do next.
4
The roles directory
Every role we prepare people for, what it pays and what it asks, at /roles.
5
A path to the live cohort
If you want the instructor, the gates and the hiring sprint, the bootcamp on the same subject is one waitlist away.

Questions people ask.

About Evaluating AI Systems, answered from the plan on this page.

Founding spots are open. Hold yours.

Recorded by an instructor who does this work, graded by Nova. Spot holders hear the opening date first, and nothing is charged before you confirm.

You can say how good an AI feature is, with evidence
Three graded projects on your record

A$45~~A$90~~