AI Evals For Engineers & PMs
Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.
ML Engineers with 25+ years combined building & evaluating AI systems
Course Overview
Eliminate the guesswork of building AI applications with data-driven approaches.
⚠ Oct 6 is the last cohort we are teaching this year. Seats are limited.
AI Evals For Engineers & PMs
Ship Confidently with AI Evaluations
A hands-on cohort to build rubrics, golden sets, and online experiments so your AI features deliver consistent, measurable quality.
Instructor
.png&w=1536&q=75)
Enrollment Notes
- All students get LIFETIME ACCESS to all materials and recordings.
- We hold 9+ hours of office hours to maximize the value of live interaction.
- October 6th Cohort Only: 10 months of unlimited access to our new AI Eval Assistant (details near the bottom of this page).
- Students get lifetime access to a Discord community with 1k+ students and instructors.
- This is a flipped classroom. Lectures are professionally recorded; live time focuses on office hours and interaction.
Do you catch yourself asking…
- How do I test applications when outputs are stochastic and require subjective judgements?
- If I change the prompt, how do I know I’m not breaking something else?
- Where should I focus my engineering efforts—do I need to test everything?
- What if I have no data or customers—where do I start?
- What metrics should I track? What tools should I use? Which models are best?
- Can I automate testing and evaluation—and if so, how do I trust it?
If you aren’t sure about the answers, this course is for you.
Hands-on for engineers & technical PMs. Great for coders and “vibe coders.”
What to Expect
Hands-on exercises with code & data. We meet twice a week for four weeks with generous office hours and an active Discord community.
You’ll build skills that set you apart (see testimonials). All sessions are recorded for async viewing.
Course Content
Lesson 1: Fundamentals & Lifecycle LLM Application Evaluation
- Why evaluation matters—business impact & risk mitigation
- Challenges unique to LLM outputs
- Lifecycle from development → production
- Basic instrumentation & observability
- Intro to error analysis & failure categorization
Lesson 2: Systematic Error Analysis
- Synthetic data generation
- Annotation strategies & quant analysis
- Translating errors into improvements
- Avoiding common pitfalls
- Exercise: Build an error tracking system
Lesson 3: Implementing Effective Evaluations
- Metrics: code-based & LLM-judge
- Per-output & system-level evaluation
- Dataset structure for inputs & references
- Exercise: Automated evaluation pipeline
Lesson 4: Collaborative Evaluation Practices
- Team workflows for evaluation
- Inter-annotator agreement
- Consensus on criteria
- Exercise: Alignment in breakout groups
Lesson 5: Architecture-Specific Strategies
- RAG: retrieval relevance & factual accuracy
- Multi-step pipelines & error propagation
- Tool use & multi-turn conversations
- Multi-modal evals (text/image/audio)
- Exercise: Targeted test suites
Lesson 6: Production Monitoring & Continuous Evaluation
- Traces, spans, sessions
- CI/CD quality gates
- Comparable experiments
- Safety & quality guardrails
- Exercise: Monitoring dashboard
Lesson 7: Continuous Human Review Systems
- Strategic sampling
- Reviewer UX for productivity
- Exercise: Continuous feedback collection
Lesson 8: Cost Optimization
- Value vs spend for LLM apps
- Intelligent model routing
- Exercise: Real-world cost optimization
What you'll be able to do
This cohort focuses on shipping. Each module ends with templates and a checklist you can adapt to your stack.
Next cohort details
Curriculum
Download full outlineWeek 1
- Offline, online, human-in-the-loop
- Representative datasets
- Golden sets & rubrics
Week 2
- From UX goals → measurable criteria
- LLM-as-a-judge pitfalls
- Inter-rater agreement & drift
Week 3
- Test runners & CI hooks
- Prompt/Model versioning
- Budgeting & observability
Week 4
- A/B, interleaving & bandits
- Guardrails & real-time feedback
- Rollout playbooks
Capstone
- KPIs & data contracts
- CI quality gates
- Executive-ready readout
Build once, reuse everywhere
Your capstone delivers a working eval suite for a real feature in your product. We provide reference data contracts, CI examples, and an executive-ready readout.
What learners say
“We replaced ad-hoc checks with a lightweight eval pipeline and cut incident rates by 52%. The capstone became our internal standard.”
“The rubric workshop finally aligned UX, product, and research. Our win-rate in experiments jumped immediately.”
“Clear, practical, and opinionated. The templates saved weeks of trial-and-error.”
Tuition & Scholarships
We reserve seats for builders from underrepresented backgrounds and offer team pricing for groups of 3+.
- 10+ hours live instruction
- Capstone project review
- Private community
- Templates & checklists
Get the syllabus
Enter your work email and we’ll send the PDF.
Talk to the team
FAQs
Ready to raise the quality bar?
Join a focused cohort of builders and leave with a reusable evaluation foundation you can ship immediately.
