Ravi Chandrasekhar
Refreshingly unhyped. Jonah spends the first three hours on evaluation before touching a prompt, which is exactly the right order and exactly what nobody else teaches.
AIAdvanced
LLM Apps in Production
Evaluation, retrieval and cost control — the unglamorous work that makes a model feature reliable.
Free preview · 24 min
$129$189
32% off this month30-day money-back guarantee
Includes
- 13 hours of video
- 8 hands-on projects
- An evaluation harness
- Lifetime access
- Certificate of completion
50% of the running time is hands-on — projects and quizzes rather than video.
What you'll learn
Evaluate before you build
A golden set drawn from real traffic, and prompt regression tests that run in CI like any other test.
Make retrieval actually help
Chunking, hybrid search and reranking that improve the answer rather than padding the context window.
Handle mid-stream failure
Much of what goes wrong in production goes wrong after the first token. Stream a response that survives it.
Control cost and latency
Caching, model routing, and the underrated skill of knowing when not to call the model at all.
Ship guardrails that fail closed
Safety checks that default to refusing, and logging detailed enough to debug the refusal afterwards.
What’s inside
Every figure below is summed from the lesson list, not estimated — so the chart and the curriculum can never disagree.
| Content type | Minutes | Share |
|---|---|---|
| Video lessons | 410 | 50.5% |
| Hands-on projects | 388 | 47.8% |
| Quizzes | 14 | 1.7% |
50% of this course is hands-on.
Curriculum
4 sections · 19 lessons · 14h total
You cannot improve what you have not defined.
- VideoWhy prompt tinkering plateaus
- VideoBuilding a golden set from real traffic
- VideoModel-graded evaluation, and its limits
- VideoRegression tests for prompts in CI
- ProjectProject: an eval harness for your own feature
Most RAG problems are retrieval problems, not model problems.
- VideoChunking decisions that change the answer
- VideoHybrid search: vectors plus keywords
- VideoReranking, and measuring whether it helped
- VideoCitations users can actually verify
- ProjectProject: lift answer accuracy 18 points
- QuizRetrieval quiz
Streaming, tools and the failures that happen halfway through.
- VideoStreaming a response into a real interface
- VideoTool calls, retries and partial failure
- VideoStructured output you can parse with confidence
- ProjectProject: a tool-using agent with a budget
The three constraints that decide whether it ships.
- VideoCaching: exact, semantic and prefix
- VideoRouting between a small model and a large one
- VideoGuardrails that fail closed
- ProjectCapstone: halve the cost, hold the quality
Your instructor
Jonah Price
ML engineer · applied language models
Jonah builds language-model features that survive contact with real users — which mostly means evaluation, cost control and knowing when not to call the model at all. He is refreshingly unromantic about the technology and it makes the course better.
Research engineering at Vellum Labs
CompletedRetrieval systems for legal search.
Head of applied AI, Corvid
CompletedShipped four LLM features to 2M users.
Open source
In progressAuthor of evalkit, a harness for prompt regression tests.
Reviews
What learners said
1,639 ratings from people who bought this course.
4.6
1,639 ratings
- 5 stars
- 4 stars
- 3 stars
- 2 stars
- 1 star
Johanna Meyer
The routing lesson cut our inference bill by 61% in a fortnight. I did the maths twice because I did not believe it. Advanced, and it does expect you to know your way around a backend.
Kwame Asante
Very strong on the engineering. Moves fast through the model fundamentals, so brush up first if that is new to you. The eval harness is now part of our repo.
Learners also took
Same category first, then whatever pairs well with it.
Full-stackShip It: Next.js 16 from Zero to Production
Build and deploy a real application on the App Router — async request APIs, streaming, caching and the deploy that follows.
15h25 lessonsIntermediate
FrontendTypeScript for Teams
Types that catch real bugs and stay out of the way — for codebases with more than one author.
10h18 lessonsIntermediate
FrontendReact Server Components, Properly
The boundary, the payload and the render — what actually happens between the server and the browser.
10h16 lessonsAdvanced
Browse the full course catalogue, or see how this page is put together on Built with VivekUI.