The problem
The team owns a customer-facing AI support chatbot — the first line of support, answering product and app questions in natural language from help-center articles, resolving issues before they ever reach a human agent. But nobody was monitoring the quality of its retrieval or generation, and there wasn't capacity to.
Checking by hand doesn't really work: because the output is non-deterministic, catching failures takes repeated runs across many test cases to see how and where it breaks — far too tedious to do manually for every new or updated article. We own the app and the help-center articles the bot answers from, and we get the third-level requests, so its quality is our problem: wrong or low-quality answers generate support requests that bind our capacity.
What I built
An end-to-end evaluation platform for the support chatbot. It ingests help-center articles, uses LLMs to derive the correct answer and generate realistic customer questions, drives the live chatbot in a real browser, and scores its answers with an LLM-as-judge — turning “does our chatbot actually work?” into a measurable, repeatable pipeline.
- Ingest: support articles added manually, by CSV bulk import, or via an n8n webhook from the CMS.
- Ground truth & questions: an LLM extracts the key answer from each article (human-editable), then generates realistic customer questions in varied phrasing and personas.
- Run: a Playwright bot drives the real chat UI in a browser, asks each question (handling follow-ups), and captures answers, cited links, and screenshots — repeating each k times to catch a non-deterministic bot's flaky answers.
- Score: an LLM-as-judge rates correctness, faithfulness, and relevance with a written rationale; cheap code-checks short-circuit obvious cases; humans can override any verdict.
- Results: pass rates by category, run-to-run comparison, batch re-evaluation, and shareable reports.
- Tunable: extraction, question-generation, and judge prompts are all versioned and editable in-app (old scores keep their version).
- Scope: stays out of retrieval entirely — I don't touch the bot's retrieval logic (we don't own that layer), keeping it focused on the output quality we can actually act on.
- Stack: TypeScript monorepo — Next.js (web) + Node/Playwright (runner) + Supabase/Postgres + Gemini, with a decoupled runner that claims jobs atomically and recovers from crashes.
- Built: solo over an extended weekend — I laid out the full plan, then let Claude Code do the heavy lifting.
What it confirmed
A live production system that nobody was checking now has a quality signal.
The lesson
Build the right small thing inside the constraints you actually have — an eval you own beats an eval you don't.