Skip to main content

LLM Evals in Practice

An eval is a repeatable test for an AI feature: a fixed set of inputs, a way to score the outputs, and a number you can track every time you change a prompt, a model or a retrieval setting.

Intermediate9 min readv1.0Updated Oct 28, 2026
AI-assisted content — reviewed by the author, but verify important details independently

Visual SummaryClick to explore

Learning Objectives

  • Explain why evals are the main quality control tool for LLM products.
  • Build a small golden dataset from real user cases.
  • Choose between code-based checks, LLM-as-judge and human review.
  • Run evals in CI so a prompt or model change cannot silently break quality.
  • Avoid the common traps that make eval scores misleading.

Why This Matters

Traditional software has unit tests. LLM features need something similar, but the output is free text and changes from run to run, so a simple "expected equals actual" check rarely works. Teams without evals ship a prompt tweak that fixes one complaint and quietly breaks ten other cases. Teams with evals see the drop before users do. AI-022 and AI-025 covered the theory. This lesson is the hands-on workflow.

Everyday Analogy

Think of a restaurant that changes its recipe. A good chef does not just taste the one dish a customer complained about. They cook the ten most popular dishes again and check each one still tastes right. Your golden dataset is that list of ten dishes. The eval is the tasting.

Step 1: Build a Golden Dataset

Start small. Twenty to fifty real examples beat a thousand synthetic ones.

  • Pull real questions from logs, support tickets or user interviews.
  • Include the easy cases, the hard cases and the ones that failed before.
  • For each input, write what a good answer must contain (not the exact wording).
Input Must include Must not include
"How do I cancel my plan?" Link to billing page, mention of refund window Any made-up phone number
"Is my data used for training?" Clear yes/no matching the policy Vague hedging

Store this as a file in your repo, next to the prompt it tests.

Step 2: Pick a Scoring Method

Use the cheapest method that works for each check.

Method Good for Cost
Code checks Valid JSON, required fields, length limits, banned words, a link is present Almost free
LLM-as-judge Tone, helpfulness, "does it answer the question", faithfulness to sources Cents per run
Human review Final sign-off, new feature types, calibrating the judge Expensive, slow

A good setup layers them: code checks catch format breaks, an LLM judge scores quality, and a person spot-checks a sample each week.

Writing a Good Judge Prompt

An LLM judge is only as good as its rubric. Give it a narrow question and a fixed scale:

You are grading a support answer.
Question: {input}
Answer: {output}
Source policy: {context}

Score 1 if the answer states anything not supported by the source policy.
Score 3 if it is correct but misses a required point.
Score 5 if it is correct and covers every required point.
Reply with only the number.

Check the judge against human scores on 20 examples before you trust it.

Step 3: Run Evals Automatically

Evals only help if they run every time something changes.

  • Run the eval suite in CI on every pull request that touches a prompt, model setting or retrieval code.
  • Fail the build if the score drops below a set threshold.
  • Save results over time so you can see trends, not just a single run.

Tools that help: Promptfoo, OpenAI Evals, Braintrust, LangSmith, or a 50-line script of your own. The tool matters less than the habit.

Step 4: Feed Production Back In

Every bad answer a user reports is a free test case. Add it to the golden dataset. Over months the dataset becomes a record of every mistake your product has ever made, and a guard against making it twice.

Common Traps

  • Testing only happy paths. If every case is easy, the score stays high and tells you nothing.
  • Trusting the judge blindly. LLM judges favour longer answers and answers that sound confident. Calibrate them.
  • Overfitting to the eval. If you tune the prompt until one dataset scores 100%, it may get worse on real traffic. Keep a held-out set.
  • One number for everything. Track separate scores for accuracy, format, safety and tone so you know what broke.
  • Ignoring randomness. Run each case more than once if temperature is above zero, and compare averages.

Worked Example

A team changes their RAG chatbot (AI-016) from one model to a cheaper one. Before evals, they would have shipped and waited for complaints. With evals:

  1. The golden set of 40 questions runs against both models.
  2. Code checks: both pass format.
  3. Judge score for faithfulness: old model 4.6, new model 4.1.
  4. The drop is concentrated in questions about refunds.
  5. They add two refund examples to the prompt, rerun: 4.5. Ship.

Total time: one afternoon. Without evals this would have been weeks of user complaints.

Try It Yourself

  1. Pick one AI feature you use or build. Write down 10 real inputs and what a good answer must contain.
  2. Write one code check (for example, "answer is under 150 words") and one judge prompt.
  3. Change the prompt slightly and rerun. See if your eval catches the difference.

Key Takeaways

  • Evals are tests for AI features: fixed inputs, a scoring method, a tracked number.
  • Start with 20 to 50 real examples, including past failures.
  • Layer code checks, LLM-as-judge and human review by cost and need.
  • Run evals in CI on every prompt or model change.
  • Every production failure becomes a new test case.

Glossary

Eval
A repeatable test for an AI feature made of a fixed set of inputs, a scoring method and a tracked score. The LLM equivalent of a unit test suite.
Golden Dataset
A curated list of real inputs with notes on what a good answer must and must not contain. It grows over time as production failures are added.
LLM-as-Judge
Using a language model with a narrow rubric to score another model's output on qualities that code cannot check, such as tone, helpfulness or faithfulness.
Calibration
Comparing an LLM judge's scores with human scores on a sample to confirm the judge agrees with people before relying on it.
Regression
A drop in quality caused by a change, such as a new prompt or a cheaper model. Evals in CI exist to catch regressions before users do.
Held-out Set
A part of the dataset you never tune against, used to check that improvements are real and not overfitting.

References

Knowledge Check

8 questions