How to Evaluate an LLM Application

An evaluation, or eval, is a repeatable test that tells you whether your language-model application does the job. It is a set of inputs, a definition of what a good result is, and a way to score outputs against it, run every time something changes.

Teams that have this ship changes with confidence. Teams that do not ship on impressions: someone tries five prompts, they look fine, and a regression reaches users a week later. The difference is rarely the model. It is the test set.

Why public benchmarks are not enough

A benchmark measures a model on a task someone else chose. That is useful for shortlisting, but it does not tell you whether the model works on your inputs, with your prompts, your retrieval and your users.

The Holistic Evaluation of Language Models project makes the broader point well. It evaluated 30 prominent models across 42 scenarios on seven kinds of metric: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. The value of that work is the principle that a single accuracy number is not a description of a system, and that the question of what to measure is a design decision.

For an application the same applies, with sharper edges. A leaderboard rank does not say how the model handles your customer's malformed input, your retrieval's wrong passage, or your tool's timeout. We have written about why a leaderboard does not predict which model wins in production. The eval is how you find out.

Start with failures, not metrics

The most useful eval sets are built from things that went wrong. Before choosing a metric, collect examples:

• real requests from your logs, especially ones that produced complaints, retries or manual fixes
• the edge cases your team already worries about
• inputs an adversary would try, including attempts at prompt injection
• examples of the same request phrased several ways
• requests the system should refuse or hand to a person

Label each with the outcome you want. For some cases that is an exact answer. For many it is a property: cites a source, does not reveal the system prompt, asks a clarifying question, calls the right tool with the right arguments.

A set of a few dozen carefully chosen cases catches gross failures immediately. A few hundred covers the main paths of most applications. Variety matters more than volume: fifty near-identical easy cases tell you little that five would not.

Choose a scoring method that fits the property

Kind of check	Good for	Limits
Exact or rule-based	Structured output, tool calls, formats, required phrases	Cannot judge open-ended quality
Programmatic	Does the code run, do the numbers match, is the JSON valid	Only works where correctness is computable
Reference comparison	Answers with a known correct form	Penalises valid alternatives
LLM judge	Open-ended quality, tone, helpfulness	Has its own biases and needs calibration
Human review	Final authority on hard cases	Slow and expensive, so use it on a sample

Use the cheapest check that actually measures the property. A great deal can be tested with rules and code, and those checks are free, fast and deterministic. Save judges and people for what only they can assess.

Using an LLM as a judge

A model grading another model's output is practical and widely used. It scales where human review does not. It also has known weaknesses.

The MT-Bench paper is the standard reference. It reports that strong judges such as GPT-4 reach over 80 percent agreement with human preferences, the same level at which humans agree with each other. It also documents biases: position bias (preferring an answer because of where it appears), verbosity bias (preferring longer answers) and self-enhancement bias (favouring outputs from the same model family), along with limited ability to judge reasoning that it cannot itself do.

Those findings translate into rules:

1. Randomise or swap the order when comparing two answers, and count a result only if it holds both ways.
2. Give the judge a rubric, with specific criteria and examples, instead of asking whether an answer is good.
3. Control for length. A longer answer is not a better one.
4. Avoid grading a model with itself where you can, or at least check for favouritism.
5. Calibrate against humans. Label a sample by hand, measure how often the judge agrees, and revisit if agreement is low.
6. Do not use a judge for facts it cannot verify. Check claims against a source, not against the judge's opinion.

Evaluating retrieval-augmented systems

A retrieval system can fail in two places, and a single score hides which. The Ragas paper proposes reference-free evaluation of retrieval-augmented generation along three kinds of dimension: whether the retrieval system finds relevant and focused passages, whether the model uses those passages faithfully, and the quality of the generation itself.

Test them separately:

Layer	Question	Example check
Retrieval	Did the right passage come back?	Is the known-relevant document in the top results?
Faithfulness	Does the answer follow from the passages?	Does each claim appear in the retrieved text?
Answer quality	Is the answer useful and complete?	Judge or human review against a rubric

When quality is poor, this tells you whether to fix the retriever, the chunking or the prompt. We have written about how a high similarity score does not mean a relevant passage, and why a vector index's recall should be measured, not assumed. Automated metrics like those in Ragas are a starting point, and they still need checking against human judgement on your own data.

Evaluating agents

An agent has more ways to fail than a single call, because errors compound across steps. Evaluate three things separately:

• The outcome. Did the task end in the right state? This is the check that matters most and the easiest to automate when the result can be verified.
• The trajectory. Did it take sensible steps, use the right tools, and avoid unnecessary or dangerous ones? Two runs can reach the same answer by very different paths.
• The cost. How many calls, tokens and seconds did it take, and what does the distribution look like? Averages hide the long tail.

Many agent failures produce a confident, well-formed wrong answer and no error at all, which is why most agent failures never throw an error. Outcome checks are the defence, and they only work if you wrote down what a correct outcome looks like in advance. A definition of the term itself helps, so see what agentic AI actually means.

Make it part of the workflow

An eval that nobody runs is a document. Three habits turn it into protection:

Run it on every change. Prompt edits, model upgrades, retrieval changes, tool changes and even small configuration tweaks. The point is to learn about a regression before users do.

Turn every production failure into a test case. When something goes wrong in the real world, add it to the set. The set becomes a record of everything that has hurt you, and the same mistake cannot return unnoticed.

Track results over time. A table of scores per version shows trends that a single run hides, such as a slow decline after several small changes that each looked harmless.

Keep a held-out portion of cases that you never tune against, so that improvement on the set reflects a genuine gain and not memorisation of the test.

Report results honestly

When you share an eval result, say what was measured, on how many cases, with what scoring method, and what the uncertainty is. A three-point gain on thirty cases is within noise. Break results down by category instead of reporting one average, because a system can improve overall while getting worse on the cases that matter most. And record the model version, prompt version and date, since an eval result without them cannot be reproduced.

A starting checklist

1. Collect 50 to 100 real or realistic inputs, weighted towards the hard and the rare.
2. Write down, for each, what a good outcome looks like.
3. Score with rules and code wherever possible.
4. Add an LLM judge with a rubric for what rules cannot capture, and calibrate it against a human-labelled sample.
5. Separate retrieval, generation and, for agents, outcome and trajectory.
6. Run the set on every change, and track scores by version.
7. Add every production failure to the set.
8. Hold out a portion you never tune against.

What we do not know

We do not know of an evaluation method that is both cheap and fully trustworthy for open-ended output. Automated judges scale but carry biases, and human review is reliable but slow, so every setup is a compromise between them. We also do not know how well scores on a test set predict satisfaction in production over time, as user behaviour shifts. The practical answer is to keep the set growing from real failures, keep a human in the loop on a sample, and treat the scores as evidence, not a guarantee.