Evaluation of LLM Applications in the real world

Evaluation of LLM Applications in the real world

Task-based evaluation, human review, golden datasets, regression testing, red-teaming, business KPIs

---

Introduction

A large language model can score impressively on public benchmarks and still perform poorly inside a real business application.

The reason is simple: organizations do not deploy benchmarks. They deploy customer-support assistants, search systems, document analysers, copilots, recommendation systems, coding agents, research assistants and increasingly autonomous AI agents that interact with databases, APIs and business software.

Evaluation therefore has to move beyond asking, “How good is this model?” The more useful question is: “How reliably does this AI application perform the task we built it to perform?”

Modern LLM evaluation is consequently becoming an engineering discipline of its own. OpenAI recommends task-specific evaluation based on real-world distributions rather than relying solely on generic benchmarks, while Anthropic similarly emphasizes evaluating agents through realistic tasks and grading their resulting behaviour.

Let’s dive deep into the topic now.

1. Evaluate the task, not merely the model

The first mistake is evaluating an LLM independently of the application in which it operates.

Suppose a company builds an AI customer-service assistant. Knowing that the underlying model performs well on reasoning benchmarks tells us relatively little about whether the application can correctly identify a customer’s problem, retrieve the correct policy, provide an accurate answer and escalate the case when necessary.

The evaluation should therefore mirror the actual job.

A customer-support system might be tested for resolution accuracy, policy adherence, unnecessary escalation and hallucination. A research assistant might be measured for evidence quality, citation correctness and completeness. A coding agent may need tests for task completion, functional correctness and appropriate tool usage.

This task-specific approach is now central to production evaluation guidance from major model providers.

2. Define what a “good answer” means before measuring it

An evaluation is only useful when the success criteria are explicit.

Instead of a vague metric such as “answer quality,” teams should break quality into measurable dimensions appropriate to the application.

For a corporate knowledge assistant, for example, these might include factual correctness, relevance, groundedness in company documents, completeness, compliance with company policy and clarity.

Some criteria can be binary: Did the system return valid JSON? Did it call the correct API? Did it cite the required source?

Others require a rubric, perhaps rating an answer from 1 to 5.

Separating these dimensions is important because an answer can be fluent but factually wrong, technically correct but irrelevant, or useful but non-compliant. Microsoft Foundry, for example, distinguishes quality, RAG, safety and agent-oriented evaluators rather than reducing everything to one generic score. An excellent collection of learning videos awaits you on our Youtube channel.

3. Build a golden dataset from realistic cases

One of the most valuable assets in an LLM project is a high-quality evaluation dataset, often called a golden dataset or reference set.

It contains representative inputs together with expected answers, labels, required facts, acceptable behaviours or scoring criteria.

The dataset should include ordinary cases, difficult cases, edge cases and historically troublesome examples. Production failures are particularly valuable: once an important failure occurs, it should usually become a permanent evaluation case.

The dataset should also reflect actual usage. A beautifully constructed test set that bears little resemblance to production traffic can produce reassuring but misleading scores.

Modern evaluation platforms increasingly support this workflow. OpenAI recommends continuously expanding datasets with new edge cases, while LangSmith and Arize Phoenix support converting production examples into reusable evaluation datasets.

The golden dataset should therefore evolve with the application rather than remain frozen after launch.

4. Keep humans in the evaluation loop

Automated evaluation is essential at scale, but human judgement remains important.

Domain experts can identify errors that automated metrics miss: misleading medical wording, inappropriate financial assumptions, subtle policy violations, poor tone, weak reasoning or answers that are technically correct but practically useless.

Human review is especially valuable during the early stages of an application, when teams are still establishing what good performance actually looks like. It is also needed to calibrate automated evaluators. If an LLM-based judge consistently gives high scores to responses that experts consider unacceptable, the evaluation system itself has failed.

OpenAI explicitly recommends combining automated metrics with human judgement, while LangSmith provides human annotation workflows designed to compare expert labels with automated LLM evaluators.

The goal is not to choose between humans and automation. It is to use human judgement where it adds the most value and automation where scale is required. A constantly updated Whatsapp channel awaits your participation.

5. Use LLM-as-a-judge carefully

One of the biggest developments in LLM evaluation has been the use of another language model as an evaluator.

Instead of checking whether an answer exactly matches a reference sentence, an evaluator model can assess questions such as: Is the answer factually supported? Does it follow the instructions? Which of these two answers is better? Does the response satisfy this five-point rubric?

Google’s Gen AI evaluation tools, for example, support pointwise evaluation of individual outputs and pairwise comparison between alternatives. LangSmith and Arize Phoenix similarly provide LLM-as-a-judge evaluation alongside deterministic evaluators.

But the judge is itself an AI model. Its verdict is not automatically ground truth.

Important evaluators should therefore be tested against human-labelled examples. Pairwise comparisons and clearly defined scoring rubrics are generally more dependable than asking another model whether an answer simply “looks good.”

6. Turn evaluations into regression tests

LLM applications change constantly.

Teams change prompts, models, retrieval algorithms, embeddings, chunk sizes, tool descriptions, system instructions, safety rules and orchestration logic. A modification that improves one category of questions can quietly damage another.

That makes regression testing essential.

Before deploying a significant change, run the new system against the established evaluation suite and compare it with the current production version. A release might require, for example, that factual accuracy does not fall below an agreed threshold, critical safety tests remain at 100%, task completion improves or stays constant, and latency and cost remain within acceptable limits.

OpenAI recommends continuous evaluation as systems change, while LangSmith supports offline experiments followed by production evaluation and the addition of new failures back into evaluation datasets.

In a mature AI engineering environment, an LLM application should not be upgraded simply because the newer model appears smarter. It should have to pass the tests. Excellent individualised mentoring programmes available.

7. Evaluate the entire RAG or agent workflow

Modern AI applications increasingly contain much more than one prompt followed by one answer.

A retrieval-augmented generation system may retrieve documents, rank passages, generate an answer and attach citations. An agent may reason across several steps, call tools, browse databases, execute code, modify files and hand tasks to other agents.

Evaluation therefore needs to examine the workflow as well as the final response.

For RAG systems, teams may measure retrieval relevance, document coverage, groundedness and citation correctness.

For agents, they may inspect whether the correct tool was selected, whether unnecessary tool calls occurred, whether required steps were completed and whether the final task actually succeeded.

OpenAI’s current agent-evaluation guidance explicitly uses traces to analyse model calls, tool calls, guardrails and handoffs, while Microsoft provides evaluators for areas such as task completion, intent resolution and tool-call accuracy.

For complex agents, evaluating only the final paragraph can hide the most important failures.

8. Red-team the application before attackers do

Normal evaluation asks whether the application works under expected conditions. Red-teaming asks how it fails under hostile conditions.

Tests can include prompt injection, jailbreak attempts, malicious retrieved documents, attempts to reveal confidential information, unsafe tool requests, policy circumvention and deliberately misleading inputs.

For agents with access to business systems, red-teaming becomes even more important because the consequences can extend beyond generating inappropriate text.

Anthropic recommends deliberately testing agent workflows with documents, emails and tool outputs containing prompt-injection attempts. Microsoft has incorporated automated adversarial testing through its PyRIT framework and AI Red Teaming Agent, including measurement of attack success rates.

Automated attacks are useful for coverage, but creative human red-teamers remain valuable because real attackers rarely restrict themselves to known test cases. Subscribe to our free AI newsletter now.

9. Continue evaluating after deployment

Passing a pre-launch evaluation does not mean an application will remain reliable.

Real users produce inputs that development teams never anticipated. Data sources change. Business policies change. Models are updated. Retrieval indexes evolve. Attack patterns appear. User behaviour itself changes as people learn how to interact with AI.

Production monitoring should therefore include quality signals, not merely infrastructure statistics such as uptime and CPU usage. Teams can sample live interactions and run automated evaluators for groundedness, safety, instruction adherence or task success. Suspicious or low-scoring interactions can then be sent for human review and potentially added to the golden dataset.

LangSmith, for example, supports online evaluation of production traces and explicitly describes a feedback loop in which production failures become new offline evaluation cases.

The evaluation dataset should increasingly become a compressed history of what the organization has learned about how its AI system can fail.

10. Connect AI evaluation to business KPIs

Ultimately, an LLM application exists to produce an outcome.

A system can achieve excellent technical evaluation scores and still provide little economic value.

Technical metrics therefore need to be connected to business KPIs.

For customer service, useful measures might include first-contact resolution, average handling time, escalation rate and customer satisfaction.

For enterprise search, they could include successful search completion, time saved and reduction in support requests. For an AI sales assistant, relevant measures might include qualified leads, conversion rates or salesperson productivity.

For coding agents, organizations may examine completed development tasks, review corrections, defect rates and engineering time saved. Cost matters as well. An application that raises task success from 92% to 93% while multiplying inference costs fivefold may not represent an improvement.

A particularly useful operational metric is therefore cost per successful task, combining quality and economics instead of optimising either one in isolation.

The final evaluation question is not simply whether the AI produced a good response. It is whether the system produced the required business result accurately, safely, quickly and economically. Upgrade your AI-readiness with our masterclass.

Conclusion

Real-world LLM evaluation is moving away from benchmark obsession toward continuous evaluation of complete AI systems.

A strong evaluation programme starts with realistic tasks and explicit success criteria. It builds a growing golden dataset, incorporates expert human review, uses automated and LLM-based evaluators for scale, and converts important failures into permanent regression tests. RAG pipelines and agents require workflow-level evaluation, while red-teaming exposes behaviours ordinary tests will never reveal.

After deployment, production interactions should continuously feed the evaluation process.

Platforms such as Microsoft Foundry, Google Vertex AI, LangSmith and Arize Phoenix, along with evaluation capabilities offered by model providers such as OpenAI, now support substantial parts of this lifecycle. The particular platform matters less than the discipline behind it.

The most mature organizations will eventually treat LLM evaluation much as software engineering treats testing: not as a final check before launch, but as infrastructure that accompanies every important change.

And one principle should remain above all others:

Do not ask whether your LLM is intelligent. Ask whether your AI system reliably completes the job you are paying it to do.