Introduction: Evaluation as the Real Foundation of AI Product Development
AI features are often introduced with the language of possibility, speed, intelligence, automation, and delight, but the practical question that determines whether they deserve to exist in a product is much less glamorous: can the team reliably tell whether the feature is working, whether it is failing, and whether the latest change made it better or worse. Eugene Yan’s writing on AI systems returns repeatedly to this central idea, arguing that evaluation is not a decorative layer added after an AI feature is built, but the operating discipline that should shape the feature from the beginning, guide every iteration, and protect users from silent regressions.
The core lesson in Yan’s approach is that AI evaluation must become a product and engineering practice rather than a one-time measurement exercise. In his view, teams cannot simply buy an evaluation tool, attach an LLM-as-judge, run a few benchmark scores, and assume the product is safe, useful, or trustworthy. They need a repeatable process that begins with looking at real data, identifying failures, labeling examples, defining criteria, running experiments, measuring outcomes, and monitoring behavior after launch. Yan explicitly frames product evaluation as a form of scientific method, where observation, hypothesis, experimentation, measurement, and iteration become the rhythm of AI product development. (eugeneyan.com)
This makes his perspective especially valuable for AI features because features are not abstract models. They live inside products, affect users, operate under messy real-world constraints, and fail in ways that generic benchmarks rarely capture. A model may perform well in a leaderboard setting and still produce irrelevant summaries, hallucinated explanations, unsafe recommendations, brittle SQL, misleading citations, or charming but useless chat responses once embedded in a real workflow. Yan’s evaluation philosophy is therefore product-centered, failure-oriented, and operationally pragmatic.
1. Evaluation Should Begin Before the Feature Is Built
1.1 Eval-Driven Development as Product Discipline
One of Yan’s strongest ideas is eval-driven development, which adapts the spirit of test-driven development to AI products. Instead of building an AI feature first and then vaguely checking whether it feels good, teams should define success criteria before development begins, establish representative examples, and create an evaluation harness that can be run whenever prompts, retrieval settings, models, guardrails, or product logic change. In this architecture, evaluation becomes the specification of the feature, because it defines what the feature must do, what it must avoid, and what level of defect rate is acceptable before release.
This matters because AI features are unusually vulnerable to ambiguity. A conventional feature often has a relatively clear expected behavior: clicking a button opens a modal, submitting a form saves a record, changing a setting modifies a preference. An AI feature, by contrast, may summarize, recommend, classify, explain, generate, retrieve, or converse, and each of those actions can be partly correct, superficially plausible, stylistically impressive, and still productively wrong. Yan’s approach forces teams to turn vague expectations into explicit evaluation dimensions, so that the feature is not judged by charisma, demo quality, or stakeholder excitement alone.
1.2 The Baseline Comes First
Yan’s evaluation logic also implies that every AI feature needs a baseline. A baseline may be a simple prompt, a non-AI workflow, a previous model, a rules-based system, or the current production configuration, but without it the team cannot know whether a new version is meaningfully better. Evaluation without a baseline becomes a mood exercise, because the team is only asking whether the new output seems acceptable rather than whether it improves the user experience, reduces defects, lowers cost, increases trust, or handles edge cases more reliably.
This baseline-first mindset protects teams from confusing novelty with progress. AI features often appear impressive because they produce fluent outputs, but fluency is not the same as correctness, usefulness, safety, or product fit. A baseline gives the team something concrete to beat, and it creates the possibility of disciplined comparison instead of theatrical demonstration.
2. The First Step Is Looking at the Data
2.1 Real Outputs Reveal Real Failure Modes
Yan repeatedly emphasizes the importance of looking directly at inputs, outputs, and user interactions, because the failure modes of AI products are often only visible in concrete examples. A team may assume that its summarizer fails mostly because of prompt wording, but real outputs may reveal that retrieval is missing key documents, that the model over-compresses critical details, that the source material contains contradictions, or that users are asking questions outside the expected domain. A team may assume that a chatbot needs a stronger model, while inspection may show that the interface is creating unrealistic expectations or that the system lacks sufficient context.
This focus on real examples is important because AI product failures are rarely evenly distributed. A feature may perform well on common cases and fail catastrophically on rare but important cases. It may satisfy average users while disappointing expert users. It may produce acceptable results in English and weaker results in other languages. It may work on short inputs and degrade on long-context tasks. By looking closely at the data, teams discover the actual shape of the problem rather than the version imagined in planning documents.
2.2 Failure Cases Are More Valuable Than Comforting Successes
In Yan’s product-eval method, teams should deliberately collect and label failures, not just random examples that make the system look competent. For pass/fail evaluation, he recommends building a useful dataset with enough failure cases to align and test evaluators, noting that the failures are often the defects that break user trust. He suggests that a dataset with many labels but very few failures is not especially useful, because it gives the team little signal about whether evaluators can detect the problems that matter. (eugeneyan.com)
This is a crucial insight for AI features because the product risk usually lives in the failures, not in the average happy path. A recommendation feature that works for popular items but fails on niche queries may alienate power users. A customer-support assistant that answers easy questions but invents policies on hard questions may create business risk. A code-generation assistant that handles simple boilerplate but fails silently on security-sensitive logic may be dangerous precisely because it looks competent most of the time.
3. Evaluation Criteria Must Be Specific, Separable, and Product-Relevant
3.1 One Evaluator per Dimension
Yan warns against building a single “God Evaluator” that attempts to judge many dimensions at once, such as faithfulness, relevance, concision, tone, safety, helpfulness, and formatting in a single prompt. His preferred approach is to create individual evaluators for individual dimensions, then combine their results with simple rules or heuristics. This gives teams granular visibility into what is improving or regressing, and it allows them to distinguish between ship-blocking guardrail metrics and aspirational quality metrics. (eugeneyan.com)
For AI features, this separation is essential because a single overall score can hide the reason a feature is failing. A generated answer may be highly relevant but unfaithful to the source. A summary may be faithful but too verbose. A recommendation explanation may sound persuasive but lack evidence. A chatbot response may be polite but fail to answer the question. If these dimensions are collapsed into one score, the team cannot tell what to fix, and each iteration becomes a guessing game.
3.2 Product Criteria Beat Generic Benchmarks
Yan’s perspective also pushes teams away from over-reliance on generic benchmarks. Benchmarks can help compare broad model capabilities, but they are not substitutes for product-specific evaluation. He notes that even widely used benchmarks can be inconsistent depending on implementation, prompts, scoring methods, and evaluation setup, which makes it dangerous to treat benchmark performance as a direct proxy for product quality. (eugeneyan.com)
A product team evaluating an AI feature must therefore ask questions that benchmarks cannot answer. Does the feature solve the user’s actual problem in the product context? Does it preserve trust? Does it fail gracefully? Does it cite sources when claims need support? Does it avoid inventing unavailable information? Does it reduce user effort rather than increasing it? Does it work under the latency and cost constraints of production? These are not abstract model questions. They are product questions, and they require product evals.
4. Human Labels Are the Foundation, Even When Automated Judges Are Used
4.1 LLM-as-Judge Is a Scaling Mechanism, Not a Replacement for Judgment
Yan’s writing on LLM evaluators is balanced: he sees LLM-as-judge as useful, especially for open-ended tasks where exact-match metrics are inadequate, but he does not treat it as a magic solution. He describes LLM evaluators as models that assess another model’s response, and explains that their adoption has grown because traditional evaluation methods struggle with tasks like long-form summarization, translation, and dialogue, while human evaluation is expensive and difficult to scale. (eugeneyan.com)
The important nuance is that an LLM-as-judge must be aligned to human judgment. It needs labeled examples, clear criteria, development and test splits, and measurement against human annotations. Without that discipline, automated evaluation can create false confidence, because the team may simply be measuring the preferences, blind spots, or biases of another model. Yan’s point is not that LLM evaluators are bad, but that they are only as useful as the process that calibrates, tests, and monitors them.
4.2 Binary Labels Often Work Better Than Overly Granular Scores
Yan recommends starting with binary pass/fail labels for objective criteria and win/lose/tie comparisons for more subjective comparisons, because numeric scales such as one-to-five ratings can be difficult for both humans and LLM evaluators to calibrate consistently. His argument is practical: the difference between a “3” and a “4” is often subtle, annotators may disagree despite rubrics, and stakeholders often end up asking for a pass/fail threshold anyway. (eugeneyan.com)
For AI features, this advice cuts through unnecessary evaluation complexity. Teams often over-design rubrics because they want evaluation to look sophisticated, but sophistication that produces noisy labels is counterproductive. If the product question is whether a summary is faithful, whether an answer cites sufficient evidence, whether a refusal is appropriate, or whether a recommendation explanation is acceptable, a clear pass/fail decision may produce more reliable signal than a pseudo-precise score.
5. Evaluation Must Be Built Into the Experiment Harness
5.1 Every Configuration Change Should Be Measured
Yan’s product evaluation method culminates in an evaluation harness that can run against each configuration change, whether the team is modifying a prompt, swapping a model, adjusting retrieval parameters, changing chunking strategy, editing guardrails, or experimenting with output format. He argues that the harness should consume experiment outputs, run relevant evaluators, aggregate results, and make improvements or regressions easy to see. (eugeneyan.com)
This is where evaluation becomes engineering infrastructure. A team that must manually inspect outputs after every small change will move slowly, skip evaluation under pressure, and eventually ship regressions. A team with a harness can iterate faster because each experiment produces comparable evidence. In Yan’s framing, the payoff of product evals is not only better measurement, but a tighter feedback loop that allows teams to improve the product more rapidly.
5.2 Evaluation Protects Against Regression
AI features are dynamic because their behavior can change when models are updated, prompts are rewritten, retrieval sources are refreshed, safety policies are adjusted, or user traffic shifts. This makes regression testing especially important. A change that improves relevance may reduce faithfulness. A new model may produce better prose but worse tool use. A shorter prompt may reduce latency but increase ambiguity. A retrieval update may improve recall while introducing irrelevant context.
An evaluation harness gives teams a way to detect these trade-offs before users do. It also helps teams distinguish between genuine improvement and movement from one defect class to another. Without this structure, teams may optimize whatever is most visible in the current sprint while damaging qualities that are harder to observe immediately.
6. AI Features Should Earn Trust Through Familiarity, Feedback, and Attribution
6.1 The Interface Is Part of the Evaluation
Yan does not evaluate AI features only at the model layer. In his writing on LLM systems and products, he argues that introducing users to AI features should rely on familiar UX patterns, because familiar interfaces reduce cognitive effort and help users focus on the task rather than the novelty of the interaction. He is skeptical that chat should be the default interface for every AI experience, noting that chat can increase user effort and therefore raise user expectations. (eugeneyan.com)
This expands evaluation beyond output quality. A feature can be technically capable but still productively weak if the interface asks too much of the user, hides controls, creates unrealistic expectations, or makes failure hard to diagnose. Evaluating an AI feature therefore requires asking whether the user understands what the feature can do, how to guide it, when to trust it, and how to recover when it fails.
6.2 Attribution and Feedback Create a Trust Loop
Yan also highlights citations, attributions, expert context, community signals, and user feedback as mechanisms that improve trust and create data for future evaluation. He notes that feedback can be explicit or implicit, and that feedback loops help teams evaluate overall system performance while also improving models, guardrails, and personalization. (eugeneyan.com)
This matters because many AI features fail not only by giving bad answers, but by giving answers that users cannot calibrate. A sourced answer is easier to inspect. A recommendation with a reason is easier to trust or reject. A feedback button attached to a specific output is more useful than a generic satisfaction survey. The strongest AI features therefore build evaluation into the user experience itself, allowing the product to learn from real usage while giving users more control over trust.
7. The Evaluation of AI Features Is Also an Organizational Problem
7.1 Tools Do Not Replace Process
The title of one of Yan’s essays makes the point directly: an LLM-as-judge will not save the product, but fixing the process might. His argument is that evaluation tools are useful only when embedded in a disciplined loop of sampling, annotation, hypothesis formation, experimentation, measurement, and monitoring. Teams that skip this process may accumulate dashboards and automated scores without developing a real understanding of product quality. (eugeneyan.com)
This is an organizational warning. AI evaluation requires time, attention, ownership, and recurring habits. Someone must define the criteria. Someone must review outputs. Someone must maintain datasets. Someone must inspect regressions. Someone must decide which failures are acceptable, which are launch blockers, and which require product changes rather than model changes. Evaluation is not merely a technical artifact; it is a team behavior.
7.2 Human Oversight Remains Necessary
Yan is clear that automated evaluators do not eliminate the need for human oversight. Teams still need to periodically sample outputs, annotate data, analyze user feedback, and improve the evaluators themselves. Automated evaluators amplify the annotation and feedback process, but they cannot compensate for neglect. (eugeneyan.com)
This is especially important for AI features deployed in changing environments. User needs evolve. New edge cases appear. Abuse patterns emerge. Data distributions shift. Models change behavior. A feature that evaluated well at launch may degrade as the product, users, or upstream model changes. Continuous human oversight keeps evaluation connected to reality rather than frozen in the assumptions of the original test set.
8. A Practical Framework Inspired by Eugene Yan
8.1 Define the Feature’s Job
The first question in evaluating an AI feature should be what job the feature is supposed to perform for the user. A summarizer may be expected to reduce reading time without losing decision-critical details. A search assistant may be expected to retrieve grounded answers from approved documents. A recommendation explanation may be expected to increase user confidence without inventing reasons. A coding assistant may be expected to produce runnable, secure, maintainable code. The evaluation architecture must begin from this job, because generic helpfulness is too vague to guide development.
8.2 Identify Failure Modes
The next step is to inspect real or realistic examples and identify the failures that would damage trust. These may include hallucination, irrelevance, missing context, unsafe advice, wrong tool use, excessive verbosity, weak personalization, unsupported claims, formatting violations, latency problems, refusal errors, or overconfident uncertainty. The goal is not to create an exhaustive philosophical taxonomy, but to define the defects that matter in the product’s actual context.
8.3 Build a Labeled Dataset
A useful evaluation dataset should include inputs, outputs, metadata, labels, and enough failure cases to measure whether evaluators can detect defects. The dataset should not rely only on synthetic failures if production-like failures are available, because exaggerated or artificial defects may train evaluators to catch the wrong things. The team should split the data into development and test sets so that evaluator prompts or rules are tuned on one portion and validated on held-out examples.
8.4 Create Separate Evaluators
Each important dimension should have its own evaluator, whether that evaluator is deterministic code, an assertion, a human rubric, an LLM-as-judge, or a hybrid mechanism. Faithfulness, relevance, safety, formatting, and concision should not be collapsed prematurely. Separate evaluators make the system easier to debug, because a regression can be traced to a specific quality dimension rather than buried inside an opaque overall score.
8.5 Run Experiments Against the Harness
Every meaningful change should be run through the evaluation harness against the baseline. The team should compare pass rates, defect rates, pairwise wins, confidence intervals, cost, latency, and qualitative error samples. If a change improves one metric while damaging another, the team should make an explicit product decision rather than accidentally accepting the trade-off.
8.6 Monitor After Launch
The final step is continuous monitoring. The team should sample production outputs, review user feedback, track defect patterns, refresh evaluation datasets, and recalibrate automated evaluators. This is where evaluation becomes a living product system rather than a launch checklist.
Conclusion: The Mature Evaluation of AI Features
Eugene Yan’s evaluation philosophy can be summarized in one sentence: AI features should be built, shipped, and improved through disciplined feedback loops rather than vibes, demos, benchmarks, or magical thinking. His approach is practical because it accepts the messy reality of AI products: outputs are open-ended, failures are context-dependent, human labels are imperfect, automated judges need calibration, and product quality cannot be reduced to a single universal score.
The deeper importance of Yan’s work is that it reframes evaluation as the architecture of trustworthy AI product development. Evaluation defines what success means, reveals where the system fails, gives teams confidence to iterate, protects users from regressions, and creates the data flywheel through which AI features improve. A team that evaluates well can move faster because it knows what it is measuring. A team that evaluates poorly may move quickly at first, but it eventually becomes trapped by uncertainty, regressions, stakeholder disagreement, and user distrust.
In this sense, the evaluation of AI features is not a narrow technical concern. It is the discipline that separates impressive prototypes from reliable products. It is the mechanism that turns AI from a demo into a dependable user experience. And in Eugene Yan’s framing, it is not optional infrastructure; it is the foundation.