Evaluate AI responses against a stable test set

An AI application may answer the same task differently across runs. Controlled improvement therefore requires a stable set of tasks with understandable evaluation criteria. Include actual usage patterns and difficult edge cases.

Define business quality explicitly

• Include typical tasks, incomplete input and conflicting sources.
• Record essential facts and claims that must not appear.
• Assess source grounding, completeness and necessary clarification separately.
• Compare results after changes to models, prompts or source data.

An assistant that asks a precise follow-up question when information is missing may be more useful than one that produces a fluent unsupported answer. Evaluation must capture that distinction. Text similarity alone is often insufficient.

Expand the test set with observed failures. Remove personal and confidential information or replace it with suitable synthetic examples. This creates a traceable basis for deciding which changes should reach production.