Purpose

ConveYour tests material AI-enabled features before release to determine whether they perform their intended function reliably enough for the proposed use, behave safely under realistic conditions, and include appropriate controls for known limitations.

Testing is not a guarantee that an AI system will always be correct. It is a documented process for finding likely errors and unsafe behavior before customers rely on a feature.

Risk-based testing

The depth of testing matches the potential impact of the feature.

A low-impact drafting or summarization feature may require representative scenario testing and human quality review. A feature used in an employment-related workflow, or one that could materially affect a person, receives more rigorous review of accuracy, fairness risks, human oversight, inappropriate output, and the consequences of failure.

Guiding Team Principle: Before testing, the feature owner documents:

• The feature's intended use and prohibited uses
• The users, data, and workflow involved
• What a useful and acceptable output looks like
• Relevant failure modes and potential harm
• The human-review controls required
• The test plan, release criteria, and person responsible for approval

Accuracy and quality testing

ConveYour tests whether the feature produces outputs that are relevant, grounded in the provided information, understandable, and useful for the intended workflow.

Testing may include:

• Representative scenarios based on realistic customer workflows
• Review of factual consistency with the input information
• Checks for omission of important information, unsupported statements, incorrect summaries, or unsuitable recommendations
• Review by a person with sufficient knowledge of the workflow