Define the product decision first

Decide whether the verifier will block signup, request confirmation, prioritize manual review, clean an existing authorized list, or estimate risk. The cost of a false invalid result differs from the cost of accepting an ambiguous address. One aggregate “accuracy” number cannot express that tradeoff.

Define output classes before testing: valid/confirmed, invalid with strong evidence, accept-all/ambiguous, temporary, unavailable, and unknown. Keep provider risk labels separate from protocol evidence.

Build an authorized sample

Use data you are permitted to process and minimize personal information. Stratify by ordinary providers, corporate/catch-all domains, disposable/privacy services, internationalized addresses, malformed input, null/no-MX domains, temporary failures, and relevant geographies/providers. Include enough recent examples in each category to report uncertainty; do not overrepresent easy syntax failures.

Create labels from observable outcomes such as controlled confirmation, authoritative DNS, or later permanent delivery failure. A past successful delivery does not guarantee current existence; date every label and allow “indeterminate.” Have disputed cases reviewed without exposing addresses to unnecessary staff.

Run a frozen evaluation

Record verifier version, provider, configuration, time, egress, retries, and source-data versions. Prevent the evaluated service from training on or manually recognizing the test set where that would invalidate the comparison. Repeat temporary/unknown cases only under the documented production policy.

Calculate per-class precision and recall where labels support them, plus false-invalid rate, false-valid rate, unknown/abstention rate, latency, provider error rate, and cost. Break results down by cohort. Report sample sizes and confidence limits rather than decimals that imply impossible certainty.

Test the actual policy

Apply the proposed product decision to the results. Count legitimate users blocked, abuse allowed, reviews created, and confirmations required. Compare alternatives such as treating unknown as review instead of invalid. Perform a shadow period before automatic rejection.

Monitor drift

After launch, track confirmation, bounce, appeal, provider-timeout, and cohort metrics. Re-evaluate when parsers, provider data, retry rules, or thresholds change. Keep a rollback.

The evaluation succeeds when stakeholders can see where the verifier is uncertain and what each error costs. A service that abstains honestly can be safer than one that labels every address “valid” or “invalid” to advertise a larger accuracy number.