Write the hypothesis and one change

Examples: “Removing the tracking redirect changes placement in our controlled provider sample” or “Variant B’s attachment type changes SMTP acceptance.” Avoid testing a new From domain, IP pool, template, subject, send time, and audience at once.

Freeze the variables that should remain equal: provider account, From/return-path/DKIM identities, sending IP/pool, list provenance, suppression, volume, rate, time window, and MIME generation. Record software/configuration versions.

Choose a comparison method

For seed accounts, use paired runs and alternate order where possible so one variant is not always first. Space messages enough to avoid the test itself creating burst or duplicate-content effects, but keep the environmental window comparable. For a real audience experiment, randomize authorized recipients within the same eligible population and use a sample approved for the decision.

Do not send both variants to people who did not expect duplicate messages. Seed observations and real audience outcomes serve different purposes.

Verify the manipulation

Inspect the received originals. Confirm only the intended factor differs and both variants have the same authentication identities/results. Compare message size, MIME structure, links, headers, and provider routing. If the sender platform silently selected another pool or rewrote one variant, the test is confounded.

Record all outcomes

Capture SMTP accepted/rejected/deferred, inbox/spam/other/missing/unknown, monitor health, provider/account, and times. Show the full matrix, not only inbox percentage. With small samples, a difference of one seed can look large; report counts and avoid statistical certainty the design cannot support.

Provider dashboards are delayed aggregate evidence. Use complaint, delivery-error, and authentication changes as context over an appropriate window, not as immediate per-message confirmation.

Decide the next test

Repeat a material result on a second controlled run. If it persists, test the suspected mechanism or use a bounded real-audience rollout with stop conditions. If it does not, keep the outcome inconclusive.

A fair comparison narrows a question. It does not prove that a word, image, or link is a universal spam trigger across providers and recipients.