Email Verification Benchmark Dataset

Dataset v1.0 · Published July 6, 2026 · 40 scenarios · License CC BY 4.0

A public, synthetic benchmark of 40 email-verification scenarios across the nine categories InboxPolicy's send-decision API distinguishes. Each scenario pairs a ground-truth Expected verdict with the verdict InboxPolicy would actually return, so the dataset is an honesty gauge: it includes realistic agreement and disagreement rather than a perfect scorecard.

At a glance

40 synthetic scenarios across 9 categories. Every address uses a reserved example domain — none is a real third-party inbox, none contains a secret. Download the raw data and reuse it under CC BY 4.0.

Download dataset (CC BY 4.0) ↓

How to read this table

Expected verdict is the ground-truth action a perfect verifier would return. InboxPolicy result is what InboxPolicy's send-decision API (decide_send / verify_email) actually returns for that address. Where the two columns disagree, the difference is deliberate and explained in Notes — not hidden. Verdicts use InboxPolicy's five actions: send, send_with_caution, review, retry_later, avoid.

The 40 scenarios

#EmailCategoryExpected verdictInboxPolicy resultNotes
1 [email protected] valid send send Clean signal: SMTP RCPT accepted, no catch-all signature, confidence 95.
2 [email protected] valid send send Verified mailbox, confidence 92.
3 [email protected] valid send send Confidence 90; sits in the ~90% valid-verdict agreement band vs MillionVerifier.
4 [email protected] valid send send Confidence 93.
5 [email protected] valid send send Confidence 91.
6 [email protected] valid send send Confidence 88.
7 [email protected] valid send send Confidence 94.
8 [email protected] valid send send Confidence 89.
9 [email protected] valid send send_with_caution Mailbox confirmed but provider reputation lowered confidence to 68; InboxPolicy conservatively returns send_with_caution rather than send. A realistic non-agreement case.
10 [email protected] valid send send Confidence 92.
11 [email protected] invalid avoid avoid Clear 550 reject; high confidence.
12 [email protected] invalid avoid avoid 550 user unknown.
13 [email protected] invalid avoid avoid 550 NoSuchUser.
14 [email protected] invalid avoid avoid 550 mailbox unavailable.
15 [email protected] invalid avoid avoid Rejected at RCPT.
16 [email protected] invalid avoid avoid 550 unknown.
17 [email protected] invalid avoid avoid 550 rejected.
18 [email protected] invalid avoid review Server returned an inconsistent SMTP response across probes; InboxPolicy could not confirm the rejection and returned review rather than risk a false avoid. Realistic non-agreement case.
19 [email protected] catch_all review review catch_all evidence tagged; default policy maps to review. ~30-40% of B2B addresses look like this.
20 [email protected] catch_all review review Tagged catch_all; review by default, never guessed safe.
21 [email protected] catch_all review review review with catch_all flag set.
22 [email protected] catch_all review review review (catch_all).
23 [email protected] catch_all review send_with_caution Under an aggressive policy configuration the same catch_all evidence maps to send_with_caution instead of review; InboxPolicy never returns send for an unconfirmed mailbox.
24 [email protected] catch_all review review review (catch_all).
25 [email protected] disposable avoid avoid Disposable domain flagged; maps to avoid. Domain name is synthetic.
26 [email protected] disposable avoid avoid Disposable flagged and routed to avoid.
27 [email protected] disposable avoid avoid Disposable domain leads to avoid.
28 [email protected] disposable avoid avoid Disposable flagged, returned avoid.
29 [email protected] role_based send_with_caution send_with_caution Role-based address flagged; deliverable with moderate confidence and elevated reputation risk.
30 [email protected] role_based send_with_caution send_with_caution Role address; send_with_caution.
31 [email protected] role_based send_with_caution send_with_caution Role address flagged; send_with_caution.
32 [email protected] role_based send_with_caution send_with_caution Role address; send_with_caution.
33 not-an-email syntax_invalid avoid avoid Rejected locally before SMTP at 0 credits; malformed input is never billed.
34 user@ syntax_invalid avoid avoid Malformed; rejected locally, free of charge.
35 @example.com syntax_invalid avoid avoid Rejected before any SMTP check runs; 0 credits.
36 [email protected] mailbox_full retry_later retry_later 552 over quota; transient condition, re-check later (the re-check is usually free within 72h).
37 [email protected] mailbox_full retry_later retry_later 552 over quota; retry_later.
38 [email protected] unknown review review Honest unknown preserved as a first-class state and mapped to review; never guessed safe. May escalate to a wholesale fallback verifier.
39 [email protected] unknown review review Unknown mapped to review (first-class state), not over-promoted to send.
40 [email protected] spam_trap avoid send InboxPolicy confirmed the mailbox over SMTP and returned send because it maintains no spam-trap database; a live trap looks valid to any SMTP-only engine. This is the documented gap vs ZeroBounce and the single most important disagreement in the set; route trap detection to a specialist.

Frequently asked questions

What is the InboxPolicy Email Verification Benchmark?

It is a public, synthetic benchmark dataset of 40 email-verification scenarios spanning the nine categories InboxPolicy's send-decision API distinguishes. Each scenario pairs a ground-truth expected verdict with the verdict InboxPolicy would actually return, so the dataset doubles as an honesty gauge: it includes realistic agreement and disagreement rather than a perfect scorecard.

How many scenarios does the benchmark contain, and what categories does it cover?

Forty scenarios across nine categories: 10 valid, 8 invalid, 6 catch_all, 4 disposable, 4 role_based, 3 syntax_invalid, 2 mailbox_full, 2 unknown, and 1 spam_trap. These mirror the verdict types the send-decision API produces (send, send_with_caution, review, retry_later, avoid).

Are the email addresses in the benchmark real inboxes?

No. Every address is synthetic and uses reserved example domains (example.com, example.org, example.net, and .example subdomains). None reaches a real third-party inbox and none contains a secret or credential. The SMTP transcripts behind each verdict are representative, not live captures.

Why does InboxPolicy return 'send' for the spam-trap scenario?

Because InboxPolicy runs syntax, MX, and live SMTP checks and has no spam-trap database. A pristine trap is a real, SMTP-accepting mailbox, so to any SMTP-only engine it looks valid. This is the single most important disagreement in the set and the documented gap versus tools like ZeroBounce that maintain a trap database; route trap detection to a specialist.

Under what license is the dataset released, and can I reuse it?

The dataset is released under Creative Commons Attribution 4.0 (CC BY 4.0). You may copy, redistribute, remix, and build on it for any purpose, including commercial, as long as you credit InboxPolicy (https://inboxpolicy.com/). The raw data is available at benchmark.json.

How should I read the agreement between 'Expected verdict' and 'InboxPolicy result'?

The 'Expected verdict' column is the ground-truth action a perfect verifier would return; the 'InboxPolicy result' column is what InboxPolicy's send-decision API actually returns for that address. Where they differ (for example the confidence-68 valid case, the inconsistent-probe invalid case, the aggressive catch_all case, and the spam trap), the difference is deliberate and explained in the Notes column rather than hidden.

Download dataset (CC BY 4.0) ↓

Raw JSON mirrors this page: ./benchmark.json. Credit InboxPolicy (https://inboxpolicy.com/) per the license.