Email Validation Test Cases & Send-Decision Benchmark

Dataset v1.0 (generated July 6, 2026) · Page updated November 9, 2026 · 40 synthetic scenarios · License CC BY 4.0

An open, machine-readable suite of 40 author-defined, synthetic email validation scenarios illustrating selected policy cases across malformed syntax, invalid recipients, catch-all domains, disposable addresses, role accounts, SMTP 4xx transient deferrals and 552 over-quota conditions, unresolved unknowns, and spam-trap boundaries. Each row pairs an illustrative expected routing action with a stated InboxPolicy policy outcome evaluated against the ordered rule list below.

Scope & provenance (synthetic scenarios, not measured outcomes)

Author-defined fixtures, not live executions or prevalence data: None of the 40 scenarios below is a recorded live SMTP transcript or measured production sample, and category counts do not represent real-world B2B catch-all prevalence or live verifier accuracy. Scenarios #1–#6, #8–#32, and #35–#40 use RFC 2606 / RFC 6761 reserved example domains (example.com, example.org, example.net, and .example); scenario #7 uses an illustrative example.edu address (not enumerated in RFC 2606), and #33–#34 are malformed strings with no domain. In automated CI, always load benchmark.json from a local file with external network calls mocked or isolated.

Download benchmark.json v1.0 (CC BY 4.0) ↓

A three-layer framework for email test cases

A practical way to structure email validation QA is to separate syntax filtering, mailbox/policy routing, and message-delivery checks so test suites do not conflate input parsing with live deliverability:

Testing layerWhat this layer evaluatesScenarios in v1.0Documented InboxPolicy behavior
1. Input field & syntax precheck Missing @ (not-an-email), missing domain (user@), and missing local-part (@example.com) in #33–#35 (extend with whitespace, length, or consecutive-dot vectors for fuller parser tests). 3 scenarios (#33–#35, syntax_invalid) Deliberately limited, pragmatic syntax filter in packages/core/src/syntax.ts (not a full RFC 5321 / RFC 5322 grammar). On a schema-valid POST /v1/decide body, malformed strings are caught by a local syntax pre-check before the policy runs, returning avoid and status: "syntax_error" at 0 credits (POST /v1/verify returns HTTP 422).
2. Domain, mailbox & policy routing Normalized status and risk signals from verification checks: accepted vs. rejected recipients, catch-all domains, disposable flags, role-based addresses, 4xx transient deferrals or 552 (5xx) quota conditions normalized as temporary_unavailable, and unresolved unknown checks. 36 scenarios (#1–#32, #36–#39) Evaluates local syntax precheck, 72-hour cache (unless force_refresh: true), and configured verifier signals (which may escalate primary unknown results to a wholesale fallback verifier per /docs) through the ordered 8-rule precedence list below.
3. Message delivery & trap databases Sender SPF/DKIM/DMARC alignment, content rendering, opt-out headers, and dedicated spam-trap or abuse databases. 1 boundary scenario (#40, spam_trap) Per /pricing, ZeroBounce maintains a dedicated spam-trap and abuse database that InboxPolicy does not offer. Authored scenario #40 illustrates why a trap assumed to normalize as status: "valid" with confidence >= 70 and no risk flags reaches Rule 8 (send).

Ordered policy precedence: first matching rule wins (packages/core/src/policy.ts)

In benchmark.json (v1.0), expected_verdict records the scenario author's illustrative target action, while inboxpolicy_verdict records the stated outcome of InboxPolicy's policy function (packages/core/src/policy.ts:17–50, exposed via POST /v1/decide and MCP decide_send) under each scenario's assumed inputs. Rules are evaluated strictly from 1 to 8 and the first matching rule wins:

OrderCondition (first match wins)Returned actionPolicy note (usual reason)
1status === "invalid" or status === "syntax_error"avoidMailbox is invalid or malformed.
2status === "temporary_unavailable"retry_laterTemporary verification issue. Retry later.
3signals.disposable === trueavoidDisposable email detected.
4status === "unknown"reviewMailbox could not be verified confidently.
5signals.catch_all === truesend_with_caution if strictness === "aggressive"; else reviewCatch-all mailbox detected. (or: Review before sending.)
6signals.role_account === true and strictness === "strict"reviewRole-based address detected under strict policy.
7confidence < 70send_with_cautionMailbox verified with moderate confidence.
8Otherwise (remaining valid results with confidence >= 70)sendVerified mailbox with acceptable risk.

The API's reason field normally carries this policy note. A verification provider's own note, or the syntax pre-check's note, can appear instead (packages/core/src/provider.ts, packages/core/src/syntax.ts).

Normalization & companion JSON note: Before Rules 1–8 run, packages/core/src/provider.ts:7–16, 54–64 normalizes upstream provider statuses: raw "greylisted" maps to status: "temporary_unavailable" (Rule 2), while raw "catch_all" maps to status: "unknown" with signals.catch_all: true (with signals.disposable: false, matching Rule 4 → review before Rule 5 even under aggressive strictness). Four scenarios (#9, #18, #23, and #40) are authored disagreements. The companion benchmark.json v1.0 draft removes legacy anecdotal notes on #3 and #19 and updates scenario notes on #18, #29–#32, #38, and #39; full per-scenario assumptions for all 40 cases appear below.

All 40 synthetic, author-defined scenarios

#EmailCategoryExpected verdictStated policy outcomeAssumed inputs & winning rule (policy.ts)
1 [email protected] valid send send Synthetic valid case: assumes status: "valid", catch_all/disposable/role_account: false, confidence: 95; no Rule 1–7 match → Rule 8 returns send.
2 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 92; no Rule 1–7 match → Rule 8 returns send.
3 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 90 (0–100 score; do not interpret as a probability without calibration evidence); no Rule 1–7 match → Rule 8 returns send.
4 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 93; no Rule 1–7 match → Rule 8 returns send.
5 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 91; no Rule 1–7 match → Rule 8 returns send.
6 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 88; no Rule 1–7 match → Rule 8 returns send.
7 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 94 → Rule 8 send. Note: example.edu is illustrative, not an RFC 2606 enumerated reserved name.
8 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 89; no Rule 1–7 match → Rule 8 returns send.
9 [email protected] valid send send_with_caution Authored disagreement: assumes status: "valid", catch_all/disposable/role_account: false, and hypothetical confidence: 68 (<70); with no Rule 1–6 match, Rule 7 returns send_with_caution.
10 [email protected] valid send send Synthetic valid case: assumes status: "valid", all risk flags false, confidence: 92; no Rule 1–7 match → Rule 8 returns send.
11 [email protected] invalid avoid avoid Synthetic invalid case: assumes hypothetical 550 rejection normalized to status: "invalid"; Rule 1 matches first and returns avoid.
12 [email protected] invalid avoid avoid Synthetic invalid case: assumes hypothetical user-unknown rejection normalized to status: "invalid"; Rule 1 returns avoid.
13 [email protected] invalid avoid avoid Synthetic invalid case: assumes hypothetical NoSuchUser rejection normalized to status: "invalid"; Rule 1 returns avoid.
14 [email protected] invalid avoid avoid Synthetic invalid case: assumes hypothetical rejection normalized to status: "invalid"; Rule 1 returns avoid.
15 [email protected] invalid avoid avoid Synthetic invalid case: assumes recipient rejection normalized to status: "invalid"; Rule 1 returns avoid.
16 [email protected] invalid avoid avoid Synthetic invalid case: assumes recipient rejection normalized to status: "invalid"; Rule 1 returns avoid.
17 [email protected] invalid avoid avoid Synthetic invalid case: assumes recipient rejection normalized to status: "invalid"; Rule 1 returns avoid.
18 [email protected] invalid avoid review Authored disagreement: assumes inconclusive probe responses normalized to status: "unknown" with signals.disposable: false (no Rule 1–3 match), so Rule 4 returns review instead of avoid.
19 [email protected] catch_all review review Synthetic catch-all case: assumes signals.catch_all: true, signals.disposable: false, status: "valid" (or "unknown"), and strictness: "default"; with no Rule 1–3 match, Rule 4 or Rule 5 returns review.
20 [email protected] catch_all review review Synthetic catch-all case: assumes signals.catch_all: true, signals.disposable: false, status: "valid" (or "unknown"), and strictness: "default"; no Rule 1–3 match → Rule 4 or Rule 5 returns review.
21 [email protected] catch_all review review Synthetic catch-all case: assumes signals.catch_all: true, signals.disposable: false, status: "valid" (or "unknown"), and strictness: "default"; no Rule 1–3 match → Rule 4 or Rule 5 returns review.
22 [email protected] catch_all review review Synthetic catch-all case: assumes signals.catch_all: true, signals.disposable: false, status: "valid" (or "unknown"), and strictness: "default"; no Rule 1–3 match → Rule 4 or Rule 5 returns review.
23 [email protected] catch_all review send_with_caution Authored disagreement: assumes status: "valid", signals.catch_all: true, signals.disposable: false, and strictness: "aggressive"; with no Rule 1–4 match, Rule 5 returns send_with_caution (raw provider "catch_all" normalizes to status: "unknown", matching Rule 4 → review first).
24 [email protected] catch_all review review Synthetic catch-all case: assumes signals.catch_all: true, signals.disposable: false, status: "valid" (or "unknown"), and strictness: "default"; no Rule 1–3 match → Rule 4 or Rule 5 returns review.
25 [email protected] disposable avoid avoid Synthetic disposable case: assumes signals.disposable: true and status valid or unknown (no Rule 1–2 match); Rule 3 returns avoid.
26 [email protected] disposable avoid avoid Synthetic disposable case: assumes signals.disposable: true and status valid or unknown (no Rule 1–2 match); Rule 3 returns avoid.
27 [email protected] disposable avoid avoid Synthetic disposable case: assumes signals.disposable: true and status valid or unknown (no Rule 1–2 match); Rule 3 returns avoid.
28 [email protected] disposable avoid avoid Synthetic disposable case: assumes signals.disposable: true and status valid or unknown (no Rule 1–2 match); Rule 3 returns avoid.
29 [email protected] role_based send_with_caution send_with_caution Synthetic role case: assumes status: "valid", signals.role_account: true, catch_all: false, disposable: false, strictness: "default", and confidence < 70; with no Rule 1–6 match, Rule 7 returns send_with_caution (Rule 6 returns review under strict; Rule 8 returns send at confidence >= 70).
30 [email protected] role_based send_with_caution send_with_caution Synthetic role case: assumes status: "valid", signals.role_account: true, catch_all: false, disposable: false, strictness: "default", and confidence < 70; no Rule 1–6 match → Rule 7 returns send_with_caution.
31 [email protected] role_based send_with_caution send_with_caution Synthetic role case: assumes status: "valid", signals.role_account: true, catch_all: false, disposable: false, strictness: "default", and confidence < 70; no Rule 1–6 match → Rule 7 returns send_with_caution.
32 [email protected] role_based send_with_caution send_with_caution Synthetic role case: assumes status: "valid", signals.role_account: true, catch_all: false, disposable: false, strictness: "default", and confidence < 70; no Rule 1–6 match → Rule 7 returns send_with_caution.
33 not-an-email syntax_invalid avoid avoid Missing @; on schema-valid POST /v1/decide calls, the local syntax pre-check returns status: "syntax_error" and avoid at 0 credits before the policy runs.
34 user@ syntax_invalid avoid avoid Missing domain; the local syntax pre-check returns status: "syntax_error" and avoid at 0 credits on schema-valid /v1/decide requests, before the policy runs.
35 @example.com syntax_invalid avoid avoid Missing local-part; the local syntax pre-check returns status: "syntax_error" and avoid at 0 credits before the policy runs or any upstream verifier is called.
36 [email protected] mailbox_full retry_later retry_later Synthetic over-quota case: assumes an SMTP 552 (5xx quota-exceeded class) normalized by the provider to status: "temporary_unavailable" (no Rule 1 match); Rule 2 returns retry_later (eligible cache hits for the same normalized address and strictness within 72h with force_refresh: false return cached data at 0 credits; pass force_refresh: true for a fresh check).
37 [email protected] mailbox_full retry_later retry_later Synthetic over-quota case: assumes provider normalization to status: "temporary_unavailable" (no Rule 1 match); Rule 2 returns retry_later.
38 [email protected] unknown review review Synthetic unknown case: assumes final normalized status: "unknown" (after any fallback verifier) and signals.disposable: false (no Rule 1–3 match), so Rule 4 returns review. Note: if a provider returns raw status "greylisted", provider.ts normalizes it to temporary_unavailable and Rule 2 returns retry_later.
39 [email protected] unknown review review Synthetic unknown case: assumes an inconclusive gateway check normalized to status: "unknown" with signals.disposable: false (no Rule 1–3 match), so Rule 4 returns review.
40 [email protected] spam_trap avoid send Authored boundary disagreement: assumes a hypothetical spam trap accepted probing and normalized to status: "valid", catch_all/disposable/role_account: false, and confidence >= 70; with no Rule 1–7 match, Rule 8 returns send. Per /pricing, ZeroBounce maintains a dedicated spam-trap and abuse database that InboxPolicy does not offer (ZeroBounce separately documents top-level spamtrap for addresses it believes are traps and possible_trap for keywords that might correlate with traps).

Running benchmark.json offline with Python 3

The standard-library Python 3 example below reads a local copy of benchmark.json (v1.0) without making live HTTP or SMTP calls, checks the dataset distribution, and exercises an example downstream queue-routing function across all 40 scenarios and all nine categories (swap route_outbound_lead for your own application's handler when integrating into your test suite):

from collections import Counter
import json
from pathlib import Path
import sys

BENCHMARK_PATH = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("benchmark.json")
if not BENCHMARK_PATH.exists():
    for candidate in (
        Path("research/2026-11-09-content-wave/drafts/benchmark.json"),
        Path("site/benchmark.json"),
    ):
        if candidate.exists():
            BENCHMARK_PATH = candidate
            break

data = json.loads(BENCHMARK_PATH.read_text(encoding="utf-8"))
assert data["version"] == "1.0"
assert data["scenario_count"] == len(data["scenarios"]) == 40

def route_outbound_lead(action: str) -> str:
    """Example queue router mapping InboxPolicy's 5 actions to application queues."""
    mapping = {
        "send": "primary_sequence",
        "send_with_caution": "throttled_caution_pool",
        "review": "manual_review_queue",
        "retry_later": "delayed_reverify_queue",
        "avoid": "global_suppression",
    }
    if action not in mapping:
        raise ValueError(f"Unhandled action: {action}")
    return mapping[action]

category_counts = Counter()
queue_counts = Counter()
disagreements = []

for case in data["scenarios"]:
    verdict = case["inboxpolicy_verdict"]
    expected = case["expected_verdict"]
    category = case["category"]
    queue = route_outbound_lead(verdict)

    category_counts[category] += 1
    queue_counts[queue] += 1
    if verdict != expected:
        disagreements.append((case["id"], category, expected, verdict))

    # Category assertions covering all 9 categories in benchmark.json v1.0
    if category in {"syntax_invalid", "disposable"}:
        assert queue == "global_suppression"
    elif category == "invalid":
        assert queue in {"global_suppression", "manual_review_queue"}
    elif category in {"catch_all", "unknown"}:
        assert queue in {"manual_review_queue", "throttled_caution_pool"}
    elif category == "mailbox_full":
        assert queue == "delayed_reverify_queue"
    elif category == "role_based":
        assert queue in {"throttled_caution_pool", "manual_review_queue", "primary_sequence"}
    elif category == "valid":
        assert queue in {"primary_sequence", "throttled_caution_pool"}
    elif category == "spam_trap":
        assert expected == "avoid" and queue == "primary_sequence"
    else:
        raise AssertionError(f"Uncovered category: {category}")

assert dict(category_counts) == data["distribution"]
assert [row[0] for row in disagreements] == [9, 18, 23, 40]

print(f"Loaded {BENCHMARK_PATH} -> {data['name']} v{data['version']} ({data['scenario_count']} synthetic scenarios)")
print("Categories:", dict(category_counts))
print("Routed queues:", dict(queue_counts))
print("Authored disagreements:", disagreements)

Method & standards note (updated November 9, 2026): SMTP reply classifications and reserved-domain references cite RFC 5321, RFC 5322, RFC 2606, and RFC 6761. InboxPolicy syntax, normalization, and ordered policy precedence match packages/core/src/syntax.ts, packages/core/src/provider.ts, packages/core/src/policy.ts, /docs, and /pricing.

Frequently asked questions

What is included in this email validation test-case dataset?

Forty author-defined, synthetic email-verification scenarios (benchmark.json v1.0) spanning nine categories: 10 valid, 8 invalid, 6 catch_all, 4 disposable, 4 role_based, 3 syntax_invalid, 2 mailbox_full, 2 unknown, and 1 spam_trap. Each scenario pairs an illustrative expected action with a stated policy outcome from the ordered first-match rule list in packages/core/src/policy.ts under assumed normalized signals and strictness, including four authored non-agreement cases (#9, #18, #23, and #40).

Does this 40-scenario dataset measure real-world catch-all prevalence or live accuracy?

No. All 40 rows are synthetic, author-defined scenarios designed to illustrate selected policy cases, not recorded live SMTP transcripts or a representative sample of production email traffic. Category counts (for example, 6 of 40 rows being catch_all) reflect fixture composition, not real-world prevalence or live accuracy.

How do email syntax test cases differ from mailbox verification and message-delivery QA?

A practical framework separates email QA into three layers: (1) input syntax filtering (such as missing @, missing domain, or missing local-part in scenarios #33–#35); (2) domain, mailbox, and policy routing (evaluating normalized status, boolean signals, confidence, and strictness through an ordered first-match rule list); and (3) message-delivery and reputation QA (SPF/DKIM/DMARC alignment, rendering, and dedicated spam-trap or abuse-address databases).

Are the email addresses in benchmark.json safe for automated CI/CD tests?

Use them strictly with offline file loading, mocks, and network isolation. Scenarios #1–#6, #8–#32, and #35–#40 use RFC 2606 reserved names (example.com, example.org, example.net, and .example), while #7 uses an illustrative example.edu domain (not enumerated in RFC 2606) and #33–#34 are malformed strings without domains. Because domain labels alone do not prevent local DNS or SMTP routing, CI suites should read benchmark.json locally and stub external network calls.

Why is scenario #40 (spam_trap) listed with a stated InboxPolicy outcome of 'send'?

Scenario #40 is an authored synthetic boundary case, not a live trap test. Per InboxPolicy's pricing page (/pricing), ZeroBounce maintains a dedicated spam-trap and abuse database that InboxPolicy does not offer. Because some spam traps accept mail over SMTP without distinguishing trap signals, an address normalized as status valid with catch_all/disposable/role_account false and confidence >= 70 matches no higher-priority rule (1–7) and returns send at Rule 8 in packages/core/src/policy.ts. Teams needing dedicated trap screening can evaluate vendors such as ZeroBounce, which separately documents top-level spamtrap (addresses ZeroBounce believes are spam traps) and do_not_mail sub-status possible_trap (keywords that might correlate with possible traps).

Under what license can I reuse benchmark.json?

Creative Commons Attribution 4.0 (CC BY 4.0). You may copy, adapt, and include benchmark.json in commercial or open-source test suites provided you give appropriate credit to InboxPolicy (https://inboxpolicy.com/benchmark), link to the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), and indicate if changes were made.

Download benchmark.json v1.0 (CC BY 4.0) ↓

Raw JSON companion: /benchmark.json (v1.0). See The Send Decision Framework and API Docs for live endpoint contracts.