Email webhook latency benchmark: a measurement protocol

What exactly should an email webhook latency benchmark measure?

Stop the clock when your receiving endpoint obtains the webhook, not when your application finishes handling the email. Start it at mailbox arrival only when you have an independent timestamp with suitable semantics and precision.

An incoming email webhook is an HTTP notification associated with an email event. Mailera connects existing mailboxes to applications through webhooks and other interfaces; your benchmark needs to distinguish the mailbox side from the receiving application.

Use these names in your measurement records. They are benchmark variables, not Mailera payload fields.

Define arrival-to-webhook latency as t_endpoint − t_arrival. Report detection-to-endpoint latency separately as t_endpoint − t_detected when instrumentation supports it.

If you only observe a message through a mailbox read, name that starting point t_first_observed. Report first-observed-to-webhook latency, including negative values if the webhook reaches your collector before the observer sees the message. Do not relabel that interval as arrival latency.

Which timestamp can represent mailbox arrival?

Choose the timestamp source before collecting samples. Do not substitute SMTP submission time, the sender's Date header, or a Mailera observation timestamp for an independently established mailbox-arrival boundary.

Google provides a useful example of why semantics matter. Its Gmail message resource documentation, read on Oct 25, 2026, defines internalDate in epoch milliseconds. For normally SMTP-received messages, it represents when Google originally accepted the message. For API-migrated mail, the client can configure it to derive from the Date header.

That makes Google acceptance a possible named measurement boundary for an appropriate SMTP test cohort, not a universal definition of mailbox visibility. Mailera's Gmail connector uses IMAP with app-password authentication; do not assume a Gmail API field is the timestamp used by that connector.

For every provider, document the chosen arrival signal and its precision. Treat IMAP INTERNALDATE and provider-specific timestamps as candidates requiring semantic checks, rather than interchangeable clocks. A field named received_at is also insufficient by itself to define your benchmark boundary.

Synchronize collector and observer clocks, then record offset bounds. Timestamp resolution is not the same as accuracy. Preserve that uncertainty in the results instead of ranking configurations whose apparent differences fall inside it.

How should you build comparable test cohorts?

Compare configurations, not provider names alone. Outlook.com and Microsoft 365 should have separate account-type labels, even when both use the same Mailera connection path.

Publish a protocol before the run:

1. Choose owned test mailboxes. Record provider, account type, account count, connection mode, and run dates. The Microsoft connection guide and custom IMAP guide cover different connection paths; keep those distinctions in the dataset.
2. Record the environment. Include collector region, software versions, clock synchronization details, and any deployment metadata available to you. Record how each timestamp is obtained.
3. Define mailbox states operationally. Write down how you establish active, idle, and sleeping cohorts. Record state observations and any interaction that could change them.
4. Control the messages. Keep sender, content size, pacing, and observation windows comparable. Give each test message a unique synthetic correlation marker and check that you can match it through the complete path during setup.
5. Interleave the runs. Alternate or randomize provider cohorts rather than testing each provider in a different time period.
6. Fix the analysis rules. Specify sample size, cutoff, exclusions, duplicate handling, percentile method, and uncertainty reporting before collecting the comparison dataset.

Separate historical imports from new test arrivals. Keep Inbox and Junk cohorts distinct, and record the observed event type and folder rather than pooling them by convenience. Mailera's event catalog includes both message.received and message.junk.

How do you set up the Mailera side?

Use a dedicated mailbox group and collector for the test. The Mailera developer overview describes the integration surface; use the API documentation for webhook creation and signature verification details.

1. In Mailboxes → Add mailbox, connect your owned test accounts. For Microsoft, choose Sign in with Microsoft. Record the connection test's Live monitoring result where shown, and let initial message loading finish before the live-arrival cohort begins.
2. In Settings, open the API key tab and choose Create key. Set Mailbox scope to the test group. Use Webhook permission for webhook management, adding Read only when your collector workflow needs mailbox or message reads. Set an Expiration and optional IP restrictions.
3. Prepare an HTTPS collector that records receipt before application processing. Keep acknowledgment status and timing as separate observations.
4. Create the subscription with POST /webhooks using the documented request schema. The API base is https://mailera.app/api/public/v1; authentication uses Authorization: Bearer $MAILERA_API_KEY. Store the signing secret securely when creation returns it: it is returned only once.
5. Validate request signatures using the documented procedure. Separate rejected requests from accepted benchmark observations rather than letting unauthenticated traffic affect your results.
6. Run the controlled message schedule, then export redacted collector records, correlation mappings, and analysis code.

Capture X-MailHub-Event, X-MailHub-Delivery, and X-MailHub-Timestamp. Retain the signature material needed for secure verification; Mailera lists X-MailHub-Signature and X-Mailera-Signature as signature headers. Do not treat a header timestamp as mailbox arrival merely because it is available.

Mailbox metadata includes protocol, capabilities.live, and fields under sync, including state, level, and sleeping_since. Save relevant setup observations, but keep observation activity out of the idle cohort's baseline wherever possible. Record every intervening API or UI action.

How should retries and missing events affect p95?

Use one latency observation per test message and event type, based on the first valid endpoint receipt. Keep later deliveries in a separate delivery-attempt record. Counting duplicate requests as additional messages changes the distribution you intended to measure.

Capture X-MailHub-Delivery for diagnosis, but establish its behavior across retries before using it as your sole deduplication key. Keep the test-message correlation mapping independent of that assumption.

At the predefined cutoff, report these quantities together:

Report delivery within the window both against attempted messages and, where appropriate, confirmed arrivals. Keep messages without an arrival observation separate from messages that arrived but had no matching webhook.

Do not turn missing observations into zero latency or silently discard them. Preserve late arrivals after cutoff in a separate report without rewriting the original window's outcome.

Publish a distribution or empirical cumulative distribution alongside the table. Attach run dates, sample counts, uncertainty bounds, and redacted dataset references. Use null with “not measured” for unavailable measurements, never zero.

What can explain delayed notifications?

Mailera receives new Microsoft mailbox messages through Microsoft Graph change notifications, with periodic background checks as fallback. The Graph change notification overview, read on Oct 25, 2026, describes the upstream notification mechanism.

Graph-to-Mailera and Mailera-to-your-collector are different delivery legs. Microsoft's webhook delivery guidance, read on the same date, considers a Graph notification delivered when its receiver returns a 2xx response within three seconds. That is an upstream acknowledgment rule, not a three-second Mailera arrival-to-webhook promise.

For IMAP cohorts, record the connection mode actually in use. RFC 2177, read on Oct 25, 2026, permits server updates while IDLE is active and requires polling when the server does not advertise IDLE capability. Provider support alone does not identify the mode used during your run. The IMAP, Graph, and SMTP guide explains the protocol distinctions.

Mailera checks actively used mailboxes more frequently than idle ones. Inactive mailboxes can sleep and wake on use. Keep sleeping-mailbox observations separate; POST /mailboxes/{id}/sync both triggers synchronization and wakes sleeping mailboxes. Use a separate wake-up cohort for that action, following the sync recovery workflow.

Only attribute delay to provider visibility, Mailera detection, dispatch, or receiver networking when correlated logs locate the delay. Otherwise, report the measured interval without assigning a cause.

Which mistakes make the benchmark misleading?

The most consequential mistakes change the measurement boundary or the population:

• Starting at send time but labeling the result mailbox-arrival latency.
• Pooling imports, live arrivals, Junk events, and forced-sync runs.
• Measuring detection-to-dispatch time and calling it end-to-end delivery.
• Dropping timeouts or counting retries as new messages.
• Subtracting timestamps from clocks without recorded offset bounds.
• Treating receipt, acknowledgment, and completed processing as one milestone.
• Interacting with idle mailboxes throughout the run without recording those actions.

For processing deadlines, budget webhook delivery and downstream work separately. Choose a deadline against both the observed latency distribution and the delivery-within-window rate. A favorable p95 alone does not describe missing events. The reliable email API workflow guide covers the surrounding application workflow.

How do you keep the test safe?

Use only mailboxes you own or are authorized to test. Keep API keys, signing secrets, and message content out of shared logs. Export synthetic correlation identifiers instead of addresses or subjects containing personal information.

If an AI agent consumes the events, treat incoming email as untrusted data, not instructions. Give it only the permissions it needs, and require confirmation before sending. Keep agent processing after the collector's receipt timestamp so it does not alter the webhook metric.

What does Mailera not guarantee?

Mailera does not guarantee inbox placement or bypass provider spam filters. Provider filtering and the sender's configuration remain outside this benchmark's control.

Treat measurements as observations under specified conditions, not a delivery SLA. Limit comparisons to the tested account types, connection modes, mailbox states, regions, and dates. A small controlled cohort is not a basis for ranking untested providers or promising the same tail latency under production load.