---
title: "Email webhook latency benchmark: a measurement protocol"
description: "Measure email arrival-to-webhook latency with clear clock boundaries, provider cohorts, p95 reporting, and explicit handling of retries and missing events."
url: https://mailera.app/en/blog/email-webhook-latency-benchmark-protocol
published: 2026-10-25
updated: 2026-10-25
---

# Email webhook latency benchmark: a measurement protocol

Measure mailbox arrival to endpoint receipt without confusing it with send time or application processing. Use this protocol to compare provider configurations, mailbox states, and delivery tails.

## What exactly should an email webhook latency benchmark measure?

Stop the clock when your receiving endpoint obtains the webhook, not when your application finishes handling the email. Start it at mailbox arrival only when you have an independent timestamp with suitable semantics and precision.

An incoming email webhook is an HTTP notification associated with an email event. [Mailera connects existing mailboxes](https://mailera.app/en/product) to applications through webhooks and other interfaces; your benchmark needs to distinguish the mailbox side from the receiving application.

Use these names in your measurement records. They are benchmark variables, not Mailera payload fields.

| Variable | Boundary | Required record |
| --- | --- | --- |
| `t_arrival` | Message arrives at the mailbox, under a documented definition | Timestamp source, semantics, resolution, clock uncertainty |
| `t_detected` | Mailera detects the message | Instrumented detection timestamp, when available |
| `t_endpoint` | Collector has received the complete webhook request body | Collector timestamp and clock uncertainty |
| `t_ack` | Collector sends its HTTP acknowledgment | Response time and status |
| `t_done` | Application completes downstream work | Application completion timestamp |

Define arrival-to-webhook latency as `t_endpoint − t_arrival`. Report detection-to-endpoint latency separately as `t_endpoint − t_detected` when instrumentation supports it.

If you only observe a message through a mailbox read, name that starting point `t_first_observed`. Report **first-observed-to-webhook latency**, including negative values if the webhook reaches your collector before the observer sees the message. Do not relabel that interval as arrival latency.

![Timeline separating mailbox arrival, Mailera detection, dispatch, collector receipt, acknowledgment, and application completion.](https://mailera.app/media/8d03f6f775caf210/email-webhook-latency-benchmark-a-measurement-protocol-en-1200.jpg)
_Stop the webhook latency clock at collector receipt. Record clock uncertainty at each timestamped boundary._

## Which timestamp can represent mailbox arrival?

Choose the timestamp source before collecting samples. Do not substitute SMTP submission time, the sender's `Date` header, or a Mailera observation timestamp for an independently established mailbox-arrival boundary.

Google provides a useful example of why semantics matter. Its [Gmail message resource documentation](https://developers.google.com/workspace/gmail/api/reference/rest/v1/users.messages), read on Oct 25, 2026, defines `internalDate` in epoch milliseconds. For normally SMTP-received messages, it represents when Google originally accepted the message. For API-migrated mail, the client can configure it to derive from the `Date` header.

That makes Google acceptance a possible named measurement boundary for an appropriate SMTP test cohort, not a universal definition of mailbox visibility. Mailera's Gmail connector uses IMAP with app-password authentication; do not assume a Gmail API field is the timestamp used by that connector.

For every provider, document the chosen arrival signal and its precision. Treat IMAP `INTERNALDATE` and provider-specific timestamps as candidates requiring semantic checks, rather than interchangeable clocks. A field named `received_at` is also insufficient by itself to define your benchmark boundary.

Synchronize collector and observer clocks, then record offset bounds. Timestamp resolution is not the same as accuracy. Preserve that uncertainty in the results instead of ranking configurations whose apparent differences fall inside it.

## How should you build comparable test cohorts?

Compare configurations, not provider names alone. Outlook.com and Microsoft 365 should have separate account-type labels, even when both use the same Mailera connection path.

Publish a protocol before the run:

1. **Choose owned test mailboxes.** Record provider, account type, account count, connection mode, and run dates. The [Microsoft connection guide](https://mailera.app/en/providers/microsoft) and [custom IMAP guide](https://mailera.app/en/providers/imap) cover different connection paths; keep those distinctions in the dataset.
2. **Record the environment.** Include collector region, software versions, clock synchronization details, and any deployment metadata available to you. Record how each timestamp is obtained.
3. **Define mailbox states operationally.** Write down how you establish active, idle, and sleeping cohorts. Record state observations and any interaction that could change them.
4. **Control the messages.** Keep sender, content size, pacing, and observation windows comparable. Give each test message a unique synthetic correlation marker and check that you can match it through the complete path during setup.
5. **Interleave the runs.** Alternate or randomize provider cohorts rather than testing each provider in a different time period.
6. **Fix the analysis rules.** Specify sample size, cutoff, exclusions, duplicate handling, percentile method, and uncertainty reporting before collecting the comparison dataset.

Separate historical imports from new test arrivals. Keep Inbox and Junk cohorts distinct, and record the observed event type and folder rather than pooling them by convenience. Mailera's event catalog includes both `message.received` and `message.junk`.

![Mailera mailbox list showing provider, status, and last-sync context for redacted test mailboxes.](https://mailera.app/media/71210558454827bd/email-webhook-latency-benchmark-a-measurement-protocol-test-en-1200.jpg)
_Record cohort configuration separately from timing observations; the mailbox list is setup context._

## How do you set up the Mailera side?

Use a dedicated mailbox group and collector for the test. The [Mailera developer overview](https://mailera.app/en/developers) describes the integration surface; use the [API documentation](https://mailera.app/en/docs) for webhook creation and signature verification details.

1. In **Mailboxes → Add mailbox**, connect your owned test accounts. For Microsoft, choose **Sign in with Microsoft**. Record the connection test's **Live monitoring** result where shown, and let initial message loading finish before the live-arrival cohort begins.
2. In **Settings**, open the API key tab and choose **Create key**. Set **Mailbox scope** to the test group. Use **Webhook** permission for webhook management, adding **Read** only when your collector workflow needs mailbox or message reads. Set an **Expiration** and optional IP restrictions.
3. Prepare an HTTPS collector that records receipt before application processing. Keep acknowledgment status and timing as separate observations.
4. Create the subscription with `POST /webhooks` using the documented request schema. The API base is `https://mailera.app/api/public/v1`; authentication uses `Authorization: Bearer $MAILERA_API_KEY`. Store the signing secret securely when creation returns it: it is returned only once.
5. Validate request signatures using the documented procedure. Separate rejected requests from accepted benchmark observations rather than letting unauthenticated traffic affect your results.
6. Run the controlled message schedule, then export redacted collector records, correlation mappings, and analysis code.

Capture `X-MailHub-Event`, `X-MailHub-Delivery`, and `X-MailHub-Timestamp`. Retain the signature material needed for secure verification; Mailera lists `X-MailHub-Signature` and `X-Mailera-Signature` as signature headers. Do not treat a header timestamp as mailbox arrival merely because it is available.

Mailbox metadata includes `protocol`, `capabilities.live`, and fields under `sync`, including `state`, `level`, and `sleeping_since`. Save relevant setup observations, but keep observation activity out of the idle cohort's baseline wherever possible. Record every intervening API or UI action.

![API key configuration limited to a test mailbox group with Webhook and Read permissions.](https://mailera.app/media/c13a9d3742894d2e/email-webhook-latency-benchmark-a-measurement-protocol-en-1200.jpg)
_Scope the benchmark key to the test group and enable only the permissions the workflow needs._

## How should retries and missing events affect p95?

Use one latency observation per test message and event type, based on the first valid endpoint receipt. Keep later deliveries in a separate delivery-attempt record. Counting duplicate requests as additional messages changes the distribution you intended to measure.

Capture `X-MailHub-Delivery` for diagnosis, but establish its behavior across retries before using it as your sole deduplication key. Keep the test-message correlation mapping independent of that assumption.

At the predefined cutoff, report these quantities together:

| Output | Reporting rule |
| --- | --- |
| Attempted messages | Count every scheduled test message actually submitted |
| Confirmed arrivals | Count messages with the required independent arrival observation |
| Observed unique events | Count correlated, accepted first receipts within the window |
| Missing at cutoff | Retain confirmed arrivals without a matching receipt |
| p50 and p95 | State that these describe the observed cohort and name the calculation method |
| p99 | Include only with enough observations for a meaningful tail estimate |
| Delivery within window | Name the denominator explicitly |

Report delivery within the window both against attempted messages and, where appropriate, confirmed arrivals. Keep messages without an arrival observation separate from messages that arrived but had no matching webhook.

Do not turn missing observations into zero latency or silently discard them. Preserve late arrivals after cutoff in a separate report without rewriting the original window's outcome.

Publish a distribution or empirical cumulative distribution alongside the table. Attach run dates, sample counts, uncertainty bounds, and redacted dataset references. Use `null` with “not measured” for unavailable measurements, never zero.

## What can explain delayed notifications?

Mailera receives new Microsoft mailbox messages through Microsoft Graph change notifications, with periodic background checks as fallback. The [Graph change notification overview](https://learn.microsoft.com/en-us/graph/change-notifications-overview), read on Oct 25, 2026, describes the upstream notification mechanism.

Graph-to-Mailera and Mailera-to-your-collector are different delivery legs. Microsoft's [webhook delivery guidance](https://learn.microsoft.com/en-us/graph/change-notifications-delivery-webhooks), read on the same date, considers a Graph notification delivered when its receiver returns a 2xx response within three seconds. That is an upstream acknowledgment rule, not a three-second Mailera arrival-to-webhook promise.

For IMAP cohorts, record the connection mode actually in use. [RFC 2177](https://www.rfc-editor.org/rfc/rfc2177.html), read on Oct 25, 2026, permits server updates while IDLE is active and requires polling when the server does not advertise IDLE capability. Provider support alone does not identify the mode used during your run. The [IMAP, Graph, and SMTP guide](https://mailera.app/en/blog/email-protocols-for-ai-agents) explains the protocol distinctions.

Mailera checks actively used mailboxes more frequently than idle ones. Inactive mailboxes can sleep and wake on use. Keep sleeping-mailbox observations separate; `POST /mailboxes/{id}/sync` both triggers synchronization and wakes sleeping mailboxes. Use a separate wake-up cohort for that action, following the [sync recovery workflow](https://mailera.app/en/blog/mailbox-sync-troubleshooting).

Only attribute delay to provider visibility, Mailera detection, dispatch, or receiver networking when correlated logs locate the delay. Otherwise, report the measured interval without assigning a cause.

## Which mistakes make the benchmark misleading?

The most consequential mistakes change the measurement boundary or the population:

- Starting at send time but labeling the result mailbox-arrival latency.
- Pooling imports, live arrivals, Junk events, and forced-sync runs.
- Measuring detection-to-dispatch time and calling it end-to-end delivery.
- Dropping timeouts or counting retries as new messages.
- Subtracting timestamps from clocks without recorded offset bounds.
- Treating receipt, acknowledgment, and completed processing as one milestone.
- Interacting with idle mailboxes throughout the run without recording those actions.

For processing deadlines, budget webhook delivery and downstream work separately. Choose a deadline against both the observed latency distribution and the delivery-within-window rate. A favorable p95 alone does not describe missing events. The [reliable email API workflow guide](https://mailera.app/en/blog/reliable-email-api-agent-workflows) covers the surrounding application workflow.

## How do you keep the test safe?

Use only mailboxes you own or are authorized to test. Keep API keys, signing secrets, and message content out of shared logs. Export synthetic correlation identifiers instead of addresses or subjects containing personal information.

If an AI agent consumes the events, treat incoming email as untrusted data, not instructions. Give it only the permissions it needs, and require confirmation before sending. Keep agent processing after the collector's receipt timestamp so it does not alter the webhook metric.

## What does Mailera not guarantee?

Mailera does not guarantee inbox placement or bypass provider spam filters. Provider filtering and the sender's configuration remain outside this benchmark's control.

Treat measurements as observations under specified conditions, not a delivery SLA. Limit comparisons to the tested account types, connection modes, mailbox states, regions, and dates. A small controlled cohort is not a basis for ranking untested providers or promising the same tail latency under production load.

**Do this with the API** — Every action in this article is available via the REST API, with curl examples, API keys and webhooks. [View API docs](https://mailera.app/en/docs)