Idempotent Event Delivery - Why Your Webhooks Process Duplicates (And How to Stop)
Duplicate webhook deliveries are normal in at-least-once systems. Learn idempotency keys, dedup strategies, and Node.js patterns that prevent double-processing.
If you send webhooks or product events long enough, you will deliver duplicates.
Duplicates are a normal consequence of building on networks, queues, timeouts, retries, and independent systems that do not share one transaction boundary. If your event producer retries aggressively enough to be reliable, your consumer will eventually see the same logical event more than once.
That is why idempotency matters. An idempotent consumer can process the same event twice and still produce one correct result.
This is the practical answer to duplicate delivery. Not magical "exactly once" marketing language. Not hoping your queue or webhook sender never retries. Just a system designed so duplicates are harmless.
Why duplicates happen in the first place
Most delivery systems operate with at-least-once semantics. The producer keeps trying until it sees evidence that the consumer accepted the message.
That sounds reasonable, but there is a gap between "the consumer processed the event" and "the producer knows it processed the event."
Here are a few common ways duplicates appear:
The consumer completes the work, but its HTTP response times out before the producer receives it.
The producer receives a 500 or connection reset after the consumer already committed the write.
A worker crashes after processing the event but before checkpointing success.
A queue redelivers the message because the acknowledgement was lost.
A retry engine intentionally resends after transient failures. That is exactly what a good retry system should do. We cover the retry side in Webhook Retry Logic Done Right.
In every case, the producer does not have enough information to safely assume "that event definitely did not apply." So it retries.
That is correct. The consumer still has to handle repeat delivery.
What an idempotency key actually is
An idempotency key is a stable identifier for the logical operation you want to happen once.
For webhooks and event delivery, that identifier usually comes from one of three places:
A provider event ID, such as evt_123 or a UUID in the payload
A delivery ID added by the event infrastructure
A deterministic key you derive from your own domain object, such as workspace_id + order_id + event_type
The important property is not where the key came from. The important property is that retries of the same logical event produce the same key.
That lets the consumer ask a simple question:
Have I already applied the side effects for this key?
If yes, return success.
If no, apply the work exactly once from the consumer's point of view, then record that the key has been processed.
Common failure modes
Teams usually understand the idea of idempotency before they get the implementation right. The bugs show up in the gap between the slogan and the storage strategy.
1. Treating payload equality as deduplication
Two deliveries with the same JSON body are not always the same logical event. Field ordering can differ. Timestamps can differ. Retries may include new transport metadata. Hashing the entire payload often creates false negatives and false positives.
Prefer a stable event or operation ID over payload comparison.
2. Using only an in-memory cache
An in-memory Set or process-local LRU cache looks fine in development. It falls apart in production:
a deploy clears it
a crash clears it
a second replica never had it
a retry that lands an hour later is outside the process lifetime
In-memory dedup can be a performance optimization. It is not your source of truth.
3. Recording the key outside the business transaction
This is one of the easiest ways to get subtle duplicates.
If you insert the dedup key after writing the business row, two concurrent workers can both reach the write before either one marks the event as processed. If you insert the dedup key before the business work, then fail midway, you may permanently mark an event as handled even though the side effect never completed.
The durable dedup record and the business write need to succeed or fail together.
4. Confusing time-windowed dedup with durable idempotency
A Redis TTL window can stop rapid duplicate storms. It does not protect you against a replay tomorrow, a backfill next week, or a queue redelivery after a long outage.
For low-risk notifications, a time window may be enough. For money movement, CRM updates, account state, or anything customer-visible, durable dedup is usually the safer choice.
5. Believing exactly-once claims too literally
Some brokers can provide exactly-once guarantees within a narrow boundary, such as a producer writing to a specific log with a specific transaction model. That is not the same thing as end-to-end exactly-once side effects across HTTP, your database, another queue, and a third-party API.
Once the event crosses multiple systems, you are back in the real world: partial failures, retries, and duplicated attempts. The practical pattern is still at-least-once delivery plus idempotent consumers.
Practical patterns that work in production
There is no single dedup pattern for every workload. The right choice depends on how expensive duplicates are and how long duplicate risk persists.
Pattern
Best for
Strength
Weakness
Redis or cache TTL window
High-volume, low-risk duplicate bursts
Fast and cheap
Not durable; misses late replays
Database dedup table
Business events with real side effects
Durable and easy to reason about
Adds a write path and table growth
Unique constraint on business object
Natural upsert cases like customer_id + external_id
Simple when the domain has a natural key
Not every side effect maps cleanly
Infrastructure delivery IDs
Standardized producer-side event identity
Keeps IDs consistent across destinations
Still requires consumer-side idempotent handling
Pattern 1: database-backed dedup with a unique constraint
For most webhook consumers, the most dependable pattern is a durable table with a unique key, wrapped in the same transaction as the business update.
Example schema:
create table processed_webhook_events ( source text not null, idempotency_key text not null, processed_at timestamptz not null default now(), primary key (source, idempotency_key));
And a realistic Node.js handler using Postgres:
import { Pool } from 'pg';const pool = new Pool({ connectionString: process.env.DATABASE_URL });type BillingWebhook = { id: string; customerId: string; invoiceId: string; amount: number;};export async function handleInvoicePaid(event: BillingWebhook) { const client = await pool.connect(); try { await client.query('BEGIN'); const dedupResult = await client.query( ` insert into processed_webhook_events (source, idempotency_key) values ($1, $2) on conflict do nothing `, ['billing-provider', event.id], ); if (dedupResult.rowCount === 0) { await client.query('ROLLBACK'); return { status: 200, duplicate: true }; } await client.query( ` insert into invoices (provider_invoice_id, customer_id, amount, status) values ($1, $2, $3, 'paid') on conflict (provider_invoice_id) do update set amount = excluded.amount, status = 'paid' `, [event.invoiceId, event.customerId, event.amount], ); await client.query('COMMIT'); return { status: 200, duplicate: false }; } catch (error) { await client.query('ROLLBACK'); throw error; } finally { client.release(); }}
A few things matter here:
The dedup insert and the invoice write are in the same transaction.
If the transaction rolls back, the dedup record rolls back too.
A second delivery of the same event sees the unique key conflict and exits cleanly.
The domain write itself is also idempotent because it uses an upsert keyed on the provider invoice ID.
That last point matters. The strongest handlers usually combine transport-level dedup with domain-level idempotency.
Verifying webhook idempotency in Node.js and Postgres
Treat the handler as unsafe until tests prove duplicate delivery cannot create duplicate side effects.
A practical verification checklist:
Send the same webhook payload twice and assert the second request returns success without writing a second business record.
Send two concurrent requests with the same idempotency key and assert only one transaction commits the business change.
Force an error after the dedup insert but before the business write, then assert the transaction rolls back both records.
Replay the same event after the retry window has expired and assert durable Postgres state still blocks duplicate processing.
Confirm the key comes from a stable event ID, delivery ID, or domain operation ID rather than a hash of the full payload.
The Postgres invariant should be boring and explicit:
alter table processed_webhook_events add constraint processed_webhook_events_once unique (source, idempotency_key);
The Node.js invariant is just as important: the dedup insert and the business write must share the same transaction. If they do not, a test should fail under concurrent delivery.
Pattern 2: Redis for short duplicate windows
If your main problem is rapid-fire duplicate attempts during retries, a cache-based window can help:
high-volume events where a one-hour window is good enough
It is less useful for:
financial events
systems that replay history
queues that can redeliver far outside the TTL
Redis is often a good first layer. It is rarely the only layer you want for important state changes.
Pattern 3: natural idempotency in the domain model
Sometimes the cleanest solution is not a separate dedup table. It is a unique constraint on the business object itself.
If every external order maps to one internal order, then external_order_id can be unique. If every upstream invoice maps to one invoice row, make that ID unique and use upserts.
This is especially effective when the side effect is "materialize or update a record" rather than "fire an external side effect."
Pattern 4: stable event IDs from delivery infrastructure
Good delivery infrastructure can help by attaching stable identifiers, normalizing retries, and giving you clearer observability around which logical event is being retried.
That reduces ambiguity, but it does not remove the need for consumer-side idempotency. A stable delivery ID is only useful if your consumer actually checks it before doing work.
At-least-once vs exactly-once: what is actually true
This is where a lot of teams get misled.
If someone says their webhook system is "exactly once," ask what boundary they mean.
Inside one broker? Maybe.
Across a queue acknowledgement, an HTTP request, your database commit, and a downstream API call? Usually no.
The producer can know it attempted the delivery. The consumer can know it committed a write. Neither side can retroactively make the network perfect.
That is why at-least-once delivery remains the standard design for reliable integration systems:
retry when the outcome is uncertain
include a stable event identifier
require the consumer to be idempotent
This is not a compromise. It is the pattern that survives contact with real distributed systems.
You can absolutely implement idempotent handling yourself, and you should. This is consumer logic, not something a platform can magically do on your behalf.
Where infrastructure becomes useful is around the edges of the problem:
generating stable delivery IDs
standardizing retry behavior
isolating failed destinations
surfacing event and attempt history
storing dead letters and replay context
That is where a delivery layer or integration platform starts earning its keep. Meshes fits that category: it can centralize routing, retries, and delivery observability so you are not rebuilding the same transport plumbing for every destination. But the consumer still needs to treat duplicate delivery as normal and handle the idempotency key correctly.
If you are deciding whether to keep hand-rolling that delivery layer, our DIY comparison is the more relevant place to evaluate the build-vs-buy tradeoff.
The practical takeaway
Duplicate webhook delivery is not a bug to eliminate. It is a condition to design for.
The winning pattern is boring in the best possible way:
accept at-least-once delivery as normal
attach or consume a stable idempotency key
make the business write and the dedup record succeed together
return success on duplicates instead of reprocessing them
Do that consistently and duplicates stop being scary. They become just another retry outcome your system already knows how to absorb.
Want reliable delivery without rebuilding the transport layer?Join Meshes and handle retries, routing, and observability in one place while your consumers stay idempotent.
