Partial Failure in Fan-Out Systems: When Three Destinations Succeed and Two Fail

Why replaying the whole event is dangerous

If Slack fails, the simplest recovery code may appear to be “send user.signup again.” That creates five new deliveries, including three destinations that already succeeded.

The outcome depends on what those destinations do with duplicates:

• HubSpot might update the existing contact harmlessly.
• Mailchimp might treat the request as an upsert.
• Intercom might record the same custom event twice.
• A webhook consumer might send a second welcome email or increment a counter again.

You have replaced one known failure with several unknown side effects.

Idempotent event delivery limits that risk, but idempotency is not permission to retry everything. A destination may not support idempotency, the key may have expired, or the original request may have succeeded even though the response never reached you. The safest retry is still the narrowest retry: the failed delivery to the specific destination.

Preserve the parent event ID and payload. Create another attempt beneath the failed destination delivery. Do not create another parent event and do not reopen terminal deliveries that already succeeded.

Recovery starts with failure classification

Not every failure deserves another immediate request. A timeout and an invalid payload both produce an unsuccessful delivery, but only one is likely to recover without human action.

Retryable failures

Temporary failures may succeed later:

• network timeouts
• rate limits
• temporary destination unavailability
• transient server errors

These belong in an automatic webhook retry strategy with exponential backoff and jitter. The retry schedule belongs to the destination delivery, so Slack can wait without delaying or reopening HubSpot, Mailchimp, or Intercom.

Permanent failures

Some failures require a change before another attempt can help:

• rejected authentication
• an invalid or missing destination
• a payload that fails validation
• a field mapping that no longer matches the destination

Repeatedly sending the same invalid request creates noise and consumes capacity without changing the outcome. Keep the delivery failed, show the reason, and let the workspace owner correct the connection, mapping, or payload before retrying it.

Ambiguous failures

Timeouts deserve special care because the destination may have completed the action before the connection disappeared. From the sender's perspective, the outcome is unknown.

This is where a stable idempotency key matters most. Reuse the delivery's key across attempts so a destination that honors idempotency can recognize the retry as the same operation. When the destination does not provide that protection, prefer an operation with natural upsert semantics or a receiver that records processed event IDs.

The safe partial-failure workflow

A reliable fan-out recovery path follows a small set of rules.

1. Confirm every intended destination

Before sending production traffic, confirm which rules match the event and which destinations should receive it. In the event history, each matched rule should remain visible separately, including deliveries that are pending, retrying, completed, failed, or canceled.

That distinction tells you whether a destination was never selected or whether its matched delivery has not completed an attempt yet.

2. Give every destination an independent state machine

Each delivery should progress without waiting for its siblings. A practical status set distinguishes pending, processing, retrying, completed, failed, and canceled deliveries, with delivered recorded as the successful attempt outcome.

Independence means:

• a slow destination does not block fast ones;
• a retry affects only the failed path;
• canceling one delivery does not erase successful siblings;
• the parent event can summarize progress without becoming the only source of truth.

3. Preserve every attempt

Replacing the previous error with the latest one makes debugging nearly impossible. Keep the attempt number, timestamp, outcome, and safe response details for each try.

Attempt history answers whether the destination is recovering, flapping between errors, consistently rejecting one field, or failing only under load. It also gives support teams evidence without asking engineers to reproduce a production incident.

4. Keep retry identity stable

A retry is another attempt at the same delivery, not a new business event. Preserve the parent event ID, destination delivery ID, and idempotency identity across attempts.

Stable identity lets the destination or receiver recognize duplicate attempts. It also prevents analytics and operational tooling from counting one business event several times simply because delivery took more than one request.

5. Stop automatic retries when they stop helping

Retries are for failures that can recover with time. Once automatic retries end — or the failure is known to require a fix — keep the failed delivery visible for investigation. Correct the connection, mapping, payload, or destination before choosing the next action.

Dead-letter handling preserves the failed destination context after automatic retries no longer help. In Meshes, Retry now can retry an eligible failed or retrying rule without reopening successful siblings. Once that delivery has exhausted its allowed attempts, re-emitting the source creates a new event and evaluates the active rules again. Review the destination set and duplicate-safety of each action before re-emitting.

Partial success is usually better than all-or-nothing

It is natural to ask whether fan-out should behave like a database transaction: either every destination succeeds or none of them do.

Across independent third-party systems, that promise is usually unavailable. HubSpot cannot roll back a Mailchimp request. A customer's webhook cannot participate in the same transaction as Slack. Even if you attempted compensating actions, those actions could fail too.

All-or-nothing behavior would also make the fastest, healthiest destinations wait for the slowest one. One expired credential could prevent every other integration from receiving an event.

The more reliable goal is eventual consistency with visible exceptions:

• deliver to healthy destinations immediately;
• isolate failures by destination;
• retry failures that can recover;
• expose failures that require a fix;
• preserve a deliberate replay path;
• make downstream effects idempotent wherever possible.

Partial success is not a compromise you hide. It is a state you model explicitly and recover safely.

An operational checklist

Before shipping a fan-out flow, verify that the system can answer these questions:

• Which destinations were selected for this event?
• Which destinations have already succeeded?
• Which destination is retrying right now?
• What was the latest outcome for each attempt?
• Is the failure retryable, permanent, or ambiguous?
• Can an operator retry or cancel one destination without touching the others?
• Will a retry reuse the original identity and avoid duplicate side effects?
• Where does an exhausted destination delivery wait for recovery?
• Does the parent status accurately roll up the mixed destination states?

If those answers live only in logs — or if the only recovery button resends the source event to every destination — partial failure will eventually become duplicate data, a silent gap, or both.

Fan-out becomes reliable when success and failure are no longer properties of the event alone. They are properties of every delivery the event created.

Recover the failed path without replaying the successful ones. Join Meshes to route one product event to every customer-configured destination with per-destination status, retry controls, and delivery history.