Order at the Endpoint, Not the Delivery

This is the sixth gap, and it is the one you own.

What Meshes guarantees — and what it doesn't

Meshes is explicit about the line. It delivers your product events with at-least-once semantics and automatic retries using exponential backoff with jitter, and it gives you the machinery to make that reliable:

• Idempotency keys with a configurable TTL, so a duplicate delivery is dropped instead of double-processed.
• Dead letter queues for events that exhaust their retries, so a failure is recoverable instead of silently lost.
• Per-attempt outcome codes — delivered, rate_limited, auth_failed, validation_failed, destination_not_found — so you can see exactly what happened to every delivery.

What it does not promise is ordering. Events may arrive and be processed out of order, especially when retries are involved. That is stated up front because the alternative — a system that quietly implies ordering it cannot deliver — is how the subscription bug at the top of this post happens.

Two clarifications worth making, because they get conflated:

• Idempotency is not ordering. Deduplicating deliveries stops you from applying the same event twice. It says nothing about the order of different events. You need both, and they are separate mechanisms.
• Emitter-side keys are not receiver-side state. Designing idempotency keys for outbound delivery is about the sending side surviving retries and replays. Ordering is a property of how the receiving side stores what it gets.

Order the data, not the delivery

Here is the shift that makes the problem tractable: you are not trying to reconstruct the order the events were sent in. That order is gone, and chasing it leads to fragile buffering logic. You are trying to make your stored state correct regardless of arrival order.

That means putting an order signal in the event and letting your write logic decide the winner — instead of trusting whoever showed up last.

1. Carry a version or sequence number on every event. A monotonic version per entity (or an emitter timestamp, if you control the clock) is all you need. Do not rely on wall-clock arrival time; across distributed producers it lies.

2. Guard every write by that version. Apply an event only if it is newer than what you already have. A stale arrival becomes a no-op instead of a regression:

When the late pro event (version 6) lands after starter (version 7), the where excluded.version > subscriptions.version guard rejects it. The customer stays on starter. The bug never happens — and you never had to order a single delivery.

3. Make the apply idempotent. Key the upsert on the entity, not on a fresh row per delivery. At-least-once means you will see the same event again; applying it twice must land in the same place it landed the first time. Combine this with the dedupe Meshes does on the idempotency key and double-processing stops being a concern.

4. Reach for a reorder buffer only when sequence is truly required. Version-guarded last-write-wins is enough when you only care about the latest state. If you genuinely need to process every transition in order — a financial ledger, an audit trail — hold events in a short per-entity buffer keyed by sequence number and release them in order, with a timeout that drops to a dead letter queue if a gap never fills. This is more machinery; most workloads do not need it.

When you don't need ordering at all

Before you build any of this, check whether you need it. A large share of event-driven work is naturally order-insensitive:

• Append-only events — logging a payment.succeeded, sending a notification, firing analytics. Order rarely changes the outcome.
• Commutative updates — incrementing a counter, adding a tag, appending to a list.
• Events about independent entities — two different customers' updates have no ordering relationship to enforce in the first place.

Ordering only matters when two events touch the same entity and the later one must win. That is exactly the case the version guard handles in a few lines of SQL. For everything else, at-least-once delivery with idempotent handling is the whole job.

Out-of-order delivery is not the part of distributed systems you defend against. It is the part you design around — and the design lives at the endpoint, not in the pipe.

Stop fighting delivery order and start designing for it. Join Meshes and get at-least-once fan-out, idempotency keys, dead letter queues, and per-attempt delivery visibility for your product events out of the box.