What an eventual consistency integration actually promises

An eventual consistency integration makes a narrow promise: given no further changes, two systems will arrive at the same value within a bounded period. It does not promise that a customer viewing a product page is looking at the same number the picker sees on a handheld. It never has, and any design that implies otherwise will be embarrassed by the first bank holiday flash sale. The design work is not in eliminating the gap. It is in bounding it, measuring it, and deciding in advance what the business does during the window in which the two disagree.

In practice the warehouse is the authority for physical stock and the storefront is the authority for demand. A Peoplevox or Mintsoft installation knows what is on the shelf, what is allocated, what is in a pick face awaiting a container, and what has been counted but not yet posted. Shopify knows what has been added to a basket, what has been paid for, and what its own inventory service believes is available at a given location. Neither has the full picture, and neither can be made to. The integration is the only component that sees both, which makes it the only place where convergence can be defined and enforced.

The failure mode we are most often called in to fix is not a missing message. It is a quiet, slow divergence: a single SKU that has been three units out since February, discovered when a customer orders the last one and the warehouse has none. Nothing errored. No dead letter queue filled up. The messages were all delivered, in the wrong order, and the last one to arrive won. An eventual consistency integration that has no way of noticing this is not eventually consistent. It is just asynchronous, which is a different and much weaker property.

So the first deliverable of the discovery stage is not a field map. It is a written answer to three questions: what value is being converged, from which side, and within how long. Everything downstream — transport, retry policy, reconciliation schedule, alert thresholds — falls out of those answers. Where a client cannot answer them, the integration has no acceptance criteria, and go-live becomes a matter of opinion.

Deltas, snapshots, and the ordering problem that causes drift

There are two ways to move a stock figure across a boundary, and they fail differently. A delta says "reduce available by two". A snapshot says "available is now seventeen as at 14:02:11". Deltas are commutative and survive out-of-order delivery, but they are catastrophic under duplication: apply the same minus-two twice and the storefront is permanently wrong with no way to detect it internally. Snapshots are idempotent — applying the same one twice is harmless — but they are order-sensitive. An older snapshot arriving after a newer one overwrites good data with stale data, and the system will sit happily on the wrong number until something forces a re-read.

Shopify offers both shapes. The delta-style adjustment is the correct choice for genuine event-driven movements such as a despatch confirmation, provided every message carries a durable idempotency key derived from the warehouse's own transaction identifier rather than from a timestamp or a generated GUID. The absolute set operation is the correct choice for reconciliation and for stock counts, because a count is by definition a statement about a moment rather than a change. Mixing them without rules is how a well-built pipeline develops a slow leak.

Ordering is where most drift originates, and it is rarely the fault of the transport. Warehouse systems batch. A Peoplevox despatch file may cover a two-hour window and land after a manual adjustment made ten minutes ago. Two workers adjusting the same bin produce two events with clock readings from two devices. Retries re-present an older payload after a newer one has already been accepted. The only reliable defence is a monotonic sequence supplied by the source system — a transaction ID, a revision number, a database change token — carried on every message and stored against the target record.

With that sequence in place, the rule is trivial to state and should be stated in the specification in these words: apply a snapshot only if its source sequence is greater than the last applied sequence for that SKU and location; otherwise discard it and record the discard. Discards are not errors, and should not page anyone, but they must be counted. A sudden rise in discards is the earliest available signal that an upstream batch job has started replaying history.

  • Delta messages — Use for real movements — despatch, receipt, adjustment. Require a source-derived idempotency key and a deduplication window measured in days, not minutes.
  • Snapshot messages — Use for counts, reconciliation and recovery. Require a source sequence number and reject anything older than the last applied value.
  • Never mixed silently — If both shapes are in play, record which one last wrote each SKU. Diagnosing drift without that field is guesswork.

Setting a convergence budget rather than chasing real time

A convergence budget is the maximum time a difference between the warehouse and the storefront may persist before it is treated as a defect. It is not one number. Fast-moving SKUs with thin cover might carry a budget of two minutes; a slow-moving spare part with four months of stock can comfortably sit on thirty. Writing this down converts an unbounded engineering aspiration into a testable requirement, and it gives operations a defensible answer when a colleague asks why the site sold one more than existed.

The budget should be derived from commercial exposure, not from what the API happens to allow. The calculation is straightforward: expected orders per minute for a SKU, multiplied by the budget in minutes, gives the number of units that could be oversold during a worst-case lag. If that number is unacceptable, either the budget shrinks or the safety buffer absorbs it. Doing this per velocity band takes an afternoon during discovery and prevents an entire category of argument later, because the oversell risk has been priced rather than denied.

Measuring against the budget requires an observation that most pipelines do not take. Record, for each applied change, the timestamp the source system assigned and the timestamp the target acknowledged. The difference is your applied lag. Publish the ninety-fifth percentile per hour. When a Mintsoft batch runs long or Shopify throttles a burst of adjustments, the percentile moves before anything fails, and that gives a team time to act rather than to explain. Lag that is rising steadily with a flat error count almost always means queue backlog, not breakage.

It is worth being explicit about what the budget does not cover. Time spent in a warehouse's own internal processing before it emits an event is outside the integration's control and must be measured separately, usually by comparing the physical pick timestamp with the event timestamp. We have seen projects where the middleware delivered in four seconds and the warehouse took eleven minutes to publish. Blaming the integration for that gap wastes a week, and the fix lies entirely on the other side of the boundary.

The reconciliation sweep, and why it must be safe to run at any time

Event streams lose things. Not often, but often enough that a system with no independent check will eventually be wrong without knowing it. The remedy is a reconciliation sweep: a periodic comparison of the full stock position on both sides, producing a list of differences, with a separate and deliberate decision about which differences are corrected automatically. The sweep is not a backup for a broken pipeline. It is the mechanism that converts eventual consistency from a claim into something observed.

Build it to compare, not to overwrite. The first version should read the warehouse position and the storefront position, exclude records changed within the convergence budget window — those are legitimately in flight and comparing them produces noise — and write the remainder to a differences table with both values, both sequence numbers and the time of comparison. Only once the difference volume is understood should automatic correction be enabled, and then only for differences above a configured age and below a configured magnitude. A SKU that is out by four hundred units is not a sync problem; it is a stocktake or a data entry error, and pushing it to the storefront automatically makes a bad situation live.

Run the sweep when the warehouse is quiet if you can, but design it to be safe at any hour, because the day will come when it must be run mid-afternoon after an incident. That means correction writes go through the same idempotent path as everything else, carry a sequence number that will not be overtaken by an in-flight message, and are tagged with a reason code so that the audit trail shows plainly that a value was set by reconciliation rather than by a movement.

Size matters here. A catalogue of two thousand SKUs across one location can be swept in a single pass against Shopify without difficulty. Forty thousand SKUs across three locations cannot, and needs partitioning by location or by an alphabetical or velocity-based shard, with each shard on its own schedule. The design question to settle at scoping is how long a complete cycle takes, because that period is the true upper bound on how long an undetected difference can survive — and it is usually considerably longer than anyone assumed.

  • Compare-only first — Run for a fortnight producing differences without correcting them. The volume and shape of that list tells you more about the pipeline than any test plan.
  • Exclusion window — Ignore records touched inside the convergence budget. Without this, the differences report is dominated by items that were about to be right anyway.
  • Correction limits — Auto-correct small, old differences. Escalate large ones to a named person. Record every correction with a reason code.

What finance needs from the same pipeline

Operations cares about whether the site can sell an item. Finance cares about what the stock position was at a specific instant, usually month end, and whether that figure can be defended. These are different requirements from the same data, and a pipeline built only for the first will not satisfy the second. The essential addition is an as-at capability: the ability to state the stock position at a nominated time, rather than only the position now.

This is cheaper to build than it sounds, provided it is decided before the schema is fixed. Retain the applied change log — SKU, location, delta or new value, source sequence, source timestamp, applied timestamp, reason code — for a retention period agreed with the client's accountant, typically covering the current and prior financial year. From that log, the position at any past moment is a replay, and any difference between the storefront view and the warehouse view at month end can be explained line by line rather than asserted. Where stock valuation flows onward into an accounting package, that log is what makes the movement between two closing positions auditable.

The cut-off convention also needs writing down. If a despatch is confirmed at 23:58 on the last day of the month and the corresponding event is applied at 00:04, does it fall in the closing period or the opening one? The answer should be based on the source timestamp, consistently, and stated in the handover documentation. It is a small decision that becomes an expensive one when it is discovered during an audit that the two systems answered it differently.

Finally, agree who reads the differences report. An eventual consistency integration generates a small, steady trickle of divergences that require human judgement — a damaged unit written off in the warehouse but still allocated to an order, a returned item received but not yet inspected. That trickle is not a defect and cannot be engineered away, but it does need an owner in the operations team and a weekly slot in which it is cleared. Handing over a report that nobody has been asked to read is how good integrations quietly stop being trusted.

Key points

  • Define a convergence budget per velocity band before writing code; without a stated maximum divergence time there are no acceptance criteria for the integration.
  • Use deltas with idempotency keys for real movements and snapshots with source sequence numbers for counts and recovery, and never let the two mix without recording which last wrote each record.
  • A reconciliation sweep that compares before it corrects, plus a retained change log, is what turns eventual consistency from an assertion into something operations and finance can both verify.