Why partial failure integration is the normal case, not the edge case

Most order flows in a mid-sized retail or wholesale business cross at least three systems. An order is captured in Shopify, allocated and picked in Mintsoft, and posted as a sales invoice in Sage 200. Each of those hops is a separate network call to a separate vendor with a separate maintenance window. A partial failure integration problem arises the moment one of those hops succeeds and a later one does not, because the business now holds an order that is real in one system, half-real in another and absent from a third.

Teams tend to design for the happy path and then bolt on retries. That works for transient network faults and nothing else. The failures that cause genuine operational damage are the ones where the first two writes landed cleanly: stock has been allocated in the warehouse system and a consignment has been created with DPD, but the invoice post into the ledger was rejected because the customer account is on hold. Nothing has crashed. There is no error page for anyone to see. The order simply sits in an inconsistent state until a human notices, which is usually at month end.

It is worth being precise about the word partial. A partial failure is not a failure of the integration as a whole. It is a successful sequence that stopped in the middle, leaving durable side effects behind it. Some of those side effects are reversible at low cost, some are reversible at administrative cost, and some are not reversible at all. A courier label that has already been included in a manifest, a pick that has already been performed, and a posted invoice that has already been included in a VAT return all belong to different categories, and a design that treats them identically will be wrong for at least two of them.

The practical consequence is that you cannot promise atomicity across three vendors. There is no distributed transaction available to you across Shopify, a warehouse management system and an accounting ledger. What you can promise is that every order reaches a known, recorded state, that inconsistent states are visible within a defined window, and that each inconsistent state has an agreed resolution path with a named owner. That is the whole of the discipline.

Model the order as a state machine, not a script

The most common architectural mistake we encounter is a single procedure that fetches the order, transforms it, calls the warehouse, then calls the ledger, all inside one execution. If that process dies at any point, the only record of what happened is in the logs, and the logs are not a data structure you can resume from. The fix is unremarkable and durable: give every order an integration record in your own store, with an explicit state, and advance that state one transition at a time.

A workable state set for a Shopify to Mintsoft to Sage 200 flow might be: received, validated, sent_to_wms, wms_confirmed, dispatched, invoice_posted, complete, plus quarantined and cancelled. Each transition is driven by a worker that reads records in a given state, attempts exactly one outbound operation, and writes the new state with the remote identifier it received. The record stores the Shopify order ID, the Mintsoft order reference, the Sage 200 invoice number, a correlation ID that appears in every log line, and a timestamp for each transition. That table becomes the single place anyone looks when asking where a given order got to.

This structure gives you three things you do not otherwise have. First, resumption: a process that dies between steps leaves a record in a state that the next worker run will pick up. Second, observability in business language, because an operations manager can be shown a count of orders in sent_to_wms for more than two hours without needing to read a log aggregator. Third, a safe place to put the retry counter and the last error message, so that repeated failure becomes a visible property of a record rather than a line buried in an application log.

Write the state transitions so that the remote call and the local state update cannot diverge. In practice that means recording the intent to call before calling, using an idempotency key derived from the order and the step, and treating a duplicate-key rejection from the target system as a success rather than an error. If the warehouse system returns a conflict because the order reference already exists, the correct response is to fetch that existing order, confirm it matches, and move the state forward. Otherwise a single timeout will have you creating the same picking job twice.

Classify the failure before you decide what to do about it

Retrying everything three times and then giving up is not a policy, it is an absence of one. Different failure classes need different handling, and the classification should be made in code at the point of failure, not inferred later by whoever is reading the exception queue.

The classification also determines who gets told. Transient failures should be invisible to operations. Validation failures should reach the person who can correct the source data. Business rejections belong to finance or account management. Ambiguous outcomes belong to the integration owner, because they require a check against the target system before anyone acts.

  • Transient — Timeouts, 429 responses, 502 and 503, connection resets. Retry with exponential backoff and jitter, respecting any Retry-After header. Cap the attempts, and escalate to the exception queue rather than retrying indefinitely.
  • Validation — A missing SKU, an address line exceeding the target field length, a country code the ledger does not recognise. Retrying is pointless because the payload will not change. Quarantine the record with the specific field named, and route it to whoever owns the source data.
  • Business rejection — Credit limit exceeded, customer account on hold, nominal code closed for the period. The call was well-formed and the target system said no on policy grounds. This is a finance decision, not a technical one, and the integration should hold the order rather than force it through.
  • Ambiguous — The request timed out after the target began processing, or the connection dropped before the response was read. The outcome is unknown. Never retry blind. Query the target by your idempotency key or external reference first, then act on what you find.
  • Partial batch — A multi-line submission where some lines were accepted and others rejected. Treat the batch as failed only for the rejected lines, record line-level status, and never resubmit the whole batch as a correction.

Compensating actions, and the ones you should not attempt

When a later step fails permanently, the question is whether to unwind the earlier ones. The saga pattern says each step should have a compensating action that reverses its effect. That is sound as far as it goes, but in an order flow the compensations are asymmetric in cost and some of them should never be automated.

Cancelling an allocation in Mintsoft before picking has begun is cheap and safe, and can reasonably be automated. Cancelling a DPD consignment that has not yet been manifested is also straightforward. Voiding a Sage 200 sales invoice is a different matter: once posted, the correct reversal is a credit note, which is a financial document with audit implications, and in most businesses it requires approval. Automating credit note creation to tidy up an integration failure is how you end up explaining an unexpected set of reversals to an auditor. Our default is that the integration raises the exception and a person in finance decides.

The hardest case is the one where the physical world has moved ahead of the data. If the pick is complete and the parcel is on a van, there is no compensation available. The goods have gone. The only correct behaviour is forward recovery: hold the order in a state that says dispatched but not invoiced, make that state visible on a daily report, and let finance resolve the account block and release the posting. Attempting to cancel a Shopify order that the customer has paid for, because a downstream ledger post failed, is worse than the original problem.

A useful rule when scoping: for each step in the flow, write down the compensating action, its cost, whether it is automatic or manual, and the point of no return after which it is unavailable. That table is short, it takes an afternoon to produce during discovery, and it prevents an enormous amount of argument later. It also tends to reveal that the correct ordering of steps is not the one the business assumed, because moving the ledger post earlier or later changes which failures are recoverable.

Reconciliation, exception queues and the people who work them

Detection is the part that gets underinvested. A partial failure is by definition a state where nothing threw an obvious error at the person who cared, so the only reliable detection mechanism is a scheduled comparison between systems. Run a reconciliation job on a fixed cadence, typically overnight and again mid-morning, that counts orders by state and flags anything that has been in a non-terminal state longer than its allowed dwell time. Orders in sent_to_wms for more than four hours, or dispatched for more than twenty-four hours without an invoice, are the two checks that catch most real incidents.

Complement the state check with a value check. Total the Shopify orders created for a given day, total the Sage 200 sales invoices with that day's reference range, and report the difference with a list of the order numbers accounting for it. Operations staff trust a report that reconciles to a number more than they trust a technical dashboard, and a report that is expected to show zero is a report that gets read. Where the day legitimately does not balance, because an order is held pending a credit decision, the reason should appear on the line rather than being left for someone to work out.

The exception queue needs an owner with a name, a working procedure for each failure class, and a service level. A queue that everyone can see and nobody owns fills up until it is ignored. We generally recommend two roles: a data owner who resolves validation failures at source, and a finance approver who resolves business rejections. The integration team owns only the ambiguous and transient categories. Make the resolution action available from the queue itself, so that fixing a SKU mapping and replaying the order is one action rather than a support ticket.

Finally, prove all of this before go-live rather than discovering it in week three. Acceptance testing should include deliberate fault injection: stop the warehouse endpoint mid-run and confirm the state machine resumes cleanly; return a business rejection from the ledger and confirm the order holds rather than being retried into a duplicate; time out a call after the target has committed and confirm the ambiguity check finds the existing record instead of creating a second one. Those three tests take an hour to script and they are the difference between an integration that degrades gracefully and one that quietly corrupts the ledger. If you would like this designed and evidenced as part of a fixed scope, we can be reached on 01303 883111 or at hello@api-integrations.co.uk.

Key points

  • Give every order a durable integration record with an explicit state, so a process that dies mid-flow can be resumed rather than reconstructed from logs.
  • Classify each failure as transient, validation, business rejection, ambiguous or partial batch, and route each class to the person who can actually resolve it.
  • Automate cheap compensations such as cancelling an unpicked allocation, but never auto-post credit notes or cancel paid orders to tidy up a downstream failure.