Why integration acceptance testing on happy-path orders proves very little

A single-line order, paid by card, shipped to a UK address, at standard VAT, with stock on hand, will pass through almost any order integration on the first attempt. If that is the shape of the test set, integration acceptance testing becomes a demonstration rather than a test: it confirms that the connection is authenticated and the field mapping compiles, and it tells you almost nothing about the behaviour you will actually depend on. The orders that cause disputes between operations and finance in the first fortnight after go-live are never that shape.

The problem is structural. The build team writes test cases from the specification, and the specification describes the intended flow. Edge cases are, by definition, the things nobody thought to specify. A developer who has not spent two years answering customer service escalations will not naturally think to test an order where the customer changed their delivery address after dispatch was created but before the carrier manifest closed. An operations manager who has handled that situation forty times will produce it from memory in under a minute.

There is also a measurement problem. Passing fifteen invented test cases creates a figure — fifteen out of fifteen — that reads as completeness and is not. Passing eighty-three real orders drawn from your own trading history, selected because they represent recognised categories of difficulty, creates a different kind of assurance. You are not proving that the integration works in principle. You are proving that it would have handled last year correctly, which is the nearest thing to evidence about next year that anyone can offer.

This matters commercially as well as technically. Under a fixed scope agreement, the acceptance test is the boundary between the build and the warranty, and between the warranty and chargeable change. A test set built from real orders makes that boundary defensible for both sides. Nobody has to argue about whether a defect was in scope if the case that exposed it was listed and signed before the first line of code was written.

Building the edge case corpus from real order history

Start with extraction rather than imagination. Pull twelve months of order headers and lines from the source system — a Shopify export, an Amazon Seller settlement and order report, or whatever the channel actually is — into a working table, and then query it for structural outliers rather than reading it. Orders with more than one shipment. Orders with a negative line. Orders where the paid total does not equal the sum of lines plus shipping plus tax. Orders with a delivery country outside the UK. Orders with more than two discount codes. Orders where the customer record has no email address. Each of those queries returns a count and a set of order references.

The counts are as important as the cases. If three-line orders with mixed VAT rates represent four per cent of volume, that is roughly one in twenty-five orders arriving at Sage 200 with a posting decision to make, and the integration must get it right without supervision. If orders shipped in two parcels represent 0.2 per cent, the correct response might still be to handle them properly, but you now know what you are buying and can price the handling against the manual cost of not having it. Frequency turns a list of curiosities into a prioritised scope.

Then add the cases that do not show up in a database query because they are sequences rather than states. A refund issued eleven days after the invoice posted. A cancellation that arrives while the warehouse is picking. A payout that lands in a different month from the orders it settles. These come from interviews, not SQL: sit with the person who currently fixes these by hand and ask what they did last Tuesday. Record the order reference each time, because a real reference lets you reconstruct the exact payload rather than approximate it.

Finally, de-identify and freeze. Replace customer names, addresses and email addresses with consistent synthetic values, keeping the structural characteristics that matter — address line lengths, missing county fields, non-ASCII characters in names, PO Box style addresses that a carrier will reject. Freeze the resulting corpus as a versioned set of JSON payloads held alongside the code. A corpus that lives in someone's spreadsheet will be out of date within a quarter; a corpus held in the repository becomes a regression suite that survives staff changes and future development.

Environments, fixtures and the limits of a sandbox

Replaying real orders requires somewhere safe to replay them into, and this is where plans usually weaken. A Shopify development store will accept orders through the Admin API and behaves close enough to production for most purposes. A Sage 200 test company can be restored from a production backup and reset between runs, which makes it the most useful of the three systems involved in a typical retail flow. Amazon Seller is the difficult one: the sandbox does not reproduce settlement behaviour in any meaningful way, so settlement scenarios have to be tested by replaying captured production reports into the integration rather than by generating them.

Be explicit in the scope document about which parts of the test are live-system, which are replayed, and which are stubbed. If carrier label generation against Royal Mail is stubbed during acceptance testing because the account cannot produce test manifests, say so, and schedule a separate pre-go-live check against a small batch of real consignments. A stub that nobody wrote down becomes an assumption, and assumptions are what fail in week one.

Reset discipline matters more than environment fidelity. Each acceptance run should start from a known state: a restored test company, a cleared queue, a wiped idempotency store. Without that, a passing run can be an artefact of residue from the previous run, and a failing run cannot be reproduced for diagnosis. Build the reset as a scripted step from the beginning rather than as a manual checklist, because it will be run forty or fifty times before sign-off and manual resets drift.

Where a warehouse system sits in the middle — Mintsoft, for example, receiving orders and returning despatch confirmations — the sequencing of the test becomes part of the test. Acceptance should include a run where the despatch confirmation arrives before the integration has finished writing the order to the ERP, because that happens in production whenever a pick is unusually fast or a queue is unusually slow, and the resulting behaviour should be defined rather than discovered.

The categories that earn their place in the test set

Every business has its own awkward orders, but a consistent set of categories recurs across retail and wholesale integrations. These are the ones worth checking the corpus against before the scope is fixed, on the basis that an absence is more likely to mean you have not looked than that the case does not occur.

  • Zero-value and fully discounted orders — Replacement shipments, staff orders and hundred-per-cent promotional codes produce an order with a payable total of nil. Confirm what posts to the ledger, whether a sales invoice is created at all, and what happens to the stock movement that still needs recording.
  • Mixed VAT rates on a single order — A zero-rated item and a standard-rated item in the same basket, with a basket-level discount and a single shipping charge, requires a defined apportionment rule. Test the rounding at line level and confirm the invoice total matches the amount actually taken.
  • Split and partial despatch — One order, two shipments, two carrier consignments, possibly two days apart. Determine whether finance expects one invoice or two, and confirm that the second despatch does not re-post the first set of lines.
  • Cancellation mid-flight — An order cancelled after it has reached the warehouse but before it ships. The test should establish whether the integration can withdraw the order, whether it raises a credit, and what the operator sees if neither is possible.
  • Refunds that cross a period boundary — A refund in April against an invoice posted in March. Confirm the posting date rule and whether a closed period causes a hard failure, a queued retry, or a silent write to the wrong month.
  • Duplicate and replayed webhooks — The same order notification delivered twice, three minutes apart. The second delivery should produce no second invoice, no second stock movement, and a log entry that an operator can interpret.
  • Customer and address data that breaks constraints — Names longer than the ERP field, missing county on an international address, apostrophes, accented characters, and the customer who placed an order with a corporate address and a personal email. These produce truncation and encoding defects that rarely surface before go-live.
  • Unknown or newly created SKUs — An order containing an item that exists on the storefront but not yet in the ERP. Define the behaviour — reject, hold, or create — and test that the chosen behaviour is visible to someone who can act on it.

Evidence, triage and what the acceptance run leaves behind

An acceptance run produces a result per case, and the result needs to be more than pass or fail. For each order in the corpus, record the input payload, the resulting records in each downstream system with their identifiers, and the expected values against which the comparison was made. Where the expectation concerns a financial figure — a net total, a VAT amount, a nominal code — state the figure explicitly in the test case rather than describing it. 'Posts correctly to the ledger' is not a testable assertion; 'net 124.17, VAT 24.83, nominal 4000' is.

Defects found during the run should be triaged into three categories, and the categories agreed before the run rather than argued afterwards. A defect against an agreed case is build work and is fixed under the fixed price. A defect that reveals a case nobody listed is a scope question, handled as a written variation with its own estimate and its own test case added to the corpus. A disagreement about the expected result is a business decision that goes back to finance or operations, not a developer judgement. Keeping those three separate is what prevents an acceptance phase from drifting into an open-ended second build.

Expect two or three full runs before sign-off. The first exposes a cluster of defects, often concentrated in one or two areas such as tax apportionment or address handling. The second confirms the fixes and usually surfaces a smaller number of consequential defects that the fixes introduced. The third should be clean, and a clean full run is the signature point. Partial re-runs are useful during development but should not be accepted as evidence, because the interactions between cases are precisely where regression occurs.

The corpus does not stop being useful at go-live. Held in the repository alongside the handover documentation, it becomes the regression suite for every subsequent change: a Shopify API version upgrade, a new VAT rate, a nominal code restructure in Sage 200, a change to how Amazon Seller reports a particular fee type. Each production incident should add a case, so that the set grows to reflect the business rather than the original specification. Over two or three years this becomes one of the more valuable assets of the integration — a written, executable description of how your orders actually behave, independent of whoever happens to be maintaining the code.

Key points

  • Draw the acceptance test set from twelve months of real order history rather than from the specification, and record the frequency of each edge case so scope can be prioritised against actual volume.
  • State expected financial results as explicit figures, agree defect triage categories before the run, and require a clean end-to-end run rather than accepting partial re-tests as evidence.
  • Keep the edge case corpus in the repository after go-live so it becomes the regression suite for API version changes, tax changes and nominal restructures.