What an idempotent api integration actually guarantees

An idempotent api integration is one where sending the same logical request twice produces one result, not two. That sounds obvious enough to skip past, which is precisely why it is the detail most often got wrong. The guarantee is not about the network being reliable; it is about what happens when the network is not. A request leaves your process, the connection drops before the response returns, and you are left with an outcome you cannot classify. The remote system may have created the invoice. It may not. Nothing in the HTTP response tells you which.

Most integration code handles two outcomes: success and failure. Production has three. The third — unknown — is where duplicates are born, because the sensible reaction to an unknown is to retry, and a blind retry against a system with no idempotency protection creates a second record. In finance systems this is not a cosmetic problem. A duplicate sales invoice affects the VAT return, the debtor ledger and the customer's remittance. A duplicate consignment with a carrier is billable. A duplicate stock adjustment quietly corrupts availability across every sales channel that reads from it.

The idempotency key is the contract between the two systems about what counts as the same request. Your side asserts "this is attempt n of one specific business event"; the remote side undertakes to act once and, on subsequent attempts, to return the result of the first action rather than performing it again. Everything that follows — derivation, scope, storage, expiry — is about making that assertion honest. A key that changes between attempts is not a key. It is a decoration.

Deriving and scoping the key

The single most common defect is generating the key in the wrong place. If the key is a fresh UUID created inside the HTTP client wrapper, or inside the retry loop, then every attempt carries a different key and the protection is worthless while looking entirely correct in code review. The key must be created once, at the point the business event is recorded, and persisted alongside that event before the first call is made. If your process restarts halfway through, the key must be recoverable from storage, not regenerated.

The safest keys are deterministic: derived from stable identifiers rather than randomness. Deterministic derivation means an operations manager re-running yesterday's failed batch produces the same keys as the original run, which is exactly the behaviour you want. Random keys are acceptable only when they are written to your own ledger before the call and read back on retry — which is more machinery for the same result.

Scope matters as much as content. Idempotency keys are almost always scoped to a tenant or credential and often to an endpoint, so the same key value used against two different endpoints may not collide in the way you expect. Check the documented format constraints too — length limits, permitted characters, and whether the key travels as a header or as a field in the payload. Assume nothing is global unless the vendor says so in writing.

  • Source system identifier — Which system originated the event. Prevents collisions when two feeds write to the same target ledger.
  • Entity type and identifier — For example, order and the store's internal order id — not the human-readable order number if that can be reissued.
  • Operation — Create, amend, void. The same order legitimately produces several distinct calls and each needs its own key.
  • Revision — A monotonically increasing version for the entity. An intentional amendment must produce a new key; a retry of the same amendment must not.

What the platforms actually give you

Support varies more than most project plans assume, and this is a discovery question, not a build-time discovery. Xero accepts an idempotency key header on create operations, with a retention window measured in hours rather than days — long enough for a transient failure and a retry, not long enough for a queue that has been paused overnight. QuickBooks takes a request identifier on transaction endpoints and will return the original response for a repeated identifier within its own window. In both cases the protection is real but time-bounded, and the bound is shorter than the interval between many overnight batch runs.

NetSuite offers a different and often better route: upsert by external id. If you write your own stable reference into the external id field, a repeated PUT updates the existing record rather than creating a second one. That is idempotency by construction rather than by header, and it survives indefinitely because the uniqueness lives in the record itself. The same thinking applies anywhere the target exposes a reliable unique reference field you control.

Shopify's Admin API does not provide a general-purpose idempotency header across resources, so a create-order or create-refund retry needs protection on your side: a lookup by your own reference before writing, plus a local ledger entry. Carrier APIs deserve particular care. Royal Mail and DPD label creation is a billable act; treat every label call as non-idempotent unless proven otherwise, key it on the consignment, and never let an automatic retry create a second label without a check against your own record of what was already produced.

Where the remote gives you nothing, the fallback is a dedupe ledger of your own with a unique constraint enforced by the database, not by an application-level "check then insert". Under concurrency, check-then-insert loses races. A unique index does not.

Storing the outcome, not just the key

A key on its own prevents nothing. What prevents duplicates is a durable record of what happened to that key. The minimum useful row contains the key, a fingerprint of the canonicalised request payload, a status of in-flight, complete or failed, the HTTP status and response body returned by the remote, the remote identifier that was created, and timestamps for first and last attempt. The fingerprint is a hash of the payload after normalising field ordering and formatting, so that cosmetic serialisation differences do not look like changed content.

Write the row before the call, not after. The insert-first pattern is what makes the ledger meaningful: if a second worker picks up the same event, its insert fails on the unique constraint and it can either wait for the in-flight attempt to resolve or abandon its own. A row written only after a successful response tells you nothing about the case you actually care about, which is the request that never returned.

The fingerprint earns its keep when the same key arrives with a different body. That is either a defect in your key derivation — a mutable field leaked into it — or a genuine amendment that should have produced a new key. Either way the correct response is to refuse the call, raise an alert, and have someone look at it. Silently overwriting is how a corrected invoice total quietly disappears.

Finally, keep your ledger longer than the remote keeps its own. Vendor retention windows exist to protect the vendor's storage, not your audit trail. When a settlement query arrives six weeks later asking why a credit note exists twice, the answer lives in your records. Retaining at least through the end of the financial period is a reasonable default, and it costs very little.

Where idempotency quietly breaks

Multi-step workflows are the first trap. An order-to-cash flow might create a customer, create an order, allocate stock and raise an invoice. One key for the whole workflow is meaningless, because the remote systems see four independent calls. Each call needs its own key, and the workflow needs to record which steps completed so that a resumed run picks up at step three rather than starting again at step one. Partial completion is the normal case after a failure, not the exception.

Batch payloads are the second. Posting fifty lines under a single key only protects you if the remote treats the batch atomically, and many do not. If thirty lines are accepted and twenty rejected, a retry under the same key may re-post the thirty. Where the platform supports per-item references, use them; where it does not, break the batch down and accept the extra calls, subject to rate limits.

The third and most frequent cause in practice is human. Someone in operations re-runs a failed export, or replays a webhook from a vendor console, or restores a queue from a snapshot. None of that is unreasonable, and all of it should be safe. It is only safe if keys are deterministic and the ledger is consulted on every path, including the manual one. If your re-run script bypasses the dedupe check because "it is a controlled re-run", it will eventually be run at the wrong moment.

The fourth is a deployment problem. Changing the key derivation algorithm — adding a field, changing a separator — makes every in-flight event look new. If derivation must change, version it, record the version in the ledger, and drain the queue before the cutover.

Proving it before go-live

Idempotency is not something to assert in a design document and hope for. It is testable, and the tests belong in the acceptance stage where they can be witnessed by the client rather than discovered in week three of live running. The scenarios below take an afternoon to build and will outlive the project.

Beyond the test cases, put two things in place for live running. First, a counter for rejected duplicates — the number of times the ledger or the remote refused a repeated key. A count of zero forever usually means the check is not wired in, not that nothing has ever been retried. Second, a daily reconciliation that compares counts by document type between source and target. Idempotency reduces duplicates; reconciliation is how you find the ones that slipped through by another route.

On handover, the key derivation rule, the ledger schema, the retention windows for each remote platform and the safe procedure for re-running a failed batch should all be written down in the documentation pack. Whoever inherits the integration in three years will not reverse-engineer it from the code under pressure at month end. This is the sort of detail that belongs in the fixed scope agreed before any code is written, because retrofitting duplicate protection into a live financial feed is considerably more expensive than building it in.

  • Interrupted send — Kill the process after the request is transmitted but before the response is read. Restart and confirm exactly one record exists at the target.
  • Concurrent submission — Fire the same event from two workers simultaneously. One should succeed; the other should be refused by the unique constraint.
  • Same key, changed payload — Resubmit with an altered total. The call must be rejected and alerted, not accepted.
  • Expired remote window — Retry after the platform's retention period has lapsed. Your own ledger, not the vendor, must stop the duplicate.
  • Operator re-run — Execute the documented re-run procedure over a batch that partially succeeded. Confirm no new records are created for the lines that already posted.

Key points

  • Generate the idempotency key once, when the business event is recorded, and persist it before the first call — a key created inside the retry loop protects nothing.
  • Platform support is inconsistent and time-bounded, so keep your own dedupe ledger with a database-level unique constraint, written before the call and retained longer than the vendor's window.
  • Test interrupted sends, concurrent submissions and operator re-runs during acceptance, and document the key derivation and safe re-run procedure in the handover pack.