Why an openai api integration usually starts badly

The first openai api integration inside a business is rarely a project. Somebody in marketing needs product descriptions drafted, somebody in finance wants supplier remittance emails summarised, and a developer adds a key to an environment variable and ships it. Within a few months there are four or five keys in circulation, no reliable record of which application spent what, and no way to answer a straightforward question from the board about what customer data has been sent to a third party and when.

That pattern is not a failure of discipline. It is what happens when a capability is easy to call and nobody has been asked to own it. The direct call is three lines of code. The controls around it — attribution, retention, fallback, approval of model changes — are a piece of infrastructure, and infrastructure only gets built when somebody names it.

An internal API gateway is the usual answer. The word covers a range of things: a commercial gateway product, a reverse proxy with policy plugins, or a small service written in-house that exposes a handful of internal endpoints and calls OpenAI on their behalf. The mechanism matters less than the consequence. Once every model call passes through one place that you control, a set of decisions that were previously implicit become explicit, and several defaults that worked fine for a single application stop working for ten.

The gateway is a contract, not a pass-through

The tempting design is a transparent proxy: accept whatever the OpenAI SDK sends, forward it, return the response unaltered. It is quick to build and it is the wrong shape for most businesses. A transparent proxy means every consuming team still chooses its own model, its own prompt, its own output format and its own error handling. You have centralised the credential and nothing else.

The design that holds up is task-shaped. Rather than exposing a chat completions endpoint, the gateway exposes endpoints that correspond to jobs the business actually does: classify this inbound email, extract these fields from this purchase order PDF, draft a reply to this support ticket in house style. Each endpoint owns its own model selection, its prompt template, its output schema and its acceptable latency. The caller sends a small, validated payload and receives a small, validated object.

This changes the unit of change control. When the prompt for supplier invoice extraction needs adjusting, it is adjusted once, tested once against a stored set of examples, and released once. Under a transparent proxy the same change has to be made in every application that does the work, and the ones nobody remembers never get updated. It also means the response contract is yours: if you later decide a particular task runs better on Anthropic Claude, the calling systems do not need to know.

There is a cost to this. Task endpoints require someone to define the tasks, which is discovery work rather than engineering work, and it is the part most often skipped. In practice a business of moderate size has between six and fifteen genuinely distinct model tasks. Enumerating them is a morning's work with the right people in the room, and it determines everything that follows.

Timeouts, retries and streaming: the settings that bite first

Most gateway deployments carry defaults inherited from ordinary web traffic, and those defaults are wrong for model calls. A default upstream read timeout of thirty seconds is generous for a CRM lookup and short for a long-form generation with a reasoning-heavy model. The first symptom is intermittent 504s under load, which developers then diagnose as an OpenAI problem rather than a gateway problem, because the provider's own status page is clean.

Retries need equal care. A blanket retry policy at the gateway layer is dangerous, because a request that timed out at your edge may still be running upstream, and a retry produces a second billable completion and, in agentic designs, a second set of side effects. Retry on connection failures and on 429 and 5xx responses; do not retry on a client-side timeout unless the downstream task is genuinely idempotent and you have an idempotency key to deduplicate against.

Streaming is where the sharpest surprises live. A gateway that buffers responses will happily collect an entire server-sent event stream and deliver it in one piece, which defeats the purpose and pushes the whole generation inside the timeout window. Proxy buffering has to be disabled on streaming routes, and any compression applied at the edge has to be checked, because chunked text events and aggressive gzip settings interact poorly.

  • Timeout budget — Set per endpoint, not globally. A classification task might get fifteen seconds; a long document summarisation might need three minutes. Publish the figure to callers so their own timeouts are longer than yours, not shorter.
  • Retry policy — Exponential backoff with jitter, a hard cap on attempts, and a rule that non-idempotent tasks are never retried automatically. Record every retry against the original request identifier.
  • Streaming routes — Buffering off, compression checked, and a heartbeat or keep-alive strategy agreed with whatever sits in front of the gateway, including any corporate load balancer.
  • Concurrency limits — A per-consumer cap so that one batch job cannot exhaust the organisation's rate limit and degrade an interactive support tool that shares the same account.

Token accounting and the arrival of a real cost record

Direct calls give you one invoice and no breakdown that means anything to a finance manager. A gateway gives you the chance to produce a cost record per request, and that is usually the change that justifies the work internally. Every response from OpenAI carries a usage object with prompt and completion token counts; the gateway records those alongside the model name, the consuming application, the task endpoint and a timestamp, and the rest is arithmetic against a price table you maintain yourself.

One detail catches people out. Streaming responses do not return usage figures by default; the request has to ask for them explicitly, and if that is missed the gateway will log a null token count for exactly the requests that tend to be the most expensive. Estimating tokens from character counts afterwards is a poor substitute and will not reconcile with the provider invoice at month end.

Once the cost record exists, budgets become enforceable rather than aspirational. A per-consumer monthly ceiling, a soft alert at seventy per cent and a hard stop that returns a clear error is a modest amount of code and prevents the single largest unpleasant surprise in this area: a retry loop in a batch process that runs unattended over a weekend. Reconciliation against the provider's own usage figures should be a monthly task with an owner, in the same way that a card statement is reconciled.

Finance teams generally want the output in a form they already work with, which in practice means a periodic journal or a cost-per-department summary landing in Sage or Xero rather than a dashboard nobody opens. That is a small piece of integration on top of the gateway, and it is worth specifying at the outset rather than retrofitting.

Logging, redaction and what you are permitted to keep

The gateway is the natural place to log prompts and completions, and it is also the place where that decision becomes a data protection question rather than an engineering preference. If support tickets containing customer names and order history are being sent for summarisation, then storing full prompts creates a new repository of personal data with its own retention obligation, access controls and entry in the record of processing activities. Some organisations will accept that and document it properly; others should redact before the call leaves the building.

Redaction at the gateway is more reliable than redaction in each application, but it is not free. Pattern-based removal of card numbers, national insurance numbers and email addresses is straightforward. Names and free-text disclosures are not, and any claim that redaction is complete should be treated sceptically. A defensible position is usually a combination: remove what can be detected deterministically, restrict which endpoints may carry customer data at all, and set a short retention period on the request log — thirty days is a common landing point — with a longer retention on the metadata that finance and audit actually need.

Where the constraint is residency rather than retention, the gateway is what makes a change of provider tolerable. Azure OpenAI Service offers the same model family under a different commercial and hosting arrangement, and moving a subset of endpoints to it is a configuration change behind your own interface rather than a rewrite across every consuming application. The same argument applies to Amazon Bedrock if the organisation's data processing agreements already sit with AWS.

Model pinning, deprecation and a credible failover story

Model identifiers are not stable in the way most enterprise interfaces are. Aliases are updated, dated snapshots are retired with notice, and behaviour shifts in ways that are hard to detect without a test set. An application that calls an alias directly can produce different output from one week to the next with no change in your codebase, which is difficult to explain to an auditor and worse to explain to a customer. Pinning to a dated model version at the gateway, and treating a version change as a release with regression tests, restores normal change control.

The regression tests themselves need to exist before they are needed. For each task endpoint, keep a stored set of representative inputs with expected outputs or acceptance criteria — twenty to fifty cases is usually enough — and run them whenever the model, the prompt or the output schema changes. For classification and extraction tasks the criteria are objective. For generative tasks they are necessarily softer, and a human review step on a sample is more honest than a brittle string comparison.

Failover is worth designing but not worth overstating. A genuine multi-provider arrangement, where the same task can run against OpenAI or Anthropic Claude depending on availability, requires that both paths are tested regularly and that the output schema is enforced by the gateway rather than assumed from the model. An untested fallback is a liability, because it activates for the first time during an incident. If the appetite for that maintenance does not exist, a clear degraded mode — queue the work, tell the user, process when the provider returns — is the more defensible choice.

Whatever is chosen, it belongs in writing before the build starts: which tasks may fail over, which must not, what the acceptance criteria are, and who signs off a model version change. Those four answers are the difference between a gateway that survives its second year and one that becomes another undocumented dependency. The code, the prompt templates, the test sets and the runbook should all be handed over together, because the gateway is now on the critical path for whatever the business has put behind it.

Key points

  • Expose task-shaped endpoints rather than a transparent proxy, so prompts, model choice and output schemas are changed once and tested once.
  • Gateway defaults inherited from ordinary web traffic will break model calls: raise per-endpoint timeouts, restrict automatic retries to idempotent work, and disable buffering on streaming routes.
  • Token usage captured at the gateway gives finance a per-application cost record and makes budgets enforceable, provided streaming requests are configured to return usage figures.