Systems integration built on written contracts, not hopeful connections

What systems integration actually is

Connecting two systems is a morning's work. Keeping them connected through a token rotation, a rate limit change, an enum that gained a sixth value and a bulk import somebody ran on a Friday is the actual job.

There are two shapes of integration failure and the second is expensive. The loud shape throws errors and gets fixed the same day. The silent shape is a connection that stopped: no exceptions, a green dashboard, two systems drifting apart for three weeks until somebody in finance notices a wrong number.

Who it is for, and who it is not for

This is for companies where the same fact lives in four or more systems and somebody keeps them in step by hand. If you have two systems and a supported native connector, install the connector. Custom work earns its cost when the pairings, the volume, or the price of being wrong outgrow what a point-to-point tool can express.

Three things that must exist before the build

1. Credentials owned by the company, not by a person. A personal OAuth token works perfectly until that person changes role, and then the seam dies with no error anybody recognises.
2. A documented rate limit for every system on the seam, and somebody who can raise it. Integrations that pass testing and fail in production are usually meeting a ceiling nobody checked.
3. A named owner per system schema. If any employee can add a required field on a Tuesday afternoon, the seam breaks on a Tuesday afternoon and nobody connects the two events.

Who should not buy this

• Teams whose real problem is two departments disagreeing about who owns the customer. A seam surfaces that argument in production rather than settling it.
• Anyone hoping the integration will clean data on the way through. Normalising a format is fine. Inventing a missing value gives you bad data that is now auditable back to you.
• Teams who want a dashboard. That is a reporting job, and building it as a live seam makes it slower and more fragile.

What we actually build

Six components, and almost none of them contain a model. Integration is where AI projects go to die, so this layer gets built the boring way and the models go on top once facts move reliably.

The seam inventory and the contracts

One file per producer and consumer pair, in version control, naming the key, the guarantee, the ordering rule, the retry policy, the rate budget, the dead letter owner and the notice period for schema changes. It is readable by an operations lead, and it is the artefact people argue about before anybody writes a transform.

The ingress layer

Webhook receivers that verify the signature, persist the raw body byte for byte, return 200 quickly and do no work in the request. Providers time out and retry if you process inline, which is how a slow downstream write becomes a duplicate storm from a well behaved sender. Where webhooks do not exist, polling with a stored cursor does the same job less elegantly, and the trade-off is in webhooks vs polling.

The queue and the workers

A durable queue with a visibility timeout longer than the slowest downstream call, partitioned on the entity key so events for one record are handled in order by one consumer. Concurrency is capped per destination, because the limit that matters is the receiving system's rate limit and not your CPU.

The transform layer

Mapping code, versioned, with recorded payloads as fixtures, rather than a mapping buried in a vendor's web interface where nobody can diff it. Transforms stay pure functions, so the whole mapping is testable offline against a folder of captured messages.

The idempotency store

A key built from the seam id and the producer's event id, held long enough to outlive your longest replay window, checked before every write. This is what turns at-least-once delivery into an effectively once outcome, and skipping it is the usual reason a customer gets the same email twice after an outage.

The dead letter queue, the replay tool and the dashboards

Messages that exhaust their retries land in a queue carrying the full envelope, the error, the attempt history and the trace id, and a command line tool replays them by seam and time range once the cause is fixed. Alongside it sits one dashboard per seam: events in, events out, lag, retry rate, dead letter depth, and an alert that fires when hourly volume falls below its floor.

How it works technically

The data flow is the same on every seam: verify, persist, acknowledge, enqueue, transform, write with an idempotency key, confirm, commit the cursor. The interesting decisions are all about what happens when one of those steps fails.

Acknowledge before you work

The receiver validates the signature, writes the raw body to durable storage exactly as it arrived, and returns 200. Providers enforce short timeouts and retry on anything slower, so a receiver doing the downstream write inline gets sent the same event again while the first attempt is still running. Most duplicate storms are caused by the receiving code, not the sender.

Ordering without serialising everything

Global ordering requires a single consumer and destroys throughput. Business logic needs something narrower: two updates to the same contact must not race, while updates to different contacts are independent. Partition on the natural key and each key is handled in sequence by exactly one worker. Where the queue cannot partition, take a short lease on the key before writing.

Idempotency, plus the stale write rule

Two protections are needed and teams build only the first. Deduplication stops the same event being applied twice. The stale write rule stops an older event being applied after a newer one: compare the event's occurred_at against the value stored on the record and drop the write if it is older. Without it, a retry landing ten minutes late overwrites a correction somebody just made, and nothing logs an error because the write was valid.

The envelope every producer writes to

One envelope shape across all seams pays for itself during the first incident. It gives every message an id to deduplicate on, a version to pin, a partition key, an actor field that kills echo loops, and a trace id that follows the fact across systems.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Integration event envelope, version 1",
  "type": "object",
  "additionalProperties": false,
  "required": ["event_id","event_type","schema_version","occurred_at","source","entity","payload"],
  "properties": {
    "event_id": { "type": "string", "format": "uuid",
      "description": "Generated once by the producer, identical across every retry of the same fact. This is what the consumer deduplicates on." },
    "event_type": { "type": "string", "pattern": "^[a-z_]+\.[a-z_]+\.(created|updated|deleted)$",
      "examples": ["billing.invoice.created", "crm.contact.updated"] },
    "schema_version": { "type": "integer", "minimum": 1 },
    "occurred_at": { "type": "string", "format": "date-time",
      "description": "When the fact happened in the source, UTC with an offset. Not when it was sent: retries make sent_at useless for ordering." },
    "source": { "type": "string", "enum": ["billing","crm","product_db","support"] },
    "actor": { "type": "string",
      "description": "What caused the change. Stamp this integration's service name on its own writes, then drop inbound events carrying it. Echo loops stop here." },
    "entity": {
      "type": "object",
      "required": ["type","natural_key"],
      "properties": {
        "type": { "type": "string", "examples": ["contact","invoice","order"] },
        "natural_key": { "type": "string",
          "description": "Also the partition key, so events for one entity stay in order without serialising the stream." }
      }
    },
    "payload": { "type": "object",
      "description": "Full current state of the fields in scope, never a diff. A diff applied out of order corrupts silently. Full state does not." },
    "trace_id": { "type": "string" }
  }
}

Delivery guarantees, and why exactly once is a sales phrase

Pick the guarantee per seam and write it in the contract, because the choice decides what the consumer has to build. There are three, and the third does not exist at the network level no matter what a product page says.

Terms used on this page

At-least-once delivery

A guarantee that a message will arrive one or more times, which is what queues and webhook senders actually provide. The receiver is responsible for recognising repeats. Anything marketed as exactly once is at-least-once delivery plus deduplication performed somewhere specific.

Idempotency key

A value derived from the event itself, usually a hash of the seam identifier and the producer's event id, stored on first processing and checked before every write. If the key is already present, the write is skipped and the message is acknowledged.

Partition key

The field a queue uses to decide which consumer handles a message. Using the entity's natural key preserves order for that entity while letting unrelated entities be processed in parallel.

Poison message

A message that fails every time it is processed, usually from malformed data rather than a transient fault. Without a retry ceiling and a dead letter queue, it blocks its partition and stalls every entity behind it.

The seam contract, written before any code

Every failed integration we have been asked to rescue was missing at least three of these six clauses. They are decisions rather than documentation, and each one changes what gets built.

# seam: billing -> crm (contact, invoice). One file per pair, in version control.
seam_id: billing_to_crm_v1
owner_team: revenue_ops          # the team paged when this breaks, named not implied

identity:
  natural_key: email_lower       # both sides agree on this and nothing else
  fuzzy_match: false

delivery:
  guarantee: at_least_once       # the only one a network actually offers
  consumer_must: deduplicate
  ack_policy: after_commit       # never ack before the write is durable

ordering:
  partition_by: natural_key      # per-entity order, global order not required
  stale_write_rule: drop_if_occurred_at_older_than_stored

idempotency:
  key: "sha256(seam_id + event_id)"
  ttl_hours: 168                 # must outlive your longest replay window
  on_duplicate: ack_and_skip

retries:
  attempts: 5
  backoff: exponential
  jitter: full                   # without it, every consumer retries in the same second
  retry_on: [429, 500, 502, 503, 504, timeout]
  never_retry_on: [400, 401, 403, 404, 422]

rate_budget:
  live_calls_per_minute: 240
  backfill_calls_per_minute: 60  # separate lane, so a backfill cannot starve live traffic

dead_letter:
  queue: billing_to_crm_dlq
  alert_after: 10
  reviewed_by: revenue_ops
  review_cadence: "every weekday 09:00"
  replay: "bin/replay --seam billing_to_crm_v1 --since"

schema_change:
  notice_days: 30
  contract_tests: "consumer payloads replayed in producer CI before merge"

health:
  expected_events_per_hour_min: 5   # alert on silence, not only on errors
  max_lag_seconds: 300
  reconcile: "nightly count and field sample, ticket on drift"

The build process stage by stage

Six stages. The first two produce no running code, and skipping them is why the third takes twice as long as anybody planned.

1. Seam inventory3 to 5 days

   List every place data moves between systems today, including the exports somebody does by hand and the script on a laptop. Each entry gets a producer, a consumer, a volume estimate, an owner and a note on what breaks if it stops for a day. Most inventories come back with two or three seams nobody knew were load bearing.

2. Contracts and payload capture3 to 5 days

   Write the contract per seam and capture real payloads from every producer, including malformed ones. Reading a hundred real messages teaches you more than the vendor's documentation does, and it is where you find the field documented as an integer that is sometimes an empty string.

3. Skeleton: ingress, queue, dead letter, observability1 to 2 weeks

   The plumbing before any business logic: receivers, the queue and its visibility timeout, the retry policy, the dead letter queue and its alert, trace propagation, dashboards. It carries no value on its own and it is what makes every later seam take days instead of weeks.

4. First seam end to end1 to 2 weeks

   One seam in production with real traffic and a rollback. Break it in staging first: kill the consumer mid-write, send the same event five times, send one with a missing required field, send one an hour late. The behaviour under each of those is the design.

5. Remaining seams, then the backfill1 to 4 weeks

   Later seams reuse the skeleton and are mostly mapping and tests. The backfill runs last, on its own rate lane, resumable from a cursor, with the destination's automations paused first, because a bulk write fires workflows built for human activity.

6. Handover and shared on-call1 week

   Runbooks, an incident walkthrough, and two weeks where your team is paged first while ours is on the call. The failure that teaches most is the one that happens while both teams are watching.

What you get at handover

The deliverable is a system your team can operate without us, so the artefacts matter as much as the running code. All of it sits in your repositories and your accounts from the first day.

The last item is the one clients remember. Every system has edges its builder knows about and its operator does not, and writing them down is the difference between an inheritance and an ambush. More on that in who owns the automation after launch.

Where systems integrations go wrong

Five failure modes account for most of the integration incidents we are called in to diagnose. None are exotic, and all are cheaper to prevent than to find.

The echo loop

System A writes to B. B fires a webhook. The handler writes back to A, which fires a webhook. This runs until a rate limit stops it or somebody notices a record updated four thousand times in an hour. The cure costs one field: tag every write with the actor that made it, and drop inbound events whose actor is your own integration.

The retry storm after an outage

A provider returns 503 for twenty minutes and every consumer retries. Without jitter they retry in the same second, so when the provider recovers it meets a thundering herd and falls over again. Exponential backoff with full jitter, plus a circuit breaker that stops calling a destination that is clearly down, turns a self-inflicted second outage into a queue that drains quietly.

The silent stop

A credential expires, a subscription lapses, a consumer dies without restarting. No errors are raised because nothing is running to raise them. Alert on the absence of expected events with a floor per seam per hour, and reconcile counts nightly. Zero events processed looks identical to a quiet Sunday unless something is watching for it.

Schema drift with no notice

An enum gains a sixth value. A nullable field becomes required. A string that always parsed as a date arrives empty. Reject and log unknown values rather than coercing them, keep the raw payload so you can replay once the mapping is fixed, and run consumer owned contract tests inside the producer's pipeline for anything internal.

The rate limit discovered in production

Limits are usually per account rather than per integration, so a new seam shares an allowance with the marketing tool, the reporting export and whatever somebody built in 2023. Budget the seam explicitly, measure real consumption in week one, and give backfills a smaller separate lane. The arithmetic is in rate limits, retries and backoff.

What it costs to run once live

An integration layer costs less to run than most teams expect and more to maintain than they budget. The infrastructure is small. The maintenance is a real recurring line, driven by other people's release schedules rather than anything you control.

The number worth running before you build is retry amplification, because it sets both your API consumption and the human time the seam quietly demands each week. A three percent transient failure rate sounds harmless until it is multiplied by five attempts across a busy seam.

Two lessons fall out of that arithmetic. Retries multiply consumption against a limit somebody set without knowing your seam existed, so the peak figure is the one to compare against your tier. And a permanent failure rate under one percent still generates a daily queue somebody has to own.

How to tell whether you need this

Integration work is easy to justify emotionally and hard to justify with evidence, which is how companies end up with an expensive layer moving fields nobody reads. Four or more of these and the case is real. Two or fewer and the connector platform you already pay for is probably enough for another year.

How to start

The first conversation is technical and takes about an hour. Bring somebody who knows the systems rather than somebody who knows the org chart. We come out of it with a seam inventory, a first contract, and an honest view of whether this is a build or a configuration job.

1. Send the list of systems, roughly how many records move between them per day, and what breaks first when a connection stops.
2. We spend an hour on the two seams that hurt most, and write one contract live so you can see what the artefact is.
3. You get that contract, a sizing with the assumptions written out, and a straight answer on whether a connector platform already covers this. That answer costs nothing and is sometimes yes.
4. If it is a build, we start with the inventory and the skeleton. The first seam reaches production within a few weeks, and it is the one costing you most today rather than the one easiest to demonstrate.

If the real problem is a whole process held together by people copying between tabs, the shape you want may be workflow automation rather than a seam layer. We will say so.